ElevenLabs adds memory and model routing to voice agents: highlights from Summit NYC

A voice agent that interrupts customers, forgets them a minute after a call and invokes an expensive model for every utterance is a familiar frustration for anyone who has built LLM-powered support. At its New York Summit on November 11, ElevenLabs unveiled a set of updates aimed squarely at those problems. Some are already available; others are still in preview.

Interaction Engine: who speaks and which model thinks

The most interesting development for engineers is Interaction Engine, currently in preview. It manages the rhythm of a conversation: when the agent should speak and when it should stop if a person interrupts. Its second function is selecting an LLM for a specific part of the conversation. A cheaper model handles triage and keeps token spending in check, while a frontier model steps in when the task becomes more complex.

The idea is not new: many teams manually build routers that use multiple models. But if ElevenLabs takes care of both turn timing and routing, it could spare teams an entire layer of custom infrastructure. The company provided no benchmarks, prices or release dates, so teams will have to test it on their own calls.

Memory and Routines: the agent remembers and finishes the work after the call

Memory is coming to ElevenAgents. After each conversation, the agent condenses it into a structured record and updates that record when the customer returns. The CRM remains the primary source of truth, while Memory captures details that have no dedicated CRM field: preferences, constraints and context. One example from the presentation: a customer speaks Spanish and is only available after 17:00; the agent will remember and take that into account. A subscriber who reported a move in March will not have to repeat it in April: the agent will proactively offer to transfer the service to the new address and close it at the old one.

For product teams, this raises two questions: what exactly the agent is allowed to remember, and how the customer is informed. The announcement refers to returning, authenticated customers, which is a sensible restriction.

Routines is an automation layer around conversations: multistep workflows before, during and after calls. Examples include prescreening a mortgage applicant and calling back within minutes; reminding patients about appointments and offering an available slot to someone on the waiting list; emailing a report, writing data to the CRM or initiating a call. All of this happens inside ElevenAgents, without a separate collection of services. Yahoo, Westgate and KPN were named among the platform’s customers.

Fine-tuning, ElevenAI and the ecosystem

ElevenLabs already develops industry-specific speech recognition models. Scribe v3 Medical distinguishes between similar drug names, such as cycloserine and cyclosporine: a tuberculosis treatment and an immunosuppressant, respectively. A finance model has been promised next. The new development: “in the coming months,” the company will help fine-tune selected open language models and its own voice models using data from live deployments. Businesses choose the metric, such as CSAT or reliability, and ElevenLabs researchers help with training. CEO Mati Staniszewski compared it to every company having its own RL stack. Terms, prices and a model list have yet to be announced.

The company also unveiled ElevenAI, a “colleague” app for employees: dictation and meeting notes powered by Scribe v3, including for in-person meetings, with summaries and action items for each participant. Participant consent is built in. Free early access opened on November 11.

On the implementation side: a Partner Program with more than 550 partners in 44 countries, including Google Cloud, IBM, Deloitte and Genesys; an alliance with McKinsey and QuantumBlack; and a Marketplace with industry-specific applications. Starting next year, customers will be able to buy agents through it under their existing ElevenLabs contracts. The NVIDIA partnership has also expanded: Nemotron 3.5 Super VL is available in ElevenAgents, 15 MW of Vera Rubin capacity is being added, and a locally deployed voice agent is being developed for Poland’s national e-Health platform to book medical appointments for patients who only have a landline.

What to do with this

If you build voice agents, revisit three things: where customer memory lives beyond the CRM, whether models are assigned according to the complexity of each step, and who handles post-call processing. Almost everything on the list is either “coming soon” or in preview, so assess it using metrics from your own calls, such as the rate of transfers to human agents, CSAT and cost per conversation, rather than the stage presentation.

One final consideration is vendor lock-in. Fine-tuning “with ElevenLabs researchers” and purchases through the marketplace tie you more closely to a single platform. That is convenient for getting started quickly, but it is worth planning your exit in advance.