ElevenLabs gives voice agents memory and plans to fine-tune models for each client

At its New York Summit, ElevenLabs previewed agents that remember returning customers, an automation layer called Routines and a meeting app, ElevenAI. It also promised fine-tuning of open-source language models and its own voice models for each business.

CEO Mati Staniszewski framed the push as infrastructure: “Within a few years, conversational agents will become critical infrastructure for every enterprise, as essential as cloud or payments.” Most of what was shown is a preview, with dates left vague.

Memory, Routines and a conversation conductor

The biggest agent change is Memory in ElevenAgents. According to ElevenLabs, an agent will recognize an authenticated returning customer and recall details they shared earlier, such as a language preference or a time window for calls. After each conversation it condenses the exchange into a structured record and updates it on the next contact. The company says the CRM stays the main customer record; Memory holds details that never had a field. Its example is a phone subscriber who mentioned a move in March and is asked in April whether to transfer service.

Routines is a new automation layer that runs multi-step work before, during and after a conversation. ElevenLabs lists pre-screening a mortgage applicant and calling back within minutes, reminding patients before a visit and offering a canceled slot to someone on a waitlist. Routines can also send a report by email, write to a CRM or start a call.

A third piece, Interaction Engine, was only previewed. It handles turn-taking and interruptions and chooses which language model serves each part of an exchange: a cheaper model for triage, a frontier model for harder steps.

On models, ElevenLabs said that in the coming months it will help companies fine-tune certain open-source language models and its voice models using data from their live deployments. Customers pick the metric, such as customer satisfaction score or reliability. Staniszewski compared it to each company having its own reinforcement-learning stack. The company also named a finance-focused speech model as in the works, after Scribe v3 Medical for healthcare.

ElevenAI, partners and what is available now

ElevenAI is a new app that listens to conversations, including in-person meetings, and produces notes and next steps. It uses Scribe v3 for dictation and meeting notes, and ElevenLabs says it records with attendees’ consent and gives each person a summary and action items. Free early access opened on the day of the keynote. Memory, Routines, Interaction Engine and fine-tuning have no stated release dates or prices.

The company also launched a Partner Program. ElevenLabs says its network of consultancies, integrators and software vendors has grown past 550 firms spread over 44 countries, and Google Cloud, IBM and Deloitte are among them. It also announced an alliance with McKinsey and QuantumBlack. A curated ElevenLabs Marketplace of industry applications from partners went live; buying through it with an existing ElevenLabs commitment starts next year.

The Nvidia strand was covered separately in our report on Nemotron and Poland’s e-Health system. ElevenLabs adds that it is adding 15 MW of Vera Rubin compute capacity.

Memory and routing target the cost of a call minute

The features read differently against ElevenLabs’ price list. According to our pricing guide, every paid ElevenAgents plan works out to $0.08 per platform minute, and language model usage and telephony are billed separately at cost. That makes the model behind each turn a real line item for customers. Interaction Engine’s routing of routine turns to cheaper models goes straight at that part of the bill, though ElevenLabs gave no savings figure and the claim is untested.

Fine-tuning has a similar logic. Our earlier coverage of Scribe v3 Live reported ElevenLabs’ own figure of a 43% drop in semantic word error rate against Scribe v2 Realtime. Tuning on a company’s own calls is meant to push accuracy further for its vocabulary and callers. A customer example from the company’s enterprise post points the same way: KPN says fine-tuning on Dutch data lifted identity-verification accuracy from about 50% to 90%. That is a customer result reported by ElevenLabs and not independently checked.

For buyers the practical gap is that Memory and Routines will add capability, not obviously cheaper minutes, and none of it comes with pricing. Storing recollections of customers also raises data-handling questions the announcement does not address, beyond saying the customer must be authenticated. Retention periods, deletion controls and regional availability are not described. Most quotes and figures come from ElevenLabs itself, and no independent testing was reported.

The Marketplace and Partner Program are the part with the clearest commercial direction: the company wants consultancies and software vendors to carry deployments, and wants customers to spend existing commitments on them rather than sign new contracts.