Speech-to-speech models: How voice AI listens and talks without turning speech into text

A speech-to-speech model is a single AI model that takes in spoken audio and produces spoken audio as its answer. It does not first convert your voice into written text and then convert a written reply back into a voice. Because it skips that detour, it can respond faster and react to how you sound, not only to what you said.

Picture calling a support line. You start a sentence, pause to think, then say “actually, wait” halfway through the agent’s reply. A human handles all of that without effort. Voice AI has to learn it, and the design of the model decides how well it can.

The three-step relay that voice assistants used to run

Most voice assistants were built as a relay race with three runners. Speech-to-text (also called speech recognition) turns your audio into words. A language model reads those words and writes a reply. Text-to-speech then reads the reply aloud. A fourth component, voice activity detection, decides when you have stopped talking.

OpenAI’s developer documentation calls this a chained pipeline. Its stated advantage is control: you can inspect or change the text between stages, store transcripts, run policy checks before the agent answers, and swap any component independently.

The cost is time and information. Each stage waits for the one before it, and the text in the middle holds only the words. Tone, hesitation, laughter and background sounds are gone by the time the language model sees anything.

One model that hears and speaks audio directly

A speech-to-speech model removes the handoffs. Audio goes in, the model reasons over it, and audio comes out, all inside one system. OpenAI’s documentation describes its Realtime API this way: the model works directly with audio, keeps the conversation state, and can call tools.

To make this possible, audio is chopped into small pieces and encoded as tokens, much like words are tokens for a text model. The model predicts the next audio tokens the way a chatbot predicts the next word. Many designs also keep text in the loop internally, so the model can reason in words while it speaks.

The gains are mainly speed and naturalness. The reply can start while the model is still working, and the model can in principle hear emotion or noise that a transcript would erase.

Listening while talking: The full-duplex idea

Speed is only half of a natural conversation. People also overlap. They say “mm-hm” while the other person talks, interrupt, and pause without giving up the floor. A system that can listen and speak at the same time is called full-duplex, a term borrowed from telephone engineering.

A landmark here is Moshi, published in 2024 by Alexandre Défossez, Laurent Mazaré and colleagues, with code from the Kyutai lab. The authors describe it as a speech-text foundation model that models its own voice and the user’s voice as two parallel streams. That removes the need for explicit speaker turns. They report a theoretical latency of 160 milliseconds and about 200 milliseconds in practice.

Moshi also introduced what the authors call an inner monologue: the model predicts text tokens aligned in time before it generates the matching audio, which they say improves the linguistic quality of the speech.

Who uses it today, and how it is measured

Speech-to-speech is now the main battleground for voice agents. OpenAI offers it through its Realtime API and points to GPT-Live for full-duplex conversations. xAI released Grok Voice 3.0 in October 2026, which ranked first on Artificial Analysis’s Speech-to-Speech Index with a score of 86.9, according to xAI. The company lists a sub-one-second response time and a price of $0.08 per minute.

Measuring these systems is hard, because a good conversation is more than correct answers. Full-Duplex-Bench, introduced in 2025 by Guan-Ting Lin, Hung-yi Lee and co-authors, tests four behaviors with automatic metrics: pause handling, backchanneling, turn-taking and interruption management. Artificial Analysis’s index, as described by xAI, mixes speech reasoning, a customer-support benchmark called τ-Voice, and blind head-to-head calls rated by people.

Most of these scores come from the vendors’ own announcements. Treat them as a guide to what each company optimizes, not as a neutral final verdict.

What you give up, and what to watch

The main trade-off is visibility. In a chained pipeline, every step produces text you can log, audit and filter. In a single audio model, that intermediate text may not exist, or may be only a side product. For regulated settings such as banking or health, that matters. OpenAI’s guide recommends the chained route when each stage needs to be visible or replaceable.

Other limits follow from the design. You are tied to one vendor’s model for ears, brain and voice together. Pricing by the minute, as xAI does, is hard to compare with per-token pricing elsewhere without measuring real call lengths. And a model that responds instantly can still be wrong, so low latency does not by itself mean a better answer.

Three things are worth watching:

• Full-duplex as the default. Interruptions and backchannels are becoming a standard test, not an extra.
• Auditability. Expect tools that give back transcripts and logs from audio-native models.
• Independent testing. Third-party benchmarks such as ServiceNow’s EVA-Bench will show whether vendor rankings hold up on real enterprise calls.