AI dubbing: How software turns one video into many languages, and where humans still step in

AI dubbing replaces the spoken audio in a video with speech in another language, using software instead of a recording studio. The better systems also try to keep the original speaker’s voice, tone and timing, so the translated version still sounds like the same person talking.

ElevenLabs describes it as automatically replacing a video’s original voice track with audio in a different language. Its newest enterprise offer builds on this technology, as covered in our story on Forward Deployed Creatives.

One ad, ten markets, no recording booth

Picture a 60-second product ad made in English. A brand wants it in Spanish, Japanese and Brazilian Portuguese. The traditional route means hiring translators, voice actors and a studio, then waiting. ElevenLabs says studio dubbing can take weeks or months, while an automated pipeline can return a localized version in hours.

Subtitles are the cheaper alternative, but viewers must read them while watching. Dubbing lets people simply watch. ElevenLabs suggests using both: dubbing for the main experience and subtitles for those who prefer reading.

Five steps between the original and the dubbed track

ElevenLabs lays out the process in five stages. Most dubbing tools follow a similar order.

1. Transcription and analysis. Speech-to-text turns the audio into a script. The system also notes who is speaking, when, and how.
2. Translation. A language model translates the script. The goal is to keep intent and style, and to adapt idioms rather than translate them word for word.
3. Voice cloning (optional). The system recreates the original speaker’s voice in the new language, or uses a stock voice instead.
4. Speech generation. A text-to-speech model reads the translated script aloud in the chosen voice.
5. Synchronization. Timing and pacing are adjusted so the new audio fits what appears on screen.

The original background audio is kept, according to ElevenLabs’ documentation, so music and effects do not need to be mixed again.

Telling speakers apart and sounding like them

Two steps carry most of the difficulty. The first is working out who says what. ElevenLabs’ docs say its dubbing detects multiple speakers automatically, including overlapping speech, with up to 32 speakers per file. Each speaker then gets a separate voice in the output.

The second is deciding how closely to copy the voice. ElevenLabs’ Dubbing v2 exposes a setting called cloning strength, from 0 to 10, with a default of 7. Higher values stay closer to the original voice but can sound less natural across very different languages and may carry over the original accent. Lower values sound more natural in the target language but resemble the speaker less.

That trade-off shows up in every system. A voice that is a perfect match in English may be an awkward fit for the rhythm of another language. Timing adds a second tension: a sentence can take more or fewer syllables in another language, so the system must either speed up, slow down or reword the line to fit the scene.

Where the idea came from

Machine speech translation predates the current wave. Meta’s Seamless Communication suite, announced in 2023, included SeamlessExpressive, a model meant to preserve features such as speech rate, pauses, emotion and style across languages. Meta said it covered English, Spanish, German, French, Italian and Chinese, and that outputs from its expressive models carry a watermark.

ElevenLabs launched a voice translation and dubbing tool in October 2023, according to Voicebot’s coverage at the time. Today its docs list 90+ languages, with regional dialect tags such as fr-CA, es-MX and pt-BR in Dubbing v2.

Limits that keep people in the loop

Output quality depends on the underlying models. ElevenLabs notes that weaker models can sound robotic or miss the emotional weight of a scene, and that human dubbing remains stronger when a project needs many script edits or a changed feel.

Practical limits also apply. ElevenLabs’ docs say live dubbing is not available and that the automatic output is a preview rather than a production master. They recommend replacing the audio track on the original video for final delivery. They also list Dubbing v2 as alpha, so rough edges are expected.

Rights matter too. ElevenLabs says dubbed content is generally treated as derivative work, so organizations should confirm they hold the necessary rights. For high-risk voices such as public figures, it says cloning is blocked without explicit permission.

Why a human layer is being added on top

The new service is a response to those limits. ElevenLabs’ Productions dubbing page describes AI models paired with human review. A native speaker checks translation accuracy and edits the voice-over so it sounds natural. The team picks the voice, adjusts pacing and runs each dub through a quality-control checklist. Projects are targeted for delivery within 7 business days, with pricing quoted case by case.

This mirrors the Extend tier of Forward Deployed Creatives, where, per ElevenLabs, trained linguists and audio professionals review localized output so it reads and sounds native in each market.

What to watch: live dubbing, which today is not offered by ElevenLabs and remains an active research area; tighter lip sync; and clearer rules on consent for cloned voices.