AI lip-sync: How a model redraws a speaker’s mouth to match new audio
AI lip-sync regenerates the mouth area in a video so it matches a different audio track. Here is how Wav2Lip-style models do it, who uses them, and where they struggle.

AI lip-sync is a technique that redraws the mouth of a person in a video so the lips move in time with a different audio track. The audio can be a translation, a rewritten line or a new voiceover. The rest of the footage, including the face, the background and the head movement, stays as filmed.
It is the step that makes a dubbed or re-scripted video look as if the speaker really said those words.
Why a dubbed ad looks wrong without it
Picture a 20-second ad in which a woman says an English sentence to the camera. A studio translates it into German. The German sentence is longer, with different sounds, so her lips still form the English shapes while the German audio plays. Viewers notice the mismatch within a second, even if they cannot say why.
Lip-sync models fix this. They take the new audio and the original video, and they regenerate the lower part of the face frame by frame so the mouth opens, closes and rounds at the right moments.
Masking the mouth and repainting it from sound
The best-known recipe comes from a 2020 paper by K R Prajwal, Rudrabha Mukhopadhyay, Vinay Namboodiri and C V Jawahar, presented at ACM Multimedia 2020. It introduced a model called Wav2Lip.
It works in three parts:
- An identity encoder looks at the current video frame, plus a copy of that frame with the lower half of the face masked out.
- A speech encoder turns a short slice of the audio into a compact description of the sounds.
- A face decoder combines the two and paints the missing lower half of the face.
Think of a sign painter who is handed a poster with the mouth covered and a recording to listen to. The painter fills in only the covered area, so the eyes, hair and lighting remain untouched. In the paper, the head pose is kept fixed from the input video, which the authors say reduces visible artifacts.
A frozen judge that knows what in-sync looks like
The paper’s main idea is how the model is trained. Earlier systems mostly compared the generated face with the real one pixel by pixel. The authors found this a weak signal for lips: by their measurement, the mouth region contributes less than 4% of that reconstruction loss, so a model can score well while getting the mouth wrong.
Their fix is a separate network, trained beforehand, whose only job is to tell whether a short stretch of mouth video matches a short stretch of audio. It is a modified version of an earlier model called SyncNet and was trained on LRS2, a dataset of broadcast-TV speech clips (about 29 hours in the training split). The paper reports it is 91% accurate on mismatched pairs from that dataset, against about 56% for the discriminator used in the earlier LipGAN model.
During generator training this judge is frozen. It acts as a fixed referee: if the generated mouth does not match the audio, the generator is penalized. Because the judge cannot be fooled by retraining, the generator has to produce believable synced lips.
Sharpening the picture without losing the sync
A strong sync penalty can leave the mouth slightly blurry. The authors therefore add a second, simpler network, a visual quality discriminator, in the usual generative adversarial network (GAN) setup. It judges whether the generated face looks like a real photo. In the paper’s loss, the sync term has a weight of 0.03 and the quality term 0.07; the rest goes to the pixel comparison.
The authors report that this version looks sharper but costs a small drop in sync accuracy. On LRS2, they measured a lip-sync error distance (LSE-D, lower is better) of 6.386 for Wav2Lip, 10.33 for LipGAN and 6.736 for real videos. The paper concludes that its synced videos are almost as good as real ones on those metrics. It also reports that human evaluators preferred its results over existing methods and unsynced video more than 90% of the time.
The authors released the code and models, which is why many later lip-sync tools build on this approach or compare themselves against it.
Where the technique shows up now
Lip-sync is now a standard final stage in AI dubbing and ad localization. In our report on ElevenLabs’ Ads Engine, the company says the speaker is lip-synced to a new hook or to dubbed audio so the ad looks recorded in that language. ElevenLabs’ own launch post for Ads Engine describes dubbing that reproduces the original speaker’s tone, emotion and pacing in each target language. It does not describe how the mouth is handled, so the technical details of its lip-sync are not public in the sources we read.
The same step appears in film dubbing tools, video translation apps and avatar products. The pipeline is usually the same: transcribe, translate, generate speech, then repaint the mouth to fit.
Limits that still show up on screen
The 2020 paper is open about its boundaries. It lists generating expressions and head poses along with the lips as future work, and it notes room for improvement when the audio comes from text-to-speech rather than a human. Because the pose is copied from the source, a translated sentence that is much longer than the original cannot be fixed by lip-sync alone; the dub has to be fitted to the available time first.
Other practical issues are common in this kind of system:
- Teeth, tongue and fast side-on movement are hard to render convincingly.
- Small faces or low-resolution footage give the model little to work with.
- Only the lower face is changed, so an unchanged expression can feel slightly off with an emotional new line.
Human review is therefore still part of the workflow. ElevenLabs describes an optional approval step before ads go live.
What to watch next
Research since 2020 aims at what the original paper left out: whole-face expression, head motion that follows the new speech, and higher resolution. Watch also for how platforms label edited video. A lip-synced clip can show a real person saying words they never said, which is why consent for voice and likeness is part of the same conversation. Brands that use these tools in ads will increasingly be asked what was changed and who approved it.
Sign Up for the Daily AI Pulse
One email a day. All the stories that matter.
By signing up, you agree to our Terms of Use and acknowledge the data practices in our Privacy Policy. You may unsubscribe at any time.
Daniel ReedAI Research, Safety & Ethics AnalystDaniel Reed currently works as an AI Research, Safety & Ethics Analyst at Aiholics, writing about how changes in artificial intelligence are affecting and will affect scholarship,…



