Sound and Speech

It is August 2020. Video generation research is deep in a lineage of silent models — VGAN, MoCoGAN, DVD-GAN — none of which produce a single sample of sound, because none of them were built to. A separate, narrower research problem, matching a mouth’s movement to a soundtrack it was never recorded with, gets solved that month with a trick that has nothing to do with generating video at all: train an expert that already knows what synced lips look like, and let it grade a generator’s homework.

Where We Were

By 2020, generative video and generative speech had been developing on tracks that never crossed. The video side, covered from its GAN-era beginnings in GAN Video, had spent four years teaching models to produce plausible motion — VGAN’s two-stream foreground-background generator in 2016, MoCoGAN’s decomposition of motion from content in 2017, DVD-GAN’s push to more complex, longer clips in 2019 — and every one of those models trained on video alone, with no audio channel in the loss, the architecture, or the evaluation. Diffusion’s arrival in video generation the following two years, from Video Diffusion Models (VDM) through Make-A-Video and Imagen Video, changed the generative mechanism but not this fact: a diffusion U-Net denoising spacetime latents had no more native notion of sound than a GAN did.

A second, older problem sat one step removed from generation entirely: dubbing. Given an existing video of a person talking and a different audio track — a translated line, a re-recorded take, a synthetic voice — how do you make the mouth on screen match the new words rather than the old ones? This is a much narrower task than generating a video from nothing: the identity, the background, and the general performance are already fixed by the source footage; only the region around the mouth needs to change, frame by frame, to track a new waveform. Prior GAN-based approaches to this problem trained a generator and a discriminator jointly from scratch, and the results were characteristic of joint GAN training on a subtle target: visually plausible mouths that were only loosely timed to the audio. The paper that became Wav2Lip’s baseline, LipGAN, is representative — its own discriminator, trained end-to-end alongside the generator it was supposed to police, caught out-of-sync frame pairs only about half the time, worse than a coin flip on a well-posed binary task.

Silent-video and generative-diffusion progress ran for years without anyone closing the audio gap. PNG ↓

What both threads were missing, in different ways, was a signal strong enough to actually enforce synchronization or produce sound at all, rather than a signal that merely correlated with it on average. Wav2Lip’s answer, arriving in August 2020, addressed the narrower dubbing problem directly and left the broader question — making a generated video make its own sound — unresolved for another four years.

The Breakthrough

Wav2Lip, from a Lip Sync Expert Is All You Need for Speech to Lip Generation In the Wild (Prajwal et al., 2020), reframes the dubbing problem around a single design decision: stop training the discriminator that judges lip-sync jointly with the generator, and instead pretrain it first, separately, until it is a genuine expert at the one job of telling synced audio-visual pairs from unsynced ones — then freeze it and use it purely as a critic. The generator itself is a fairly ordinary encoder-decoder: an identity encoder processes a reference face crop and a pose prior, a speech encoder processes the target audio, and a face decoder combines both to synthesize the mouth region frame by frame. What is not ordinary is what grades the output. The pretrained “expert,” built on the SyncNet architecture but deepened and switched from grayscale to color input, watches five consecutive generated frames alongside the corresponding audio window and outputs a sync score. Isolated from the generator’s training dynamics, this expert reaches 91% accuracy at spotting an out-of-sync pair in evaluation, against roughly 56% for the jointly trained discriminator in LipGAN, the prior approach — the gap between an expert critic and a critic learning the target and the criticism at the same time.

Two encoders feed one decoder, and the output is judged twice: once for timing by a critic that never trains, once for realism by an ordinary discriminator. PNG ↓

Three signals train the generator: a pixel-level L1 reconstruction loss against the ground-truth frame, a synchronization loss read off the frozen expert’s score, and an adversarial loss from a separate, ordinary visual-quality discriminator that polices realism rather than timing. Wav2Lip’s own benchmarks — introduced in the same paper because no adequate ones existed — report the result on LRS2, LRW, and LRS3: two new metrics, LSE-D (a distance, lower is better) and LSE-C (a confidence, higher is better), both computed from the same expert’s embeddings, put Wav2Lip in the roughly 6.4–6.7 / 7.5–7.9 range across the three benchmarks versus roughly 10.0–10.7 / 3.2–3.4 for LipGAN — close enough to the range measured on genuinely synced ground-truth video that the paper characterizes its own results as “almost as good as real synced videos.” On a real-world evaluation set the paper built for dubbed, randomly-paired, and text-to-speech-driven video, human raters preferred Wav2Lip’s output over prior methods and over the unsynced original more than 90% of the time.

Wav2Lip solved dubbing — retiming an existing performance to new words — without touching the separate question of where a video’s soundtrack comes from when there is no separate audio track to synchronize against at all, because the video itself was generated. That question stayed open through Sora’s silent spacetime patches (Sora and Spacetime Patches) and stayed open through the open-weights video models that followed it. Meta’s Movie Gen, disclosed in October 2024 as Movie Gen: A Cast of Media Foundation Models (Polyak et al., 2024), closes it for the first time inside a single, disclosed foundation-model suite: alongside a 30-billion-parameter text-to-video model, the suite includes Movie Gen Audio, a 13-billion-parameter model trained to generate synchronized sound effects and music for a given video, or a soundtrack from a text prompt alone, at 48kHz.

Movie Gen Audio treats a video and a text prompt as interchangeable conditioning signals into one diffusion transformer. PNG ↓

Movie Gen Audio is a diffusion transformer trained with a flow-matching objective — the transformer backbone and the flow-matching training target both taught in full in Flow Matching and Transformer Backbones — conditioned on either an encoded video (for video-to-audio generation) or an encoded text prompt (for text-to-audio generation), and able to extend a generated soundtrack past the length it was trained on to cover multi-minute videos. Meta pretrains it on roughly a million hours of audio before fine-tuning on a smaller, curated set targeting cinematic sound design, and reports it outperforming commercial systems, including Pika Labs and ElevenLabs, on sound-effect generation in the paper’s evaluations. The paper’s own framing of what the model has learned is telling: not just the physical association between an on-screen action and the sound it would make, but the psychological or mood-setting association a human sound designer adds on top — non-diegetic score that was never physically present in the scene at all.

The Mathematics

Wav2Lip’s total generator loss combines three terms, weighted to keep the model from overcorrecting on timing at the expense of image quality:

\mathcal{L}_{\text{total}} = (1 - \lambda_{\text{sync}} - \lambda_{\text{gan}})\,\mathcal{L}_{L1} + \lambda_{\text{sync}}\,\mathcal{L}_{\text{sync}} + \lambda_{\text{gan}}\,\mathcal{L}_{\text{gen}}, \qquad \lambda_{\text{sync}} = 0.03,\; \lambda_{\text{gan}} = 0.07

$\mathcal{L}_{L1}$ is a plain pixel-wise reconstruction loss between the generated frame and the ground-truth frame — the dominant term at a weight of 0.90, because most of a talking-face frame (background, hair, everything but the mouth) should simply match the reference. $\mathcal{L}_{\text{sync}}$ is the synchronization signal read off the frozen expert, and $\mathcal{L}_{\text{gen}}$ is an ordinary adversarial loss from a separate discriminator judging visual realism — the standard adversarial-training setup taught in Generative Adversarial Networks, applied here only to image quality, not timing. The three weights sum to 1, and the two auxiliary terms get small shares, 0.03 and 0.07, because either one dominating the L1 term would trade a sharp, correctly-timed face for a blurrier or more artifact-prone one; the small weights are what let the sync signal steer the mouth without wrecking everything around it.

Symbol	Meaning
$\mathcal{L}_{\text{total}}$	the full training loss for the generator
$\mathcal{L}_{L1}$	pixel-wise reconstruction loss against the ground-truth frame
$\mathcal{L}_{\text{sync}}$	synchronization loss, read from the frozen pretrained expert
$\mathcal{L}_{\text{gen}}$	adversarial loss from a separate visual-quality discriminator
$\lambda_{\text{sync}}, \lambda_{\text{gan}}$	fixed weights (0.03, 0.07); $\mathcal{L}_{L1}$ takes the remaining $1 - \lambda_{\text{sync}} - \lambda_{\text{gan}} = 0.90$

The sync loss itself is built from the expert’s own scoring function — a cosine similarity between the expert’s audio embedding and its video embedding for the same five-frame window, turned into a probability of being in sync:

P_{\text{sync}} = \frac{v \cdot a}{\lVert v \rVert_2 \, \lVert a \rVert_2}, \qquad \mathcal{L}_{\text{sync}} = -\log\big(P_{\text{sync}}\big)

Symbol	Meaning
$v$	the expert’s embedding of the five-frame video window
$a$	the expert’s embedding of the corresponding audio window
$P_{\text{sync}}$	cosine similarity between $v$ and $a$ , read as a sync probability
$\mathcal{L}_{\text{sync}}$	the sync loss fed into the total loss above, $-\log(P_{\text{sync}})$

Movie Gen Audio reuses the conditional flow-matching objective that Flow Matching and Transformer Backbones derives in full, with the conditioning vector swapped for one built from encoded video or text rather than text alone:

\mathcal{L}_{\text{CFM}} = \mathbb{E}_{t,\, x_0,\, x_1}\Big[\big\lVert v_\theta(x_t, t, c) - (x_1 - x_0) \big\rVert_2^{2}\Big]

$x_0$ is a sample of noise and $x_1$ is the clean target — here, a latent representation of the audio waveform rather than an image or video latent — and $x_t$ is a point on the straight-line path between them at time $t$ . $c$ is the conditioning vector, built either from an encoded video (for video-to-audio) or an encoded text prompt (for text-to-audio). $v_\theta$ , the diffusion transformer, is trained to predict the constant velocity $x_1 - x_0$ along that path; sampling then integrates the predicted velocity field from noise to a clean audio latent. Nothing about the objective itself is new — it is the same flow-matching target used for images and video — the news is which modality $x_1$ now lives in, and that $c$ can be a video rather than only a caption.

Symbol	Meaning
$x_0, x_1$	the noise sample and the clean audio latent, the two endpoints of the flow path
$x_t$	a point on the straight-line path between them at time $t$
$c$	the conditioning vector, from an encoded video or an encoded text prompt
$v_\theta$	the diffusion transformer, predicting velocity along the path
$t$	the position along the noise-to-data path, in $[0,1]$

Both systems in this chapter are bound by a latency budget that most of this book’s other chapters can ignore. Wav2Lip’s generator produces a talking-face video frame by frame, which is exactly the shape of workload live dubbing and real-time translation need, not just the offline batch case the paper evaluates; Movie Gen Audio’s ability to extend a soundtrack past its trained length implies chunked, sequential generation with the same constraint. Serving either use case live, rather than as a batch job, is the same inference-latency problem Inference Engines covers for text generation — how much compute and memory a serving stack can spend per output unit before the output arrives too late to be useful — now applied to video frames and audio chunks instead of tokens.

What It Enabled

The two breakthroughs in this chapter did not converge into a single system; they solved adjacent problems four years apart, and both outlived the specific papers that introduced them.

	Prior dubbing approach, jointly-trained GAN	Wav2Lip, 2020
Discriminator training	Jointly with the generator, from scratch	Pretrained separately, then frozen, as an expert critic
Out-of-sync detection accuracy	~56% (LipGAN)	91%
LSE-D / LSE-C (lower / higher is better)	~10.0–10.7 / ~3.2–3.4	~6.4–6.7 / ~7.5–7.9
Human preference vs. prior methods (ReSyncED)	—	over 90%

	Silent video generation, 2016–2023	Movie Gen, 2024
Audio channel	None — VGAN through Sora generate picture only	Movie Gen Audio: 13B-parameter video-conditioned or text-conditioned model
Sound source	External: separately recorded, licensed, or dubbed after the fact	Generated jointly by the same disclosed foundation-model suite as the video
Training data (audio)	—	~1M hours pretraining, fine-tuned on curated cinematic audio
Public release	Stable Video Diffusion open weights, Nov 2023	Research disclosure only; no public weights or API for Movie Gen Audio

Wav2Lip’s pretrained-expert idea outlived the specific lip-sync task it was built for: freezing a strong, separately-trained critic and using it purely as a differentiable evaluation signal, rather than training a critic jointly and hoping it stays sharp, is now a standard move anywhere a generator needs a hard-to-hand-craft loss — the same shape of idea, applied to a different critic, that underlies the video-to-audio evaluation in Movie Gen’s own paper. Movie Gen Audio, in turn, did not ship as a public product, but it is the clearest disclosed precedent for the next step the field actually shipped: Google’s Veo 3, announced May 2025 with native audio generated jointly with video rather than added afterward by a separate model (The Commercial Race). Whether that native-audio approach or a Movie-Gen-style separate-but-conditioned audio model becomes the durable pattern is still being decided in the same commercial race, and either approach still faces the same cost pressure toward fewer, cheaper sampling steps taken up directly in Making Video Cheap.

Wav2Lip's frozen critic becomes a dubbing bolt-on; Movie Gen Audio is the clearest precedent for audio generated inside the video model itself. PNG ↓

Key Papers

Paper	Authors	Year	Link	Why it matters
A Lip Sync Expert Is All You Need for Speech to Lip Generation In the Wild	Prajwal et al.	2020	arXiv:2008.10010	Replaced jointly-trained discriminators with a separately pretrained, frozen sync expert, solving audio-driven dubbing with near-real-sync accuracy
Movie Gen: A Cast of Media Foundation Models	Polyak et al.	2024	arXiv:2410.13720	Disclosed a 13B video- and text-conditioned diffusion-transformer audio model alongside a 30B video model, the first foundation-model suite to generate sound for video rather than treat sound as a separate track

PM Lens

The build-versus-wait question in this chapter is concrete today. Wav2Lip’s pretrained-expert trick is old enough that it now sits quietly inside commercial dubbing pipelines; ElevenLabs prices its Dubbing Studio API at $0.50 per minute of video, a bolt-on cost anyone can add to an existing video pipeline this quarter. Movie Gen Audio never shipped as a product — Meta disclosed it as research, with no public weights or API — which is the pattern to watch for before scoping a roadmap around a technical report rather than a priced endpoint. What did ship into a real product is Veo 3’s native audio, sound generated inside the same model as the video rather than bolted on afterward. For a PM deciding how to add sound to a video feature, the honest options right now are the same as they were the moment Wav2Lip shipped: license or dub with a separately priced audio model on top of whatever generates the picture, or wait for the native-audio video model to be good enough and cheap enough for your use case — and that second option’s price and quality bar move every few months, not every few years.

Search