Making Video Cheap

It is late 2024. Sora has just shown that a diffusion transformer can produce a minute of coherent video (Chapter 9), and Stable Video Diffusion has put an open video model in anyone’s hands (Chapter 6) — but every one of these models still generates a clip the way a print shop fills an order: submit the whole request, wait, collect the finished reel. Over the following year, three papers attack the cost of that wait from three different angles, and by mid-2025 a video model can stream frames as fast as a viewer watches them.

Where We Were

Video diffusion models inherited image diffusion’s step-count problem and made it worse. A single image sample from a guided latent diffusion model already meant tens of sequential network calls, each one doubled by classifier-free guidance (Chapter 10); video multiplies that per-step cost by the extra temporal dimension a video transformer has to attend over, and multiplies the wait again because a clip is many frames, not one. The step-distillation methods that solved this for images — progressive distillation, consistency models, Latent Consistency Models (LCM), adversarial diffusion distillation — collapsed tens of guided steps into as few as one for a single frame (Chapter 18), and by late 2023 LCM had already shown that the same consistency-distillation trick worked directly on Stable Diffusion’s latent space (Luo et al., 2023). Porting that trick to video was not simply a matter of re-running the same recipe on a bigger tensor.

Two obstacles were specific to video. First, the video diffusion transformers built on top of spacetime patches (Chapter 9) and temporal attention layers (Chapter 6) are bidirectional: to denoise frame 40, the network attends over frames 1 through 80, so it cannot start emitting frame 1 until it has already committed to what the whole clip will contain. That is workable for a batch job that returns a finished file, but it rules out streaming — there is no way to hand a viewer frame 1 while the model is still deciding on frame 80. Second, even a distilled few-step video model still pays that few-step cost once per clip, and a longer clip, or an interactive session where a user’s input should change what happens next, needs the model to keep generating indefinitely rather than stopping after one fixed-length sample.

By the second half of 2024, a state-of-the-art bidirectional video diffusion model took on the order of a couple of minutes on a single high-end GPU to produce ten seconds of video — the CausVid paper’s own bidirectional teacher measured 219.2 seconds for a 10-second clip on one GPU (Yin et al., 2024) — a wait that is fine for a render queue and unworkable for anything a user expects to talk to.

Step distillation for images is six years old by the time this chapter's three routes finally point it at video, and each one removes a different part of the wait Sora still has. PNG ↓

The Breakthrough

LTX-Video: making the architecture itself cheap. Rather than distilling an existing model, the Lightricks team redesigned the video autoencoder and transformer together so that the expensive part of the pipeline — the per-step transformer forward pass — operates on far fewer tokens per second of video. LTX-Video’s video variational autoencoder (VAE) compresses each clip by a factor of 1:192, folding a 32×32×8 pixel block into a single latent token, and moves the patchifying step that most video diffusion transformers perform on the transformer’s input into the VAE itself, which lets the transformer run full spacetime self-attention over a much smaller token grid (HaCohen et al., 2024). Because the VAE decoder also absorbs the last stage of denoising rather than handing off a fully clean latent, the paper reports generating five seconds of 24fps video at 768×512 resolution in two seconds on a single H100 GPU — faster than the clip’s own playback time, without any step-distillation training at all. The result shows that “cheap” and “few-step” are not synonyms: an aggressively compressed latent space, in the tradition of latent diffusion generally (Chapter 13), buys real-time throughput on its own.

CausVid: turning the bidirectional teacher into a causal student. Tianwei Yin and colleagues at Adobe and MIT took the opposite route: keep an existing large bidirectional video diffusion model as a frozen teacher, and train a second, causal model to reproduce it frame-by-frame instead of clip-by-clip (Yin et al., 2024). A causal model generates frame $n$ using only frames before it, the same autoregressive dependency structure that language models and earlier autoregressive image models use (Chapter 5), which means it can start emitting output before it has “seen” the rest of the clip — and, paired with a key-value (KV) cache so each new chunk of frames does not have to re-attend over every previous frame from scratch (Engine Room, Chapter 17), it can stream indefinitely rather than stopping at a fixed clip length. The training recipe has two stages: first the causal student is initialized by regressing directly onto trajectories the bidirectional teacher already traced out with an ODE solver (Chapter 9, Score-SDEs); then an asymmetric distribution-matching loss refines it further, asymmetric because the teacher stays bidirectional and full-context while the student stays causal and chunked. The paper reports the resulting 4-step causal student generating a 10-second clip in 1.3 seconds on one GPU — sustaining 9.4 frames per second of streaming output — a roughly 170x wall-clock improvement over the 219.2-second bidirectional teacher, while scoring 84.27 on the VBench-Long benchmark, ahead of the prior state of the art at the time.

Two signals point the causal student the same way: a warm start copied from the teacher's own solved trajectories, then a live loss comparing the teacher's judgement of realism against the student's. PNG ↓

Seaweed-APT: adversarial post-training for a single step. A third route, from ByteDance’s Seaweed team, skips distillation-from-a-teacher’s-trajectory entirely and instead continues training an already-pretrained video diffusion model with an adversarial objective, in the spirit of adversarial diffusion distillation for images (Chapter 18) but built for video from the start (Lin et al., 2025). The discriminator reuses the pretrained diffusion transformer’s weights directly as its starting point; the generator instead starts from a consistency-distilled checkpoint of that same pretrained model — a fast but blurry one-step version that the paper reports as an effective initialization for the adversarial stage that follows. From there, the discriminator is trained to tell real clips apart from the generator’s single forward-pass output, while an approximated R1 regularization term keeps that adversarial signal from destabilizing training — the same instability problem that limited earlier GAN-based image and video generators (Chapter 2). The resulting model produces a 2-second, 1280×720, 24fps video in a single forward pass: 6.03 seconds on one H100 GPU, or close to real time — about 2 seconds — when the forward pass is split across 8 H100 GPUs. A follow-up from the same group extends the recipe to autoregressive, chunk-by-chunk generation, so the single-step generator does not stop after one clip but keeps producing new latent frames conditioned on a viewer’s ongoing input, streaming 24fps video continuously — 736×416 on a single H100, or 1280×720 across 8 H100 GPUs for clips up to about a minute long (Lin et al., 2025).

No fixed-length clip has to finish before the viewer sees a frame: the same read, emit, write loop just repeats for as long as the viewer keeps watching. PNG ↓

The Mathematics

CausVid restricts the video transformer’s attention to a block-causal pattern: frames are grouped into chunks of size $k$ , and a frame in chunk $i$ may attend only to frames in chunks up to and including $i$ :

M_{i,j} = \begin{cases} 1 & \text{if } \lfloor j / k \rfloor \le \lfloor i / k \rfloor \\ 0 & \text{otherwise} \end{cases}

Symbol	Meaning
$M_{i,j}$	attention mask entry, whether frame $i$ can attend to frame $j$
$i, j$	frame indices
$k$	chunk size, frames processed together

The causal student is first initialized by regressing directly onto the teacher’s own ODE-solved trajectories, over $N$ chunks with per-chunk timesteps $t^i$ :

\mathcal{L}_{\mathrm{init}} = \mathbb{E}_{x, t^i} \left\| G_\phi\!\left(\{x^i_{t^i}\}_{i=1}^N, \{t^i\}_{i=1}^N\right) - \{x^i_0\}_{i=1}^N \right\|^2

The second stage refines the student with an asymmetric distribution-matching loss, whose gradient compares the teacher’s score against a second, continuously updated score network trained on the student’s own outputs:

\nabla_\phi \mathcal{L}_{\mathrm{DMD}} \approx -\,\mathbb{E}_t \left[ \int \big(s_{\mathrm{data}}(x_t, t) - s_{\mathrm{gen}}(x_t, t)\big)\, \frac{\partial G_\phi(\epsilon)}{\partial \phi}\, d\epsilon \right], \qquad x_t = \alpha_t\, G_\phi(\epsilon) + \sigma_t\, \epsilon

$s_{\mathrm{data}}$ is the frozen bidirectional teacher’s score function — the direction that pushes a noisy sample back toward real data (Chapter 8); $s_{\mathrm{gen}}$ is a second score network, updated throughout training, that instead learns the score of whatever distribution the causal student $G_\phi$ currently produces. $\epsilon$ is the student’s input noise, and $\alpha_t, \sigma_t$ are the usual signal- and noise-retention terms of the forward diffusion process (Chapter 8). The gradient pushes the student’s output distribution to close the gap between what the teacher’s score says real data looks like and what the student’s own current output looks like. What this buys you: the student is trained against the teacher’s judgment of realism rather than against a fixed set of teacher samples, which is what lets a bidirectional teacher supervise a structurally different, causal student — the general step-distillation idea is the same one that produces few-step image models (Chapter 18), adapted here to cross the bidirectional-to-causal gap rather than only to cross a many-step-to-few-step gap.

Symbol	Meaning
$s_{\mathrm{data}}, s_{\mathrm{gen}}$	teacher’s frozen score function, and the score of the student’s current output distribution
$G_\phi$	the causal student generator
$\epsilon$	input noise to the student
$\alpha_t, \sigma_t$	signal- and noise-retention terms at diffusion timestep $t$
$x_t$	the noised version of the student’s output at timestep $t$

Seaweed-APT’s discriminator combines a standard adversarial term with an approximated R1 regularizer, computed by comparing the discriminator’s judgment of a real clip to its judgment of that same clip after a small amount of Gaussian noise is added:

\mathcal{L}_D = \mathbb{E}\big[f_D(D(x,c))\big] + \mathbb{E}\big[f_G(D(G(z,c),c))\big] + \lambda\, \mathbb{E}\left[\, \big\| D(x,c) - D(x + \sigma\,\eta,\, c) \big\|_2^2 \,\right]

$D$ is the discriminator, initialized directly from the pretrained diffusion transformer’s weights, and $G$ is the single-step generator, initialized instead from a consistency-distilled checkpoint of that same pretrained model; $x$ is a real clip and $z$ is the generator’s input noise, $c$ is the text or image conditioning, and $\eta$ is Gaussian noise of scale $\sigma$ . $f_D$ and $f_G$ are the standard non-saturating GAN loss functions, and $\lambda$ weights the third term, the approximated R1 penalty, which discourages the discriminator from changing its judgment sharply under a tiny perturbation of a real input. What this buys you: the stabilizing effect of the classic R1 gradient penalty — long used to keep GAN training from diverging (Chapter 4, WGAN-GP)— without computing an exact gradient penalty, which is expensive at video resolution.

Symbol	Meaning
$D, G$	discriminator (DiT-initialized) and single-step generator (initialized from a consistency-distilled checkpoint)
$x, z, c$	real clip, generator input noise, and conditioning
$\eta, \sigma$	Gaussian perturbation and its scale
$f_D, f_G$	non-saturating adversarial loss functions
$\lambda$	weight on the approximated R1 term

What It Enabled

None of these three approaches replaced the others outright; they occupy different points on a training-cost, latency, and interactivity trade-off, and the fastest-moving products through 2025 combined more than one.

Method	Year	Mechanism	Reported latency	Streaming/interactive
Bidirectional baseline (CausVid’s teacher)	2024	full-clip diffusion, tens of steps	219.2s for a 10s clip, 1 GPU	no — must finish before output
LTX-Video	2024	compressed VAE plus DiT, no distillation	5s of 768×512 video in 2s, 1 H100	no — still clip-at-a-time
CausVid	2024	asymmetric distillation to a causal, 4-step, KV-cached student	10s clip in 1.3s, 9.4 FPS streaming	yes — frame-chunk streaming
Seaweed-APT	2025	adversarial post-training, single step	2s of 1280×720 video in 6.03s on 1 H100, ~2s on 8 H100	no — single clip per call
Autoregressive APT	2025	single-step generator made causal and continuous	736×416 at 24fps on 1 H100, 1280×720 on 8 H100, up to ~1 min	yes — indefinite interactive stream

The immediate effect was to make “video generation” stop meaning only “submit a prompt, wait, download a file.” A causal, KV-cached, few-step (or single-step) generator can, in principle, keep producing new frames for as long as a viewer keeps watching, and it can accept new conditioning — a changed prompt, a camera-control signal, a player’s input — between chunks rather than only at the start of a clip. That capability is the direct prerequisite for treating a video model as something closer to a real-time simulator than a renderer: the world-model framing introduced when video models first learned to predict physical dynamics (Chapter 3) only becomes an interactive product once generation is fast enough to run in a loop with a user’s actions, which is exactly what Chapter 15 and Chapter 16 build on. It is also, separately, a distinct compression axis from shrinking a model’s parameter count — the general teacher-student framing for that is in Engine Room, Chapter 16 — and production video pipelines increasingly combine both: a smaller model, distilled to fewer steps, running on a compressed latent space.

LTX-Video is a dead end on this map, a complete answer on its own; the distillation lane keeps folding one system's output into the next, and hands the result to Chapter 15. PNG ↓

Key Papers

Paper	Authors	Year	Link	Why it matters
Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inference	Luo, Tan, Huang, Li, and Zhao	2023	arXiv:2310.04378	Established that consistency-style step distillation ports directly into guided latent diffusion, the base case video distillation extends
LTX-Video: Realtime Video Latent Diffusion	HaCohen et al.	2024	arXiv:2501.00103	Showed real-time video generation is reachable through architecture and compression alone, with no distillation step
From Slow Bidirectional to Fast Autoregressive Video Diffusion Models	Yin, Zhang, Zhang, Freeman, Durand, Shechtman, and Huang	2024	arXiv:2412.07772	Distilled a bidirectional teacher into a causal, KV-cached, 4-step streaming student — the mechanism behind CausVid
Diffusion Adversarial Post-Training for One-Step Video Generation	Lin, Xia, Ren, Yang, Xiao, and Jiang	2025	arXiv:2501.08316	Reached single-forward-pass video generation via adversarial post-training of a pretrained diffusion transformer
Autoregressive Adversarial Post-Training for Real-Time Interactive Video Generation	Lin, Yang, He, Jiang, Ren, Xia, Zhao, Xiao, and Jiang	2025	arXiv:2506.09350	Extended single-step adversarial post-training to continuous, interactive, multi-GPU streaming generation

PM Lens

This chapter is where “generate a video” starts to split into two different products with two different cost models. Batch generation is priced per second of finished output — Veo 3 on Vertex AI lists $0.50 per second without audio and $0.75 per second with it, so an 8-second clip runs several dollars regardless of how the vendor produced it. The methods in this chapter attack the vendor’s side of that equation: a causal, few-step, KV-cached model turns the same GPU-minute into many more billable seconds of video, which is margin, not something the customer sees directly. Streaming, interactive generation is a different pitch entirely — you are not selling a finished clip, you are selling continuous compute time, closer to a game server than a render farm, and I have not yet seen that priced as a shipped consumer product; Seaweed-APT2 and similar systems are still research demos. When I evaluate a video-gen vendor now, step count and causality are the first questions, because they predict which of these two businesses the vendor is actually building.

Search