Video Diffusion Models

It is April 2022. Diffusion models have spent nearly two years rewriting the state of the art in still images — matching GANs on likelihood-free sample quality in 2020, beating them outright on ImageNet in 2021, and about to power a wave of text-to-image products later this same year. Video has watched from the sidelines: GAN-based video generators, VQ-token autoregressors, and latent world models have each produced short, low-resolution, or narrowly-conditioned clips, but nothing that generalizes the way image diffusion had just learned to. Over the following six months, three papers — one from Google Brain, one from Meta, one more from Google — establish that the same forward-noise, learned-reverse-process recipe extends to the fourth dimension, and do so at three different points on the trade-off between architectural minimalism, data efficiency, and raw output resolution.

Where We Were

By early 2022, video generation had been worked from three separate angles, none of which had produced a general-purpose, high-fidelity text-to-video system. The earliest angle treated video as “images with a discriminator that also watches motion”: VGAN in September 2016 and MoCoGAN in July 2017 adapted the adversarial framework to spatiotemporal tensors, and DVD-GAN in July 2019 scaled that approach with separate spatial and temporal discriminators (GANs Learn to Move). These models produced short, low-resolution clips and inherited GAN training’s instability without inheriting a clean path to conditioning on open-ended text.

A second angle treated video less as a generation target and more as a environment to predict and control: World Models in March 2018 and PlaNet in November 2018 learned compact latent dynamics from pixels for planning and control rather than for producing videos a person would watch, and Dreamer in December 2019 pushed the same latent-imagination idea further (Predicting the Next Frame). This line matters more for what it becomes later in this book than for what it produced in 2022 — the world-model framing this line established is picked up again once video models start being used to simulate environments rather than just render clips.

A third angle borrowed the discretize-then-model recipe from image generation: VQ-VAE’s codebooks (Pixels, One at a Time) let VideoGPT, in April 2021, treat a video as a sequence of discrete tokens a transformer could predict autoregressively, and CogVideo would push the same idea to open-domain text conditioning a month after this chapter’s first paper (Tokens for Video). Autoregressive video generation inherited a real strength from language modeling — a clean likelihood objective and mature transformer tooling — but also its weakness: generating a video one token at a time is slow, and errors compound frame after frame.

Meanwhile, on the image side, diffusion had already had its breakthrough. DDPM, in June 2020, reduced diffusion training to a plain denoising regression loss with no adversarial component (Diffusion Foundations), and by mid-2021 diffusion models were beating GANs on ImageNet fidelity using classifier guidance (Guidance). Every one of the three video approaches above had been trying to solve temporal coherence and sample quality with tools — adversarial losses, latent dynamics models, discrete token sequences — that diffusion’s own image-generation lineage had already found less stable or less scalable than iterative denoising. The obvious question, unaddressed as of early 2022, was whether the same recipe would work once the tensor being denoised had a time axis.

Three separate roads to moving pictures, each with its own failure mode, and none of them borrowing image diffusion's denoising recipe until this chapter's paper does. PNG ↓

The Breakthrough

Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David Fleet’s Video Diffusion Models, submitted in April 2022, made the minimal change necessary to test the question: keep DDPM’s U-Net and noise-prediction objective almost entirely intact, and factorize its operations across space and time rather than redesigning them from scratch (Ho et al., 2022). Every 3×3 spatial convolution in the image U-Net becomes a 1×3×3 convolution — a spatial-only filter applied identically to every frame — and every spatial self-attention block is followed by a new temporal attention block that attends across the frame axis while treating each spatial location independently, using relative position embeddings so the network can tell which frame comes first. Nothing in this design requires a full, unfactored 3D operation; the paper’s framing is explicit that this is “a natural extension of the standard image diffusion architecture,” not a new architecture family.

That architectural minimalism paid off twice. First, it made the model cheap enough to train that Ho et al. could jointly train on independent still images alongside video clips — a video-diffusion U-Net can process a single image by simply skipping the temporal attention blocks for that batch element — and joint training measurably stabilized and improved text-to-video results, cutting Fréchet Video Distance from 202.28 to 57.84 on their text-conditioned setting once eight independent image frames were mixed into every training batch. Second, because a trained video-diffusion model has learned to denoise short, fixed-length clips, the paper needed a way to produce videos longer or at higher resolution than any single training clip. Its answer — reconstruction-guided sampling, detailed in the next section — became the tool every subsequent cascaded video-diffusion system reused to stitch a fixed-length denoiser into arbitrary-length output.

Every box keeps a 2D or frame-wise operation; nothing here is a full, unfactored 3D layer, which is why the pretrained image weights still mean something. PNG ↓

Five months later, Meta’s Make-A-Video, led by Uriel Singer with twelve co-authors, took a different route to the same target audience: open-domain text-to-video generation, but with no text-video paired dataset at all (Singer et al., 2022). Rather than train a video diffusion model from scratch, Make-A-Video starts from a pretrained text-to-image diffusion model, so all of the text-to-visual alignment is inherited rather than relearned, then bolts on new temporal machinery: pseudo-3D convolutional and attention layers that decompose the full spatiotemporal operation into a spatial pass followed by a temporal pass, a frame-interpolation network to increase apparent frame rate, and a cascade of spatial and temporal super-resolution models. Crucially, the temporal layers and the interpolation network are trained on unlabeled video footage alone — no caption needed — so the model learns how things move from video while it learned what things look like and mean from image-text pairs. The pipeline generates 16 frames at 64×64, upsamples spatially to 256×256 and then 768×768, and interpolates temporally up to 76 frames, with frame rate conditioned as a training input ranging from 1 to 30 frames per second.

A month after Make-A-Video, Google returned to the problem with Imagen Video, again with Jonathan Ho as lead author alongside Tim Salimans, Chitwan Saharia, and eight further co-authors (Ho et al., 2022). Where Make-A-Video’s answer to scale was to inherit a pretrained image model, Imagen Video’s answer was to cascade: one base video-diffusion model generates a low-resolution clip, and six further diffusion models — three spatial super-resolution models and three temporal super-resolution models, interleaved — progressively upscale it in resolution and frame count, for seven diffusion models total and 11.6 billion parameters across the cascade. The system reuses Imagen’s frozen T5-XXL text encoder for conditioning (The Year Images Went Mainstream), and its final output reaches 1280×768 resolution at 24 frames per second for 128 frames — roughly 5.3 seconds of near-HD video, generated end to end by a chain of denoising networks with no adversarial component anywhere in the pipeline.

The Mathematics

Notation follows the video-diffusion papers; the underlying forward-noising process, reverse denoising step, and simplified training loss are DDPM’s, taught once in full in Diffusion Foundations and not re-derived here. What is new to video is how the denoising network is factorized across space and time, how a fixed-length denoiser is stitched into videos longer than its training window, and how the network’s prediction target is reparameterized for the stability a seven-model cascade demands.

Video Diffusion Models factorizes every spatial operation from every temporal one rather than treating a video clip as one undifferentiated 3D volume. For a clip with $T$ frames, spatial attention is applied within each frame independently, folding the frame axis into the batch dimension:

\mathrm{SpatialAttn}(\mathbf{x})_t = \mathrm{Attention}(\mathbf{x}_t), \qquad t = 1, \dots, T

Temporal attention runs the complementary operation, attending across frames at each fixed spatial location while treating every pixel position as an independent batch element:

\mathrm{TemporalAttn}(\mathbf{x})_{h,w} = \mathrm{Attention}\big(\mathbf{x}_{1,h,w}, \dots, \mathbf{x}_{T,h,w}\big)

Symbol	Meaning
$\mathbf{x}$	the noisy video tensor being denoised, shape $T \times H \times W \times C$
$\mathbf{x}_t$	frame $t$ of that tensor, a single noisy image
$\mathbf{x}_{t,h,w}$	the feature vector at frame $t$ , spatial location $(h,w)$
$T$	number of frames in the clip
$H, W$	spatial height and width

Because a video-diffusion U-Net is trained on clips of one fixed length, generating anything longer — or jointly super-resolving a clip once it exists — requires stitching separately-denoised blocks together without visible seams. Video Diffusion Models’ reconstruction-guided sampling nudges the denoiser’s prediction for one block of frames using its prediction for another, already-known or already-generated block:

\tilde{\mathbf{x}}_\theta^{\,b}(\mathbf{z}_t) = \hat{\mathbf{x}}_\theta^{\,b}(\mathbf{z}_t) - \frac{w_r\, \alpha_t}{2}\, \nabla_{\mathbf{z}_t^{\,b}} \left\lVert \mathbf{x}^{a} - \hat{\mathbf{x}}_\theta^{a}(\mathbf{z}_t) \right\rVert^2

Say block $a$ is a clip’s already-generated first half and block $b$ is the second half still being sampled. $\hat{\mathbf{x}}_\theta^{a}(\mathbf{z}_t)$ is what the network currently thinks block $a$ looks like at this noise level; the gradient term measures how wrong that guess is relative to the actual, already-fixed $\mathbf{x}^a$ , and pushes block $b$ ‘s prediction in whatever direction would have made block $a$ ‘s guess more accurate too — coupling the two blocks so they stay consistent with each other. The same mechanism, applied with $a$ and $b$ as a low-resolution clip and its higher-resolution or higher-frame-rate counterpart rather than as two time ranges, is what lets a fixed-size denoiser be reused for spatial super-resolution, temporal super-resolution, and autoregressive frame extension — exactly the three operations Imagen Video’s seven-model cascade chains together.

Symbol	Meaning
$\mathbf{z}_t$	the full noisy latent at diffusion step $t$ , spanning both blocks
$\mathbf{x}^{a}$	frames in block $a$ that are already known or already generated
$\hat{\mathbf{x}}_\theta^{a}, \hat{\mathbf{x}}_\theta^{b}$	the network’s direct denoised-image prediction for block $a$ and block $b$
$w_r$	a guidance weight controlling how strongly block $b$ is pulled toward consistency with block $a$
$\alpha_t$	the fraction of clean signal remaining at step $t$ , from the same noise schedule taught in Diffusion Foundations

Running seven diffusion models in sequence, each conditioning on the last, raises a stability problem that a single-model image pipeline does not face: small numerical errors in an early-cascade model compound through every model downstream of it. Imagen Video addresses this by having every model in the cascade predict a quantity called $v$ rather than predicting the noise $\epsilon$ directly, as DDPM does:

v_t \equiv \alpha_t\, \epsilon - \sigma_t\, \mathbf{x}

$\epsilon$ is the actual Gaussian noise added during the forward process, and $\mathbf{x}$ is the clean data; $\alpha_t$ and $\sigma_t$ are the signal and noise coefficients of the same forward-noising schedule from Diffusion Foundations, reparameterized so that $\alpha_t^2 + \sigma_t^2 = 1$ . Instead of training the network to output $\epsilon$ or $\mathbf{x}$ directly, $v_t$ blends the two in proportions that shift smoothly with $t$ , which the paper reports avoids a specific failure mode — visible color drift across a video’s duration — and gives smoother, more numerically stable training at the high spatial and temporal resolutions the later stages of the cascade operate at. It is the same $v$ -prediction reparameterization that also underlies fast, few-step sampling elsewhere in the diffusion literature (Few-Step Generation), applied here to keep a seven-model chain from accumulating error.

Symbol	Meaning
$v_t$	the reparameterized prediction target, a blend of noise and signal
$\epsilon$	the actual Gaussian noise added during the forward process
$\mathbf{x}$	the clean data the forward process started from
$\alpha_t$	signal coefficient of the forward-noising schedule at step $t$
$\sigma_t$	noise coefficient of the forward-noising schedule at step $t$ , with $\alpha_t^2+\sigma_t^2=1$

What It Enabled

Within a single year, video generation moved from three competing, individually-limited paradigms to one dominant recipe — denoise a spatiotemporal tensor — with the remaining research effort going into how the tensor is represented and how the cascade is organized rather than whether diffusion was the right tool.

	GAN video, pre-2022	Autoregressive tokens, 2021–2022	Video diffusion, 2022
Training signal	adversarial minimax game	next-token cross-entropy	closed-form denoising regression
Open-domain text conditioning	limited	yes, CogVideo	yes, all three papers
Sampling cost	one generator pass	one token at a time, sequential	many denoising steps, per cascade stage
Best demonstrated resolution/length (2022)	short, low-res clips	short, low-res clips	1280×768, 128 frames, 24fps (Imagen Video)
Failure mode	mode collapse, training instability	compounding token errors over time	seam artifacts across cascade stages, mitigated by reconstruction guidance

A second table lines the three anchor papers up directly, since each made a different trade-off rather than one simply superseding the last:

	Video Diffusion Models	Make-A-Video	Imagen Video
Starting point	image U-Net, factorized space-time	pretrained text-to-image diffusion model	Imagen-style cascade, built from scratch
Needs paired text-video data	no (unconditional/prediction focus)	no	yes, for the base model
Cascade depth	base model plus reconstruction-guided extension	base plus interpolation plus 2 SR stages	7 diffusion models total, 11.6B params
Reported quality signal	UCF-101 FVD 295±3 (real data: 60.2)	UCF-101 zero-shot FVD 367.23, IS 33.00	CLIP score 25.19±0.03, evaluated at 192×320, 128 frames
Final output	16×64×64 and 9×128×128 clips	76 frames, up to 768×768 (released at 512×512)	1280×768, 24fps, 128 frames

The cascade pattern Imagen Video formalized — a small base diffusion model plus a chain of super-resolution diffusion models — became the template the field kept reusing rather than discarding: the next chapter’s move into a compressed latent space rather than raw pixels (Latent Video Diffusion) keeps the cascade idea while making each stage far cheaper to run. The pseudo-3D, inherit-from-a-pretrained-image-model strategy Make-A-Video introduced also proved durable: later image-to-video and motion-control systems start from a fixed text-to-image backbone and add temporal layers on top, exactly as Make-A-Video did (Image-to-Video and Motion Control). And reconstruction-guided sampling’s core idea — use one part of a diffusion model’s own prediction to keep another part consistent — recurs under different names throughout later video-to-video editing and control work (Video-to-Video and Editing).

This chapter's three papers share one ancestor and hand off to the same two futures: a cheaper cascade in latent space, and a later unification under one tokenization scheme. PNG ↓

Running a single image-diffusion U-Net for a thousand steps was already inference’s dominant cost in 2020 (Diffusion Foundations). Imagen Video multiplies that cost by seven separate networks chained end to end, each with its own denoising steps, to produce five seconds of video — a compute bill that only large, well-funded labs running on datacenter-class accelerators like the A100 could absorb in 2022. That escalating compute-per-output-second is one instance of the broader pattern this book’s engine-room volume tracks across every model generation (The Co-evolution Thesis): the model class that becomes possible is bounded by the hardware available to run it, and video diffusion’s cascades were only economical because A100-class accelerators, announced two years earlier, existed to run them.

Key Papers

Paper	Authors	Year	Link	Why it matters
Video Diffusion Models	Ho, Salimans, Gritsenko, Chan, Norouzi, Fleet	2022	arXiv:2204.03458	First direct extension of DDPM to spatiotemporal tensors via factorized space-time attention; introduced joint image-video training and reconstruction-guided sampling for length/resolution extension
Make-A-Video: Text-to-Video Generation without Text-Video Data	Singer et al.	2022	arXiv:2209.14792	Showed open-domain text-to-video was achievable by extending a pretrained text-to-image diffusion model with pseudo-3D temporal layers trained on unpaired video, avoiding the need for text-video pairs
Imagen Video: High Definition Video Generation with Diffusion Models	Ho, Chan, Saharia, Whang, Gao, Gritsenko, Kingma, Poole, Norouzi, Fleet, Salimans	2022	arXiv:2210.02303	Formalized the cascaded-diffusion template — 7 models, 11.6B parameters — and introduced v-parameterization to keep a deep cascade numerically stable; reached 1280×768 at 24fps

PM Lens

The number I keep coming back to from this chapter is seven — the count of separate diffusion models Imagen Video chains together to produce one clip. Every “generate video” button on a product built this way is quietly asking a user to wait on the slowest and most expensive stage in a pipeline, not a single model call, and that shapes what you can promise about latency and unit economics long before the marketing copy is written. It also explains why, when Runway, Pika, and other 2023-era video products shipped, the visible product decision was almost always about clip length and resolution tiers rather than raw quality — those are exactly the two knobs a cascade like this exposes most directly. The other thing worth internalizing: Make-A-Video’s trick of inheriting a pretrained image model rather than training video from scratch is a build-vs-reuse decision, and it is the one that made a five-figure-GPU-hour problem tractable for teams without Google-or-Meta-scale compute budgets.

Search