Latent Video Diffusion

It is April 2023. Video diffusion models trained from scratch — the cascades and joint image-video objectives of the previous year — can produce a few seconds of motion, but every one of them pays for that motion by training a video-shaped model on video-scale compute from the first step. A team spanning NVIDIA, LMU Munich, and the University of Toronto asks a narrower question: Stable Diffusion already knows what a photograph looks like; can a model be taught what motion looks like without re-learning what an image looks like at all?

Where We Were

By early 2022, video diffusion had already proven the underlying generative mechanism could stretch from pixels to pixels-over-time. Video Diffusion Models, or VDM, extended the DDPM objective (Diffusion Foundations) to space-time volumes using factorized spatio-temporal attention and a joint image-video training objective, and later that year Make-A-Video and Imagen Video pushed the same family to text-conditioned generation via cascades of spatial and temporal super-resolution stages (Video Diffusion Models). Every one of these systems, however, trained its spatial and temporal parameters together, from the same random initialization, on video data — and video data suitable for training a generative model was two to three orders of magnitude scarcer than the billion-image web corpora that text-to-image diffusion had by then consumed. Imagen Video’s own reported scale, more than 11 billion parameters spread across its base model and cascade of super-resolution stages, reflected what that joint approach cost: a video-sized model trained end-to-end had to relearn visual fidelity itself before it could turn to the added problem of coherent motion.

A separate development, arriving in December 2021 and shipping as open weights that August, had already solved the “what does a photograph look like” problem about as well as it had ever been solved: Latent Diffusion Models, and the Stable Diffusion checkpoint trained on that recipe, compressed image generation into a small latent grid and let a single 860-million-parameter U-Net produce photorealistic images on a consumer GPU (Latent Diffusion and Stable Diffusion). By April 2023, Stable Diffusion had been public for eight months, fine-tuned into thousands of community checkpoints, and had become the default backbone for image generation research and products alike. The gap between the two lineages was obvious once stated plainly: video diffusion research kept re-learning visual fidelity that image diffusion had already solved, while nobody had yet asked whether a frozen, pretrained image LDM could be turned into a video generator by adding motion on top rather than training motion and appearance together.

Everything before Video LDM keeps relearning visual fidelity that image diffusion had already solved; everything after it freezes that fidelity and reuses it. PNG ↓

The Breakthrough

Blattmann, Rombach, Ling, Dockhorn, Kim, Fidler, and Kreis’s “Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models,” or Video LDM (Blattmann et al., 2023), answered the question directly. The recipe starts from a pretrained image LDM — its perceptual autoencoder and its spatial U-Net, already trained to denoise individual frames — and inserts new temporal mixing layers between the existing spatial layers, initialized so that at the start of training they contribute nothing and the network behaves exactly like the original image model. Training then proceeds on encoded video sequences, but during that optimization the image backbone’s original parameters are held fixed and only the new temporal layers’ parameters are updated. The video model that results has, by construction, never forgotten how to render a photorealistic frame; it has only learned how successive frames should relate to one another.

That same frozen-backbone idea extended past the base generator to the pipeline around it. Video LDM trains a temporal alignment on its diffusion-based upsampler too, turning what was originally an image super-resolution model into a temporally consistent video super-resolution model — applying the same “freeze the spatial expert, add temporal layers” trick a second time, at a different stage of the pipeline, rather than training a separate video-native upsampler from scratch. On real driving-scene video the fine-tuned pipeline reached 512×1024 resolution; adapting the same recipe onto the public Stable Diffusion checkpoint, rather than a custom base model, produced a text-to-video generator running up to 1280×2048, output beyond what the video diffusion cascades of the prior year had reached at comparable training cost.

The freeze-and-add trick runs twice, once on the base generator and once on its upsampler, so every gradient step on scarce video data buys motion, never fidelity the image model already had. PNG ↓

Three months later, Guo, Yang, Rao, Liang, Wang, Qiao, Agrawala, Lin, and Dai’s “AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning” (Guo et al., 2023) took the frozen-backbone principle one step further: rather than fine-tune temporal layers onto one specific checkpoint, train a single motion module once, on a generic text-to-image base, and let it plug into any personalized checkpoint fine-tuned from that same base without any further training at all. The motion module is a temporal Transformer — a stack of self-attention blocks operating along the frame axis rather than the spatial axes, with sinusoidal position encoding marking each frame’s position in the sequence, trained on WebVid-10M video clips. Because the module only ever attends across time and never touches the spatial weights it is inserted alongside, it transfers cleanly onto the thousands of community-fine-tuned Stable Diffusion checkpoints built on the same base, animating a specific art style or character design that the motion module itself never saw during training. AnimateDiff also introduced MotionLoRA, a low-rank adaptation of the motion module (LoRA and PEFT) that adapts a trained motion module to a specific camera movement — pan, zoom, tilt, rotation — from as few as 20 to 50 reference clips and roughly 2,000 training iterations, at a fraction of the motion module’s own training cost.

Stable Video Diffusion, or SVD (Blattmann et al., 2023), released by Stability AI that November, applied the same “start from an image LDM, add temporal layers” architecture at production scale and, unlike Video LDM and AnimateDiff, released the trained weights themselves rather than only the method. Built on Stable Diffusion 2.1, with temporal convolution and temporal attention layers inserted after each of the base model’s spatial layers, SVD trains in three explicit stages: image pretraining, which simply reuses SD 2.1’s already-trained spatial weights; video pretraining, on a large curated corpus at lower resolution; and a final high-resolution finetuning stage on a smaller set of higher-quality clips. The paper’s ablations found that each of those choices mattered independently and additively — initializing from a pretrained image model beat random initialization, and filtering the pretraining corpus beat training on the same volume of unfiltered clips, with the improvement from curation persisting through the final finetuning stage rather than washing out. The released model comes in two variants sharing one 1.5-billion-parameter backbone: SVD, generating 14 frames, and SVD-XT, fine-tuned from it to generate 25 frames, both taking a single conditioning image and animating it at selectable frame rates from 3 to 30 fps — image-to-video rather than the text-to-video framing Video LDM and AnimateDiff had used.

The Mathematics

Video LDM’s core training modification is not a new loss function — the noise-prediction objective is the same one taught in full in Diffusion Foundations and reused unchanged by Latent Diffusion and Stable Diffusion — but a constraint on which parameters that loss is allowed to update:

\phi^{*} = \arg\min_{\phi} \; \mathbb{E}_{z=\mathcal{E}(x),\, t,\, \epsilon \sim \mathcal{N}(0,\mathbf{I})}\Big[\big\lVert \epsilon - \epsilon_{\theta,\phi}(z_t, t) \big\rVert_2^2\Big], \qquad \theta \text{ held fixed}

$\mathcal{E}(x)$ , $z_t$ , and $\epsilon$ mean exactly what they meant in the latent-diffusion chapter: a frozen autoencoder’s latent, that latent after $t$ steps of noising, and the true noise the network must predict. The only change is the subscript on the denoising network: $\epsilon_{\theta,\phi}$ now depends on two disjoint parameter sets, the original image model’s weights $\theta$ and a new set of temporal-layer weights $\phi$ . The $\arg\min_\phi$ states plainly that this loss is only ever minimized over $\phi$ — $\theta$ still shapes the network’s output, but no optimizer step is ever taken with respect to it, so it stays fixed at its pretrained image-model value throughout video training. What this buys the model: every gradient step spent training on scarce, expensive video data goes exclusively toward learning temporal structure, none of it toward re-deriving visual fidelity the frozen image model already has.

Symbol	Meaning
$\theta$	the pretrained image LDM’s spatial parameters, frozen during video training
$\phi$	the new temporal-layer parameters, the only ones updated
$z_t$	the video latent sequence after $t$ steps of forward noising
$\epsilon_{\theta,\phi}$	the noise prediction, computed by the combined spatial-plus-temporal network

Within the network, each temporal layer does not simply replace its neighboring spatial layer’s output — it blends with it, and how much it blends is itself learned:

z^{i} = \alpha_\phi^{i} \cdot z_\theta^{i} \;+\; \big(1 - \alpha_\phi^{i}\big) \cdot z_\phi^{i}

At internal layer $i$ , $z_\theta^i$ is the frozen spatial layer’s own output — what the image model would have produced if it were still just denoising a single frame — and $z_\phi^i$ is the new temporal layer’s output, computed from that same feature map but mixed across the time axis. $\alpha_\phi^i$ is a learned scalar between 0 and 1, one per layer, that decides how much of the final feature map comes from the spatial path versus the temporal path. Initializing $\alpha_\phi^i$ at 1 makes the network behave exactly like the original image model on day one of video training — every temporal layer starts as a no-op — and training only pushes $\alpha_\phi^i$ down, and the temporal layer’s influence up, where the data shows it actually helps.

Symbol	Meaning
$z_\theta^i$	layer $i$ ‘s output from the frozen spatial path alone
$z_\phi^i$	layer $i$ ‘s output from the new temporal path
$\alpha_\phi^i$	a learned per-layer blend weight between the two paths
$z^i$	the combined feature map passed to the next layer

AnimateDiff’s motion module reuses the exact attention mechanism taught in The Transformer, with one change of axis:

\mathrm{Attention}(Q,K,V) = \mathrm{softmax}\!\left(\frac{QK^{\top}}{\sqrt{d}}\right)V, \qquad Q,K,V = W_{Q,K,V}\,\mathrm{reshape}_{\text{frame axis}}(h)

The scaled dot-product attention operation on the left is identical to ordinary self-attention; nothing about softmax, scaling by $\sqrt{d}$ , or the query-key-value structure changes. What changes is what gets reshaped into a sequence before the projections $W_Q, W_K, W_V$ are applied: $h$ is a hidden feature map with spatial and temporal axes both present, and $\mathrm{reshape}_{\text{frame axis}}$ folds the spatial positions into the batch dimension so that attention runs along the frame axis instead of across pixel positions. Every spatial location in a video ends up attending only to itself across time, at every other frame, rather than to its neighbors within one frame. That is the entire architectural difference between spatial self-attention, as used in an image U-Net, and a temporal-attention motion module: same operation, different axis treated as the sequence.

Symbol	Meaning
$h$	the hidden feature map, spatial positions times frames
$Q, K, V$	query, key, value projections computed after the frame-axis reshape
$W_{Q,K,V}$	the motion module’s own learned projection matrices
$d$	the key dimensionality, used to scale the dot product

What It Enabled

The bolt-on-temporal-layers pattern collapsed the cost of entering video generation from “train a video-scale model from scratch” to “add and train a comparatively small set of new parameters on top of an existing, working image model” — and it did so for both custom research pipelines and the broader open ecosystem.

	Video diffusion trained end-to-end, 2022	Video LDM, temporal layers on frozen image LDM, 2023	Stable Video Diffusion, open weights, 2023
Spatial fidelity source	learned jointly with motion, from video data	inherited from a frozen pretrained image LDM	inherited from Stable Diffusion 2.1
Trainable parameters for video capability	the entire model	only the new temporal layers	temporal layers plus curated-data finetuning
Reported scale	Imagen Video, over 11B parameters across its cascade	not disclosed as a single total; temporal layers added to an 860M-parameter image U-Net	1.5B-parameter backbone, shared by both released variants
Weights publicly released	no	no, method only	yes, research and non-commercial license
Output	driving video at 512×1024; text-to-video up to 1280×2048	14-frame or 25-frame clips at 576×1024, from a single conditioning image	—

Video LDM’s zero-shot UCF-101 result — an FVD (Evaluating and Monetizing Motion) of 550.61, against Make-A-Video’s 367.23 on the same benchmark — showed the trade-off plainly: adapting a frozen image model did not yet match a model trained end-to-end on video at comparable scale, but it reached a usable result at a fraction of the training cost, on a fraction of the video data, because most of the network’s capacity had already been paid for by image pretraining. AnimateDiff’s contribution was orthogonal to that fidelity question: because its motion module attaches to any checkpoint sharing the same frozen base, it turned every one of the thousands of community-fine-tuned Stable Diffusion variants into a video generator without touching a single one of their weights, extending the personalization ecosystem documented in Personalization and LoRA from stills into motion.

Stable Video Diffusion’s public release repeated Stable Diffusion’s own playbook — open weights beat a closed, better-resourced competitor to the hands of developers — one level up the stack, from images to video:

Open weights beat a closed, better-resourced competitor to developers' hands twice in this lineage: once for images with Stable Diffusion, once for video with Stable Video Diffusion. PNG ↓

SVD’s image-to-video framing and its released weights are the direct foundation that camera-trajectory and drag-based motion control methods build on (Image-to-Video and Motion Control); its open checkpoint and three-stage training recipe are the template the next wave of open video models — CogVideoX, HunyuanVideo, and their contemporaries — scale up rather than reinvent (Open Video Models); and step-distillation techniques developed for image diffusion later target this same SVD backbone to cut its sampling cost toward real time (Making Video Cheap).

Key Papers

Paper	Authors	Year	Link	Why it matters
Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models	Blattmann, Rombach, Ling, Dockhorn, Kim, Fidler, Kreis	2023	arXiv:2304.08818	Introduced frozen-spatial-backbone plus trainable temporal layers, turning a pretrained image LDM into a video generator and its upsampler into a temporally consistent super-resolution model
AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning	Guo, Yang, Rao, Liang, Wang, Qiao, Agrawala, Lin, Dai	2023	arXiv:2307.04725	A portable motion module, trained once, that animates any personalized checkpoint sharing the same frozen base, plus MotionLoRA for low-cost camera-motion adaptation
Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets	Blattmann et al.	2023	arXiv:2311.15127	Applied the same architecture at production scale with a three-stage training recipe and curated dataset, and released the trained weights publicly

PM Lens

The number that mattered to me in this chapter isn’t a parameter count, it’s a training bill avoided. Freezing the spatial backbone and training only temporal layers is what let Stability AI ship Stable Video Diffusion’s weights in November 2023 under a research-and-non-commercial license rather than keep it a closed API, the way most competing image-to-video systems of that era stayed. If I’m scoping a video feature today, this chapter is where I ask a vendor which of the two architectures they actually shipped: a video model trained end-to-end, which usually means a closed API and a per-second bill, or an image model with bolted-on motion, which is far more likely to have an open or self-hostable variant I can put behind my own product economics. AnimateDiff’s MotionLoRA is the other lesson worth carrying forward — a specific camera move learned from 20 to 50 clips is a weekend project, not a training run, and that gap between “adapt an existing motion model” and “train a video model from scratch” is still the first question I ask before greenlighting any generative-video roadmap item.

Search