Video-to-Video and Editing

It is December 2022. Text-to-video diffusion models can already generate a short clip from a caption, but nobody has yet shown a general way to take a video that already exists — a phone recording, a stock clip, a rendered animation — and change it: swap the subject, restyle the footage, or apply an edit described in words while keeping the motion intact. Over the following four months, three papers close that gap by three different routes, and one of them, Runway’s Gen-1, ships as a commercial product before most of the field has even finished reading the preprint.

Where We Were

By late 2022, text-to-video generation had a real toolkit, but it was a generation toolkit, not an editing one. Video Diffusion Models (April 2022), Make-A-Video (September 2022), and Imagen Video (October 2022) — covered in Chapter 5 — had shown that a diffusion model could turn a caption into a short clip by extending image diffusion with factorized spatio-temporal attention and joint image-video training. CogVideo and Phenaki, covered in Chapter 4, took the autoregressive route to the same goal, tokenizing frames and predicting them one after another. What none of these systems accepted as input was a video. They ran forward from a text prompt and noise, once, to a finished clip; there was no slot in the pipeline for “and here is the footage I already have.”

Image diffusion, one book over, had a head start. SDEdit (August 2021) and Prompt-to-Prompt (August 2022) — covered in Chapter 17 of the image-generation book — had already shown two ways to edit a real image with a pretrained diffusion model: partially renoise it and let the reverse process finish the job, or intervene on the model’s own cross-attention maps to hold a layout in place while a prompt changes. Both tricks depend on DDIM’s proof, covered in Chapter 9 of the image-generation book, that the reverse diffusion process can be run deterministically and, therefore, inverted. Applying either trick to video was not a matter of copy-paste: a video is not one image but dozens of them, and any per-frame edit that is not forced to agree with its neighbors flickers instead of moving.

Every 2022 system ran forward from a prompt to a finished clip once. None of them had a slot for footage that already existed. PNG ↓

The missing piece was a way to keep a video’s motion and layout fixed while changing everything else about how it looks — without collecting a paired before-and-after video dataset, which barely existed at any scale, and without generating from scratch and hoping the result happened to match the input’s motion. Three groups, working independently, reached three different answers within a single winter.

The Breakthrough

Tune-A-Video (Wu et al., 2022), submitted to arXiv on December 22, 2022, took the narrowest and cheapest route: treat a single input video as its own training set. The method starts from a pretrained text-to-image latent diffusion model — the kind Chapter 13 of the image-generation book covers in full — and briefly fine-tunes it on one video and its caption, updating only a small slice of the network’s weights: the query projections inside the model’s existing cross-attention layers, plus a new temporal self-attention module, plus the query projection of a modified spatial self-attention module that now also looks at other frames. Everything else in the pretrained model — the layers that know how to draw a face, a car, or a snowy street — is left untouched. Once fine-tuned, the model has learned this one video’s motion well enough that swapping the caption’s subject (“a man is skiing” to “a panda is skiing”) and re-running generation, seeded from the original video’s own DDIM-inverted latent, produces a new video with the same motion and a different subject.

Ten minutes of fine-tuning on one video is enough to turn a general-purpose image model into a bespoke, one-video editor. PNG ↓

The mechanism that lets a model fine-tuned on a single video still generate frames that agree with each other is what the paper calls sparse-causal attention: each frame’s self-attention no longer looks only at itself, but also at the first frame and the immediately preceding frame, so a change introduced anywhere propagates forward through the sequence instead of being decided independently, frame by frame. Ten minutes of fine-tuning on one NVIDIA A100 GPU, and about a minute of sampling on the same card, is enough to turn a general-purpose image model into a bespoke, one-video video editor.

Gen-1 (Esser et al., 2023), submitted on February 6, 2023 by Runway, rejected the per-video fine-tuning approach entirely in favor of training one model, once, that generalizes to any input video at inference time. Gen-1 factors the editing problem into two separate conditioning signals. Structure — the coarse shape and layout of each frame, extracted as a monocular depth map — is concatenated to the noisy latent at every denoising step, telling the model where things are without telling it what they look like. Content — what the output should actually look like, supplied either as a driving image or as text — is injected through the model’s cross-attention layers as a CLIP embedding, the same joint image-text representation Chapter 11 of the image-generation book develops in full. Splitting “where” from “what” means a user can hand Gen-1 a video of someone dancing, describe the desired look in a sentence or a reference image, and get back a video with the original dancer’s motion and a new appearance.

Splitting where from what means one trained model handles any input video, not just the one it was tuned on. PNG ↓

Gen-1 is built the way Chapter 6 describes in general for later models: start from a pretrained image latent-diffusion model and add temporal convolution and attention layers so the network can reason across frames, not only within one. Gen-1’s specific contribution sits on top of that shared pattern — the depth-plus-CLIP conditioning split — and it was trained jointly on 240 million images and 6.4 million video clips, mixing image and video batches so the model retains the visual breadth of an image model while learning temporal behavior from video. Runway shipped Gen-1 as a feature inside its product within weeks of the preprint, turning a video-editing paper into a subscription feature before most competing labs had a comparable demo.

Text2Video-Zero (Khachatryan et al., 2023), submitted on March 23, 2023, went the opposite direction from Gen-1: instead of training anything at all, it repurposed a frozen, off-the-shelf Stable Diffusion checkpoint and produced temporally consistent video purely through how sampling is orchestrated. Two changes, applied at inference time only, do the work. First, the self-attention inside the pretrained U-Net is replaced with cross-frame attention: every frame’s queries are matched against the keys and values of a single fixed frame — the first frame — rather than its own, so every frame keeps rendering the same object identity and background instead of each one independently reimagining it. Second, before any denoising happens, the initial noise latents for frames after the first are not sampled independently; they are produced by warping the first frame’s own partially-denoised latent along a chosen motion direction, then re-noising, so the frames start from correlated points instead of unrelated random noise.

Neither trick touches a single model weight. Consistency comes entirely from how sampling is orchestrated. PNG ↓

Because neither trick touches a single model weight, Text2Video-Zero’s authors also showed it composing directly with existing conditioning tools: feeding it ControlNet-style pose or edge maps for conditional generation, or combining it with InstructPix2Pix — covered alongside Prompt-to-Prompt in Chapter 17 of the image-generation book — to perform instruction-guided video editing, again with no video-specific training anywhere in the pipeline.

The Mathematics

Tune-A-Video’s central mechanism is sparse-causal attention. For frame $v_i$ , the query still comes from that frame alone, but the keys and values are drawn from the first frame and the immediately preceding frame:

Q = W^{Q} z_{v_i}, \qquad K = W^{K}\big[z_{v_1};\, z_{v_{i-1}}\big], \qquad V = W^{V}\big[z_{v_1};\, z_{v_{i-1}}\big]

\mathrm{ST\text{-}Attn}(Q,K,V) = \mathrm{Softmax}\!\left(\frac{QK^{\top}}{\sqrt{d}}\right)V

$z_{v_i}$ is frame $i$ ‘s own features; $z_{v_1}$ and $z_{v_{i-1}}$ are the first frame’s and the previous frame’s features, joined together with the semicolon before the key and value projections are applied. $W^{Q}$ , $W^{K}$ , and $W^{V}$ are the (shared, across every frame and spatial position) linear projections that ordinary self-attention already uses — Chapter 5 of the language-models book derives the attention formula itself in full; nothing about that formula changes here. What changes, and what this equation buys you, is which features get to supply the keys and values: instead of a frame only ever attending to itself, it is forced to also look at the video’s anchor frame and its immediate predecessor, so an edit or a detail introduced early in the sequence has a direct channel to propagate into every later frame rather than being independently reinvented, frame by frame, from a caption alone.

Symbol	Meaning
$z_{v_i}$	features of the current frame $i$
$z_{v_1}$	features of the first frame
$z_{v_{i-1}}$	features of the previous frame
$W^{Q}, W^{K}, W^{V}$	shared query, key, value projection matrices
$Q, K, V$	the resulting query, key, and value tensors
$d$	the attention head dimension

Only a subset of Tune-A-Video’s weights — the query projections in this attention, plus a comparable set in the model’s cross-attention layers — are updated when the network is briefly fine-tuned on the single input video; the diffusion training loss itself is the same denoising objective Chapter 8 of the image-generation book derives, and generation afterward reuses the DDIM inversion and sampling Chapter 9 of the image-generation book covers, applied here to a video’s frames rather than a single image.

Gen-1 conditions its denoiser on two additional signals beyond the usual noisy latent and timestep: a structure representation $s(x)$ , derived from per-frame depth, and a content representation $c(x)$ , a CLIP embedding of a driving image or text. Its training objective extends the DDPM posterior-mean loss with these two extra arguments:

\mathcal{L} = \mathbb{E}\Big[\lambda_t \big\lVert \mu_t\big(\mathcal{E}(x)_t, \mathcal{E}(x)_0\big) - \mu_\theta\big(\mathcal{E}(x)_t, t, s(x), c(x)\big) \big\rVert_2^2\Big]

$\mu_t$ is the true forward-process posterior mean — the target the network is trying to match — and $\mu_\theta$ is the network’s prediction of that same quantity, both defined exactly as Chapter 8 of the image-generation book develops for ordinary image diffusion. $\mathcal{E}(x)_t$ and $\mathcal{E}(x)_0$ are the noisy and clean latent encodings of the video at diffusion step $t$ , and $\lambda_t$ is a per-timestep loss weight. The only change from the plain diffusion loss this chapter does not re-derive is that $\mu_\theta$ now also takes $s(x)$ and $c(x)$ as extra arguments. What it buys you: a single denoiser that has learned to separate “where things are,” carried by $s(x)$ , from “what they look like,” carried by $c(x)$ , so the two can be supplied independently at inference time — the original video’s structure, and a completely different content signal.

Symbol	Meaning
$\mu_t$	true forward-process posterior mean at step $t$
$\mu_\theta$	the network’s predicted posterior mean
$\mathcal{E}(x)_t, \mathcal{E}(x)_0$	noisy and clean latent encodings of the video
$s(x)$	structure conditioning, a per-frame depth map
$c(x)$	content conditioning, a CLIP embedding
$\lambda_t$	the per-timestep loss weight

At inference, Gen-1 extends classifier-free guidance — the single-signal mechanism Chapter 10 of the image-generation book covers in full — to three terms instead of one:

\tilde\mu_\theta(z_t,t,c,s) = \mu_\theta^{\mathrm{img}}(z_t,t,\varnothing,s) + \omega_t\big(\mu_\theta(z_t,t,\varnothing,s) - \mu_\theta^{\mathrm{img}}(z_t,t,\varnothing,s)\big) + \omega\big(\mu_\theta(z_t,t,c,s) - \mu_\theta(z_t,t,\varnothing,s)\big)

The first term, $\mu_\theta^{\mathrm{img}}$ , is a prediction from the model’s image-only pathway — ignoring temporal information, conditioned only on structure. The second term is a temporal-consistency correction, scaled by $\omega_t$ : the difference between the full video model’s prediction and that same image-only prediction, extrapolated outward to push the result toward whatever the temporal layers add over a single-frame guess. The third term is a content correction, scaled by $\omega$ : the difference between predicting with the content signal $c$ present versus absent ( $\varnothing$ ). What it buys you: two independent dials, $\omega_t$ and $\omega$ , that separately control how strongly the output favors temporal smoothness versus how strongly it follows the requested content — tunable at inference time, with no retraining, on top of a structure signal that is never toggled off.

Symbol	Meaning
$\tilde\mu_\theta$	the final, guided prediction used for sampling
$\mu_\theta^{\mathrm{img}}$	the model’s image-only-pathway prediction
$\mu_\theta$	the full video model’s prediction
$z_t$	the noisy latent at step $t$
$c$	the content conditioning signal
$s$	the structure conditioning signal
$\omega_t$	the temporal-consistency guidance scale
$\omega$	the content guidance scale
$\varnothing$	the null conditioning signal

Text2Video-Zero’s cross-frame attention replaces ordinary self-attention with a rule that always looks at the first frame:

\mathrm{Cross\text{-}Frame\text{-}Attn}\big(Q^{k}, K^{1:m}, V^{1:m}\big) = \mathrm{Softmax}\!\left(\frac{Q^{k}(K^{1})^{\top}}{\sqrt{c}}\right)V^{1}, \qquad k = 1,\dots,m

Symbol	Meaning
$Q^{k}$	frame $k$ ‘s own query
$K^{1}, V^{1}$	frame one’s key and value
$m$	the total number of frames
$c$	the attention feature dimension

Cross-frame attention alone only fixes appearance; motion still needs a push, which Text2Video-Zero supplies before denoising even begins:

\begin{aligned} x^{1}_{T} &\sim \mathcal{N}(0,\mathbf{I}) \\ \delta^{k} &= \lambda\,(k-1)\,\delta \\ \tilde{x}^{k}_{T'} &= W_k\big(x^{1}_{T'}\big) \\ x^{k}_{T} &\leftarrow \mathrm{DDPM\text{-}Forward}\big(\tilde{x}^{k}_{T'}, \Delta t\big) \end{aligned}

Frame one’s initial noise $x^{1}_{T}$ is sampled normally, then partially denoised by $\Delta t$ steps to $x^{1}_{T'}$ . A fixed direction $\delta$ — a two-dimensional translation — is scaled by $\lambda(k-1)$ to give each later frame $k$ its own displacement $\delta^{k}$ , growing with how far the frame sits from the start of the clip. $W_k$ warps frame one’s partially-denoised latent by that displacement to get $\tilde{x}^{k}_{T'}$ , which is then re-noised forward by the same $\Delta t$ steps to produce frame $k$ ‘s own starting point, $x^{k}_{T}$ . What it buys you: every frame’s denoising still runs independently from here, but instead of starting from unrelated random noise, each one starts from a version of the same noise pattern, nudged along a single consistent direction — enough to read as a camera pan or an object drifting across the frame, without any additional network or training run.

Symbol	Meaning
$x^{1}_{T}$	frame one’s initial Gaussian noise
$x^{1}_{T'}$	frame one’s latent after $\Delta t$ backward steps
$\delta$	the base motion-direction vector
$\lambda$	the motion-magnitude scale
$\delta^{k}$	frame $k$ ‘s scaled displacement
$W_k$	the warping operator applying $\delta^{k}$
$\Delta t$	the number of forward/backward diffusion steps
$x^{k}_{T}$	frame $k$ ‘s resulting initial noise

What It Enabled

The three methods trade training cost, generality, and edit fidelity against each other in ways that were never fully resolved by 2023 — later chapters show the industry continuing to pick different points on this same trade-off:

	Tune-A-Video, Dec 2022	Gen-1, Feb 2023	Text2Video-Zero, Mar 2023
Training required	one short fine-tune, per input video	one large training run, reused for any video	none — frozen pretrained checkpoint
Editing signal	edited caption	driving image or text (content), plus structure	text prompt, optionally with ControlNet-style conditioning
Generalizes without retraining	no, needs 10 minutes per new video	yes	yes
Shipped as a product	research code only	Runway Gen-1, within weeks of the preprint	research code only

A second table places the numbers each paper reported against CogVideo — covered in Chapter 4 — as a common autoregressive baseline, though each pair was measured on its own set of prompts, not a shared benchmark, so the two CogVideo rows below are not the same run:

Metric	CogVideo	Tune-A-Video
Frame consistency, CLIP embedding similarity	90.64	92.40
Text-video CLIP score	23.91	27.58
Human preference, frame consistency	12.14%	87.86%
Human preference, textual alignment	15.00%	85.00%

Metric	CogVideo	Text2Video-Zero
Text-video CLIP score, 25 sampled prompts	29.63	31.19

Gen-1’s own human study, run against per-frame image-editing baselines rather than CogVideo, found it preferred over SDEdit (at 50% noising strength) 85.29% of the time and over Text2LIVE 88.24% of the time — evidence that editing video directly, rather than editing each frame independently and hoping the results agreed, was already the stronger approach by early 2023.

None of the three methods solved every problem the category would eventually need solved. Tune-A-Video’s per-video fine-tuning does not scale to a product serving many users’ videos at once. Gen-1’s depth-based structure signal ties the edit to the input video’s exact geometry, which is a poor fit for changes that should alter motion itself rather than just appearance — the subject Chapter 8 takes up directly, with explicit camera-trajectory and drag-based motion conditioning. Text2Video-Zero’s warped-latent motion is a single global direction, not per-object or camera-aware motion, and its cross-frame attention alone starts to lose consistency over longer clips — a limitation Chapter 12 addresses for video generation broadly. Gen-1’s product path, meanwhile, continued directly into Runway’s later Gen-2 and Gen-3 models, covered in Chapter 10, as video editing and video generation increasingly ran through the same commercial pipelines rather than separate research tracks.

Key Papers

Paper	Authors	Year	Link	Why it matters
Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation	Wu et al.	2022	arXiv:2212.11565	Shows a pretrained image diffusion model can be turned into a single-video editor with ten minutes of fine-tuning and a sparse-causal attention trick
Structure and Content-Guided Video Synthesis with Diffusion Models	Esser et al.	2023	arXiv:2302.03011	Separates structure (depth) from content (CLIP) conditioning in a single trained model; shipped as Runway Gen-1 within weeks
Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators	Khachatryan et al.	2023	arXiv:2303.13439	Produces temporally consistent video from a frozen Stable Diffusion checkpoint via cross-frame attention and latent warping, with no training at all

PM Lens

This chapter is where “video editing” stopped being a research demo and became a line item in a product roadmap — Runway shipped Gen-1 as a subscription feature within weeks of the arXiv preprint, and every AI video tool since has had to answer the question these three papers first posed differently: pay for a general model up front (Gen-1, expensive to train, cheap per edit), pay per input at fine-tuning time (Tune-A-Video, cheap to build, slower per new video), or pay nothing beyond a frozen checkpoint’s existing cost (Text2Video-Zero, free to deploy, weaker consistency). When I evaluate a video-editing pitch today, the first question is which of these three trade-offs is running under the hood, because the answer predicts where the demo breaks: a Gen-1-style tool struggles once an edit needs to change motion rather than appearance; a Tune-A-Video-style approach struggles at scale; a Text2Video-Zero-style tool struggles past a few seconds of footage. None of that is a flaw to fix later — it is the bet each team made in the winter of 2022–2023, and usually still the bet their product makes now.

The three methods sit at three different points on when compute gets spent. Text2Video-Zero pays nothing at training time — it reuses a frozen Stable Diffusion checkpoint and moves the entire cost to inference-time attention bookkeeping and a handful of extra warping operations per frame. Tune-A-Video pays a small, per-video training cost — about ten minutes on a single NVIDIA A100 — before every edit, trading a short wait for a model specialized to that one input. Gen-1 pays the largest cost furthest upstream: one training run over 240 million images and 6.4 million video clips, after which any user’s edit costs Runway roughly a minute of inference and no further training at all. That spectrum, from spend-nothing-until-inference to spend-everything-once-upfront, is the same question Engine Room, Chapter 19 formalizes for language models choosing between a bigger pretraining run and more inference-time computation per query — video editing ran into the same economics a year earlier, one input at a time.

Search