Long Video and Consistency

It is March 2024. Sora’s technical report, published a month earlier, has told the field that a diffusion transformer can be trained jointly on images and clips of whatever length, resolution, and aspect ratio arrive in the training set — but the model it describes still tops out at a minute, and nothing in spacetime patches or the diffusion training objective says how a clip that long, or several times longer, stays coherent from its first frame to its last. Two lines of work spend the next ten months answering that question from opposite directions: give a diffusion model an explicit memory of what it already generated, or give it the sequential structure of a language model without giving up diffusion’s training stability.

Where We Were

By early 2024, every disclosed video model shared the same underlying limit even where its headline numbers differed. Stable Video Diffusion, released openly in November 2023, generated 14 or 25 frames per sample, a ceiling fixed by the length of clip it was trained on (Sora and Spacetime Patches). Sora pushed that ceiling to a minute by removing the fixed-resolution, fixed-duration training grid entirely, but OpenAI’s own report was explicit that the result was not fully solved even within that minute: object permanence held “often, though not always,” and the report said nothing about behavior beyond the disclosed duration. Longer than that, the only available technique was to chain the model to itself — generate a clip, condition a new clip on the last frame of the one before it, repeat. StreamingT2V’s authors, describing exactly this technique in their own paper, named its failure mode directly: applied naively, autoregressive image-to-video chaining is “prone to video stagnation,” where later chunks stop moving, drift into a different scene, or lose the objects and characters the first chunk established.

That failure was not new. The compress-then-predict lineage covered in Tokens for Video had already built an autoregressive extension mechanism for video, seventeen months earlier: Phenaki’s causal C-ViViT tokenizer let a new clip be seeded from the tail of the one just generated, re-encoding it and re-conditioning on a new prompt, which is how Phenaki reached minutes of coherent video from a changing sequence of captions back in October 2022 (Villegas et al., 2022). But that trick lived inside a discrete-token, mostly bidirectional-masked generator, and the field’s center of gravity had moved to continuous diffusion models specifically because diffusion training is more stable than sampling a discrete autoregressive prior. A plain diffusion U-Net or DiT trained to denoise one fixed-length clip has no built-in notion of “condition on my own not-yet-finished output” the way an autoregressive token model does by construction — extending it past its training length means bolting an ad hoc conditioning mechanism onto an architecture that was never asked to support one, which is exactly where naive chaining and its stagnation came from.

None of the three fixes is the diffusion training objective itself. Each bolts a mechanism on top of it. PNG ↓

Three separate problems needed solving at once, and none of them were the same problem diffusion’s core denoising objective addresses. First, memory: a model generating chunk 40 needs some signal about what chunk 1 established, not just what chunk 39 ended on, or scenes and characters drift away from where they started. Second, stability: whatever mechanism supplies that signal has to avoid the error accumulation that plagues any autoregressive process, where a small mistake in one step becomes the flawed input to the next. Third, speed: a mechanism that revisits the entire generated-so-far sequence at every new step scales its cost with length, which is the opposite of what a genuinely long or indefinitely streamed video needs. The next ten months produced one answer to each.

The Breakthrough

StreamingT2V (Henschel et al., 2024), posted in March 2024, answers the memory problem with two purpose-built modules bolted onto an existing image-to-video diffusion backbone rather than a change to the diffusion objective itself. A Conditional Attention Module acts as short-term memory: at each new chunk, it attends over features extracted from the immediately preceding chunk and injects them into the current denoising process, which is what keeps motion continuous across a chunk boundary instead of visibly restarting. An Appearance Preservation Module acts as long-term memory: it extracts high-level scene and object features from the very first chunk, generated once at the start, and keeps re-injecting those same features into every later chunk — the explicit fix for the “forgetting” that naive last-frame conditioning cannot prevent, since a last frame carries no information about what the scene looked like forty chunks earlier. A final randomized-blending step lets a separate super-resolution enhancer be applied chunk by chunk across an arbitrarily long sequence without visible seams between enhanced segments. The paper reports generating videos of 80, 240, 600, and 1,200 or more frames — up to roughly two minutes — with what the authors describe as consistent motion and no stagnation, a direct answer to the failure mode the paper itself names.

A last-frame-only chain has no box like appearance preservation, which is exactly what StreamingT2V's paper says stagnates. PNG ↓

Diffusion Forcing (Chen et al., 2024), posted four months later, answers the stability problem at a more fundamental level: rather than adding memory modules to a diffusion model that still treats a whole clip as one block to denoise together, it changes what “one denoising step” means. A standard diffusion model, taught in full in Diffusion Foundations, assigns a single noise level to an entire sample and denoises every part of it together. Diffusion Forcing instead assigns every frame in a sequence its own, independently chosen noise level, so a training batch might ask the network to treat frame 1 as almost fully denoised while frame 50 is still nearly pure noise. The paper’s own framing of the problem this solves is direct: “teacher forcing easily blows up while causal full-sequence diffusion models suffer from serious consistency issues” — teacher forcing being the standard training trick for autoregressive sequence models, where a network is trained to predict each next step from ground-truth past steps, a technique that alone does not transfer cleanly to continuous, high-dimensional video frames. By letting past frames sit near a clean, low-noise state while future frames stay noisy, the same network can be run either as something close to a language model — predict one new, mostly-clean frame at a time, conditioned on everything already generated — or as something close to a standard diffusion model — denoise a whole block of equally noisy frames together — as two extremes of the identical trained model, chosen at sampling time rather than baked into the architecture. The project’s own reported result for how far that buys a model past its training window is concrete: a model trained on 36-frame sequences from the DMLab video-prediction dataset “can rollout for 2000 frames or more,” without the sliding-window re-encoding trick that ordinary autoregressive extension needs.

The causal video diffusion model described in From Slow Bidirectional to Fast Autoregressive Video Diffusion Models (Yin et al., 2024), released under the project name CausVid five months later, answers the speed problem by taking a pretrained bidirectional diffusion transformer — one that, like Sora’s, denoises an entire clip’s patch sequence jointly and therefore needs the whole future to generate any single frame — and distilling it into a causal student that generates one frame at a time from only the past. The distillation target is not simply matching pixels: it adapts distribution matching distillation, the same few-step technique Few-Step Generation covers for image diffusion, into an asymmetric setup where a causal student is supervised by a bidirectional teacher’s full-sequence judgments, which the paper reports “effectively mitigates error accumulation in autoregressive generation” — directly targeting the drift that plain frame-by-frame chaining suffers from. The result compresses a 50-step diffusion process into a 4-step causal generator that, combined with key-value caching so each new frame’s generation reuses rather than recomputes attention over every earlier frame, streams video at 9.4 frames per second on a single GPU, and the paper reports a VBench-Long score of 84.27, ahead of every video generation model it compares against on that benchmark (Evaluating and Monetizing Motion covers the VBench family of metrics in full).

Distillation, not architecture search, is what turns a slow full-clip denoiser into a streaming one. PNG ↓

The Mathematics

A standard diffusion model, in the notation Diffusion Foundations introduces, assigns one shared timestep $t$ to an entire sample $x_0$ and diffuses all of it together: $x_t = \sqrt{\bar\alpha_t}\,x_0 + \sqrt{1-\bar\alpha_t}\,\epsilon$ . Diffusion Forcing’s central change is to let a sequence of $T$ frames carry $T$ independent noise levels instead of one:

x^{i}_{k^i} = \sqrt{\bar\alpha_{k^i}}\, x^i_0 + \sqrt{1-\bar\alpha_{k^i}}\,\epsilon^i, \qquad \mathbf{k} = \big(k^1, \dots, k^T\big) \in \{0, \dots, K\}^T

$x^i_0$ is the clean, original version of frame $i$ , and $x^i_{k^i}$ is that same frame after $k^i$ steps of the same fixed forward-noising process Diffusion Foundations defines — the only change from the single-frame case is that the noise level $k^i$ is now allowed to differ from frame to frame, collected into a vector $\mathbf{k}$ over the whole sequence instead of a single shared $t$ . $\bar\alpha_{k^i}$ and $\epsilon^i$ carry over their single-frame meaning exactly, just indexed per frame. What this buys the model: choosing $\mathbf{k}$ to be all-equal recovers ordinary full-sequence diffusion, choosing it so past frames sit at $k=0$ while only the newest frame is noisy recovers something close to next-token prediction, and every schedule in between is a point the same trained network can be asked to run at sampling time, without retraining.

Symbol	Meaning
$x^i_0$	the clean, original version of frame $i$
$x^i_{k^i}$	frame $i$ after $k^i$ steps of forward noising
$\mathbf{k} = (k^1,\dots,k^T)$	the vector of independent per-frame noise levels
$\bar\alpha_{k^i}$	the cumulative signal-retention term at noise level $k^i$
$\epsilon^i$	the Gaussian noise added to frame $i$
$T$	the number of frames in the sequence
$K$	the maximum noise level a frame can be assigned

Training samples a noise-level vector at random for every sequence, rather than one shared timestep, and asks the network to recover each frame’s own added noise conditioned on that frame, its noise level, and a running summary of every frame before it — not the whole sequence at once, which is what keeps the model strictly causal:

\mathcal{L}(\theta) = \mathbb{E}_{\mathbf{k},\, x_0,\, \epsilon}\left[ \sum_{i=1}^{T} \big\lVert \epsilon^i - \epsilon_\theta\big(z^{i-1},\, x^i_{k^i},\, k^i\big) \big\rVert_2^{2} \right]

$\epsilon_\theta$ is the network, with trainable parameters $\theta$ , predicting the noise added to frame $i$ from three things: the current frame at whatever noise level $k^i$ it has been given, that noise level itself, and a hidden state $z^{i-1}$ that summarizes only the frames before it. The paper’s own minimal implementation builds $z^{i-1}$ with a plain recurrent network, updating it frame by frame as $z^i \sim p_\theta(z^i \mid z^{i-1}, x^i_{k^i}, k^i)$ — the mechanism that keeps frame $i$ ‘s prediction from ever depending on frame $i+1$ or later, whatever noise-level schedule $\mathbf{k}$ is chosen at sampling time. The expectation is taken over randomly sampled noise-level vectors as well as over data and noise, the same way a standard diffusion loss is taken over randomly sampled timesteps. What this buys the model, per the paper’s own analysis, is a single training objective whose special cases include both a standard full-sequence diffusion loss and something close to an autoregressive next-token loss — so one trained network, not two, can be sampled either way, or anywhere between, and the paper reports this schedule flexibility is what lets a model trained on 36-frame clips roll out past 2,000 frames without diverging.

Symbol	Meaning
$\theta$	the trainable parameters of the denoising network
$z^{i-1}$	the recurrent hidden state summarizing frames before $i$
$\epsilon_\theta(\cdot)$	the network’s predicted noise for a given frame
$i$	the index of the frame whose noise is being predicted
$\mathbb{E}_{\mathbf{k},x_0,\epsilon}$	expectation over random noise-level vectors, data, and noise

The distillation loss that compresses a bidirectional teacher into the 4-step causal student described above is the same distribution-matching-distillation machinery Few-Step Generation teaches for image diffusion, applied to video with an asymmetric teacher-student pairing rather than a new loss function; this chapter does not re-derive it, and Making Video Cheap picks the video-specific pipeline back up in full.

What It Enabled

	Naive autoregressive chaining, 2023	StreamingT2V, March 2024	Causal distillation, December 2024
Mechanism	Condition next chunk on last frame only	Short and long term memory modules per chunk	Bidirectional teacher distilled into causal student
Reported max length	Not extended beyond training length without stagnating	1,200+ frames, roughly two minutes	Streamed indefinitely, one frame at a time
Failure mode addressed	None — this is the failure mode	Scene forgetting and motion stagnation	Autoregressive error accumulation
Throughput	Same as base model per chunk	Same base model, plus enhancer pass	9.4 frames per second on one GPU

Model	Year	Mechanism	Longest disclosed coherent output
Phenaki	2022	Discrete tokens, autoregressive clip re-seeding	Minutes, via changing prompts
Sora	2024	Spacetime patches, joint variable-length diffusion	One minute
StreamingT2V	2024	Chunk-level short and long term memory	1,200+ frames, ~2 minutes
Causal distillation (Yin et al.)	2024	Distilled causal diffusion, KV-cached streaming	Streamed continuously, real time

Every later model in this book’s timeline that generates video for more than a few seconds inherits one of these three mechanisms rather than reinventing memory or causality from scratch. The open-weights models covered in Open Video Models adopt 3D video autoencoders and, in several cases, the same causal-attention pattern the distillation work above popularized, trading some of a fully bidirectional model’s global coherence for tractable long-form and streaming generation. The distillation half of this chapter’s story continues directly into Making Video Cheap, which picks up step-count reduction and causal decoding as an economics problem rather than a consistency one. And Diffusion Forcing’s own stated second application — planning and decision-making, not just video — is the same generalization that action-conditioned world models need: a model that can be sampled either as a next-step predictor or as a full-sequence generator is the mechanism Video Models Become Game Engines and World Models as Foundation Models build interactive rollouts on top of.

Phenaki's token trick and Diffusion Forcing's per-frame theory both run into this chapter's causal distillation, which then forks again. PNG ↓

Key Papers

Paper	Authors	Year	Link	Why it matters
Phenaki: Variable Length Video Generation From Open Domain Textual Description	Villegas et al.	2022	arXiv:2210.02399	Solved autoregressive clip extension for discrete-token video two years before diffusion models needed to solve it again
StreamingT2V: Consistent, Dynamic, and Extendable Long Video Generation from Text	Henschel et al.	2024	arXiv:2403.14773	Added explicit short and long term memory modules to a diffusion backbone, reaching 1,200+ frame video without stagnation
Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusion	Chen et al.	2024	arXiv:2407.01392	Generalized diffusion training to independent per-frame noise levels, unifying next-token and full-sequence generation in one objective
From Slow Bidirectional to Fast Autoregressive Video Diffusion Models	Yin et al.	2024	arXiv:2412.07772	Distilled a bidirectional video diffusion teacher into a causal, KV-cached student streaming at 9.4 FPS with a leading VBench-Long score

PM Lens

The gap I watch for is between what a paper’s architecture makes technically free and what a product still bills for. Diffusion Forcing and the causal-distillation work above both argue that continuing a video should not cost what generating it from scratch costs — that is the entire point of carrying memory or a KV cache forward instead of re-denoising everything. Commercial pricing has not caught up to that argument yet: Runway’s current Gen-4.5 tier charges 60 credits per five seconds of video, and its $12-per-month Standard plan’s 625 monthly credits buy roughly 52 seconds of it, whether that footage is a brand-new clip or the continuation of one already on screen. For a roadmap, that is the tell for how far a research result actually is from a line item — when “extend this video” costs the same per second as “generate a new one,” the memory and causality this chapter covers has not yet been passed through to the price a customer pays, even where it has already changed what is technically possible.

Search