Open Video Models

It is August 2024. OpenAI’s Sora report described a spacetime-patch diffusion transformer six months earlier, but shipped no weights, no code, and no parameter count — a method paper with nothing to download (Sora and Spacetime Patches). The open-weight lineage of video generation is still built on 2023’s bolt-on U-Nets: Stable Video Diffusion tops out at 25 frames. Then, within five months, two research labs release full model weights that implement the DiT recipe for video at a scale nobody outside a closed lab had shipped before.

Where We Were

By mid-2024, video generation had split into two lineages that were not converging. On the closed side, a wave of commercial systems adopted diffusion-transformer architectures behind an API: Google’s Veo was announced in May 2024, Runway’s Gen-3 Alpha and the Kling model both launched in June 2024, each demonstrating multi-second, high-motion clips with no published weights, training data, or architecture diagram (The Commercial Race). On the open side, the newest publicly downloadable video model was still Stable Video Diffusion, released open in November 2023 — a U-Net inherited from Stable Diffusion’s image lineage, fine-tuned with temporal layers, generating 14 or 25 frames from a single conditioning image (Latent Video Diffusion).

The gap between the two sides was architectural, not just a matter of scale. Sora’s report, and DiT before it (Flow Matching and Transformer Backbones), had already established that replacing a U-Net with a patchify-and-attend transformer over a compressed latent produces a model whose output quality scales predictably with training compute. No open-weights video model had adopted that architecture yet. Every open checkpoint anyone could download and fine-tune in mid-2024 was still a convolutional U-Net with motion bolted on after the fact — the compute-scaling curve DiT had demonstrated for images, and Sora’s report claimed for video, was theoretical for anyone without access to a closed lab’s training run.

By early 2025 four teams had independently reproduced the DiT-plus-VAE recipe Sora's report withheld. PNG ↓

That gap was also a data and infrastructure gap, not only a published-paper gap. Training a diffusion transformer on video requires a video autoencoder good enough to compress motion without discarding it, a captioning pipeline dense enough to align text with fast-changing content, and enough accelerator memory and interconnect bandwidth to fit a multi-billion-parameter transformer over a token sequence that a spacetime patch grid inflates far beyond an image model’s. Every open lab that had tried to train at this scale before ran into the same three-way bottleneck: a video autoencoder inherited from image diffusion compressed each frame independently and left motion uncompressed, existing video-text datasets carried captions too short and too generic to teach a model what distinguished one clip’s motion from another’s, and no open-source training stack had been built to hold a spacetime patch sequence — longer than any text transformer’s context window at the time — across enough accelerators at once. Two teams — one at Zhipu AI and Tsinghua University, the other at Tencent — had all three pieces solved by late 2024, and released the results as downloadable weights rather than a technical report alone.

The Breakthrough

CogVideoX, released in August 2024 by the team behind the earlier autoregressive CogVideo — covered in Tokens for Video — was the first open model to pair a spacetime-patch diffusion transformer with a purpose-built video autoencoder (Yang et al., 2024). Its 3D Causal VAE compresses a clip along both space and time at once — rather than compressing each frame independently, the way an image autoencoder repurposed frame-by-frame would, and rather than leaving time uncompressed, the way most 2022–2023 video diffusion U-Nets did — and the causal convolutions inside it are ordered so that decoding a later frame never depends on frames that come after it, which keeps long clips coherent frame-to-frame instead of flickering. On top of that compressed latent, CogVideoX runs an “expert transformer”: a single stack of transformer blocks processes concatenated video and text tokens together, but a per-modality adaptive LayerNorm — separate modulation parameters for the vision tokens and the text tokens, both computed from the shared diffusion timestep — lets the two modalities, which live at very different numerical scales, be normalized differently before they attend to each other. The paper also describes a dedicated video-recaptioning pipeline built to replace the short, generic alt-text that accompanies most web video with denser, motion-specific descriptions, on the premise that a diffusion transformer can only align text and motion as tightly as its training captions describe motion in the first place. The result was released as two open-weight variants: a 2B model under Apache 2.0 that generates up to six seconds at 8 frames per second and 720×480, and a 5B model under a custom research license that reaches CogVideoX’s ceiling of ten seconds at 16 frames per second and 768×1360. Both reached wide adoption within weeks through integration into the Hugging Face Diffusers library and the ComfyUI node-based interface, the same distribution channels Stable Diffusion had used two years earlier.

HunyuanVideo followed in December 2024, released by Tencent as, in the paper’s own framing, the largest open-source video generative model at the time, at more than 13 billion parameters (Kong et al., 2024). Its diffusion transformer uses a dual-stream-to-single-stream design: 20 blocks first process video and text tokens through separate transformer branches, so each modality gets parameters tuned to its own statistics, before 40 further blocks concatenate the two token streams and process them jointly, so cross-modal attention still happens with full depth. A 3D rotary position embedding extends RoPE — the position scheme this book covers in full for text (Long Context) — with separate rotation frequencies along the time, height, and width axes, which is what lets the same trained weights generate at multiple resolutions and durations without retraining a fixed positional grid. Its own 3D VAE compresses by a factor of 4 in time and 8 in each spatial dimension, down to 16 latent channels — the same 4×8×8 compression ratio CogVideoX’s VAE independently converged on, a sign that two teams training on similar video datasets found a similar compute-versus-fidelity tradeoff rather than copying one design from the other.

The same 4x8x8 compression CogVideoX converged on independently is reused here, feeding a 13 billion parameter transformer. PNG ↓

Two more open releases extended the same recipe within the following six months. Mochi 1, released by the startup Genmo in October 2024 under Apache 2.0, introduced an Asymmetric Diffusion Transformer that gives its video branch roughly four times the parameters of its text branch, reaching 10 billion parameters and generating up to 5.4 seconds at 30 frames per second and 480p (Genmo, 2024). Wan, released by Alibaba’s Wan team with public weights in February 2025 and a technical report the following month, shipped four size-and-task variants — a 1.3-billion-parameter model that runs on under 8.2 GB of VRAM alongside a 14-billion-parameter model for text-to-video and image-to-video — and reported the highest overall VBench score of any model, open or closed, evaluated at the time (Wan Team, 2025). By the first quarter of 2025, the DiT-plus-video-VAE recipe Sora’s report had described without releasing a year earlier had been independently reproduced, at open weights, by four separate teams.

The Mathematics

The diffusion training objective and the attention mechanism inside each transformer block are unchanged from what Diffusion Foundations and Sora and Spacetime Patches already teach; nothing in this chapter modifies the DDPM loss or the patchify-and-attend token scheme. What CogVideoX and HunyuanVideo add is a specific, disclosed compression ratio for turning a raw clip into the latent that gets patchified, and a per-modality twist on the adaptive normalization DiT already introduced.

A causal 3D video VAE compresses a clip of $T$ frames at height $H$ and width $W$ into a latent with a smaller time, height, and width extent:

T' = \frac{T - 1}{c_t} + 1, \qquad H' = \frac{H}{c_s}, \qquad W' = \frac{W}{c_s}

$T$ , $H$ , and $W$ are the raw clip’s frame count, height, and width; $c_t$ and $c_s$ are the VAE’s temporal and spatial compression factors — 4 and 8 for both CogVideoX’s and HunyuanVideo’s VAEs. The $-1$ and $+1$ around $T$ come from the causal convolution treating the first frame specially, so it can be decoded on its own without needing “future” frames that do not exist yet at the start of a clip. $T'$ , $H'$ , and $W'$ are the resulting latent’s dimensions, the ones that actually get cut into spacetime patches for the diffusion transformer, exactly as Sora and Spacetime Patches describes for the patch-token count $N$ . What this buys the model: a 4×8×8 reduction turns a clip with hundreds of frames at HD resolution into a latent small enough that a patch-and-attend transformer can afford to attend over the whole sequence, the same problem an image latent autoencoder solves for a single frame, now solved along a third axis.

Symbol	Meaning
$T, H, W$	raw clip frame count, height, and width
$c_t, c_s$	the VAE’s temporal and spatial compression factors
$T', H', W'$	the compressed latent’s time, height, and width extent

DiT’s adaptive LayerNorm, taught where DiT itself is introduced (Flow Matching and Transformer Backbones), computes a single set of scale and shift parameters from the diffusion timestep and conditioning signal and applies them to every token alike. CogVideoX’s expert transformer instead computes two separate sets — one for video tokens, one for text tokens — from the same shared embedding:

y_{\mathrm{vid}} = \mathrm{LN}(x_{\mathrm{vid}}) \cdot \big(1 + \gamma_{\mathrm{vid}}(e)\big) + \beta_{\mathrm{vid}}(e), \qquad y_{\mathrm{txt}} = \mathrm{LN}(x_{\mathrm{txt}}) \cdot \big(1 + \gamma_{\mathrm{txt}}(e)\big) + \beta_{\mathrm{txt}}(e)

$x_{\mathrm{vid}}$ and $x_{\mathrm{txt}}$ are the video and text token embeddings inside a shared transformer block; $e$ is the conditioning embedding built from the diffusion timestep; $\mathrm{LN}$ is ordinary layer normalization; and $\gamma_{\mathrm{vid}}, \beta_{\mathrm{vid}}, \gamma_{\mathrm{txt}}, \beta_{\mathrm{txt}}$ are separate small MLPs — the “experts” — that map the same $e$ to a different scale and shift for each modality. What this buys the model: video and text embeddings enter a shared attention layer at very different numerical scales, and normalizing them with one shared set of parameters, the way plain adaLN does, would force one modality’s statistics onto the other; a per-modality expert keeps both streams individually well-scaled while still sharing every attention weight and every other parameter between them, so the same block does the cross-modal fusion DiT’s shared adaLN cannot.

Symbol	Meaning
$x_{\mathrm{vid}}, x_{\mathrm{txt}}$	video and text token embeddings entering the block
$e$	the shared conditioning embedding, from the diffusion timestep
$\gamma_{\mathrm{vid}}, \beta_{\mathrm{vid}}$	the video-expert scale and shift, functions of $e$
$\gamma_{\mathrm{txt}}, \beta_{\mathrm{txt}}$	the text-expert scale and shift, functions of $e$
$y_{\mathrm{vid}}, y_{\mathrm{txt}}$	the normalized, modality-scaled outputs

What It Enabled

The four open releases did not match, still less exceed, every closed model on every axis — HunyuanVideo’s own human-evaluation table, reproduced below, still shows one closed comparison model scoring marginally higher on text alignment — but they gave the field something the Sora report alone could not: weights anyone could download, fine-tune, quantize, or distill.

	CogVideoX-5B	HunyuanVideo	Mochi 1	Wan 2.1
Released	August 2024	December 2024	October 2024	February 2025
Organization	Zhipu AI / Tsinghua (THUDM)	Tencent	Genmo	Alibaba
Parameters	~5B (2B variant also released)	13B	10B	1.3B and 14B variants
Max resolution / duration	768×1360, up to 10s at 16fps	720×1280, up to 129 frames	480p, up to 5.4s at 30fps	up to 720p, multiple durations
VAE compression	4× temporal, 8×8 spatial	4× temporal, 8×8 spatial	AsymmDiT, no separate ratio disclosed	disclosed in technical report
License	Apache 2.0 (2B) / custom (5B)	Tencent community license	Apache 2.0	Apache 2.0

HunyuanVideo’s own professional human evaluation, run over 1,533 prompts with 60 evaluators, put it ahead of the closed models it was benchmarked against on overall preference:

Model	Text alignment	Motion quality	Overall preference	Rank
HunyuanVideo	61.8%	66.5%	41.3%	1st
Closed Chinese model (API)	62.6%	61.7%	37.7%	2nd
Closed Chinese model (web)	60.1%	62.9%	37.5%	3rd
Runway Gen-3 Alpha	47.7%	54.7%	27.4%	4th
Luma 1.6	57.6%	44.2%	24.8%	5th

That result — an openly released model outscoring closed API products it was never trained to imitate — is the concrete version of the “Stable-Diffusion moment for video” this chapter’s title claims: the point after which a research team, a startup, or a hobbyist with enough GPU memory could run, fine-tune, or build a product on a model competitive with what a well-funded lab was charging per second to access. Open weights, once released, also compress faster than closed ones: CogVideoX and HunyuanVideo weights became the base checkpoints the field’s step-distillation and quantization work targets, because there was no closed API to distill from (Making Video Cheap). And the progressive multi-resolution training CogVideoX uses to reach 10 coherent seconds, and the frame-count ceilings all four models still hit, both feed directly into the harder problem of generating video that stays consistent well past ten seconds (Long Video and Consistency).

Wan’s release also widened what an open video model was expected to do out of the box. Rather than shipping text-to-video alone, its technical report describes up to eight downstream tasks built on the same backbone — image-to-video, instruction-guided video editing, and personalized generation among them — packaged as one open suite rather than one narrow checkpoint. That breadth, combined with the 1.3-billion-parameter variant’s sub-8.2 GB VRAM footprint, moved the open lineage’s floor from “runnable on a data-center accelerator” to “runnable on a single consumer GPU,” a threshold Stable Diffusion had already crossed for images in 2022 and that video was crossing for the first time only in early 2025.

Mochi and Wan complete the recipe, but this chapter's later citations run through CogVideoX and HunyuanVideo. PNG ↓

Key Papers

Paper	Authors	Year	Link	Why it matters
CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer	Yang et al.	2024	arXiv:2408.06072	First open-weights video diffusion transformer, pairing a 3D causal VAE with per-modality adaptive LayerNorm
HunyuanVideo: A Systematic Framework For Large Video Generative Models	Kong et al.	2024	arXiv:2412.03603	Largest open-weights video model at release, dual-stream-to-single-stream DiT with 3D RoPE, beat closed competitors in human evaluation

PM Lens

The number that matters here is not parameter count, it is licensing. CogVideoX-5B ships under a custom research license while its 2B sibling, Mochi 1, and Wan 2.1 all ship Apache 2.0 — the difference between “evaluate this” and “put this in a product.” When a team pitches “self-host our own video model,” the real comparison is not accelerator cost versus a subscription, it is accelerator cost plus fine-tuning and serving engineering time versus a metered API like Sora’s, priced per second of output. Open weights only win that comparison if the team fine-tuning them can actually reach the closed model’s quality bar within their compute budget — HunyuanVideo’s human-evaluation win over Gen-3 Alpha is the first time that bar was demonstrably cleared by a model anyone could download, which is why it, more than CogVideoX, is the release worth citing when a roadmap review asks whether open video is production-ready yet.

Search