Tokens for Video

It is April 2021. GANs have spent five years chasing convincing motion and mostly delivered short, low-resolution clips that fall apart under scrutiny; the discrete-codebook trick that just gave still images a tractable transformer prior has not yet been pointed at video. Over the next eighteen months, three papers apply that trick to time itself, and the last of them generates videos minutes long from a sentence that changes partway through.

Where We Were

Video generation’s adversarial branch had, by 2021, produced a string of steadily larger models without solving the underlying problem. VGAN in 2016 first tried scene dynamics with a two-stream generator; MoCoGAN in 2017 split content from motion; DVD-GAN in 2019 scaled the discriminator and generator to Kinetics-scale data — all covered in GAN Video. Every one of them inherited the GAN’s core property: a discriminator supplies a training signal, but nothing in the setup assigns a number to how probable a given video actually is, and nothing prevents motion from drifting into incoherence a few frames past what the discriminator was shown during training.

Meanwhile, image generation had already solved an adjacent problem. Pixels One at a Time traces how PixelRNN and PixelCNN treated an image as a sequence and scored it with an exact likelihood, then how VQ-VAE fixed autoregressive sampling’s brutal cost — one network call per raw pixel — by first compressing an image into a small grid of discrete codebook indices, then running the autoregressive prior over that short grid of tokens instead of over pixels directly. A 128×128 image’s 49,152 pixels became roughly 1,024 tokens. The same chapter traces that codebook idea forward into VQGAN and DALL-E for still images. Video is a stack of images plus a time axis, and by 2021 the obvious question was whether the same compress-then-predict recipe would survive the extra dimension — whether a video’s tokens, once you had them, behaved enough like a language model’s tokens that a GPT-style transformer could just be pointed at them.

Three years of scaling GANs never solved drifting motion. Borrowing VQ-VAE's codebook trick and running a transformer over it closed the gap in eighteen months. PNG ↓

Three constraints stood in the way of a naive answer. First, cost: a raw video clip has an order of magnitude more values than a single image, so any tokenizer had to compress far more aggressively than VQ-VAE did for stills, or the resulting token grid would still be too long for a transformer to attend over. Second, data: large, well-captioned video-text datasets did not exist at anything like the scale of image-text pairs scraped from the web, which meant a video model trained from scratch on video alone was starved relative to its image-generation cousins. Third, length: real videos have no fixed duration, but every tokenizer and every transformer built so far assumed a fixed-size input. VideoGPT, CogVideo, and Phenaki answer these three constraints in that order — one paper each — across eighteen months.

The Breakthrough

VideoGPT (Yan et al., 2021) is the direct video analogue of VQ-VAE plus a GPT prior, and its main engineering contribution is making that combination survive the extra time axis. The tokenizer replaces VQ-VAE’s 2D convolutions with 3D convolutions, so a clip is downsampled in height, width, and time simultaneously, and adds axial self-attention inside the encoder and decoder — attention restricted to run along one axis (height, width, or time) at a time, rather than jointly over every position in the 3D volume — which keeps the attention cost from exploding as a cube instead of a square. On BAIR robot-pushing videos the encoder compresses a 16-frame, 64×64 clip down to an 8×32×32 grid of codes drawn from a 1,024-entry codebook; on the harder UCF-101 and Tumblr GIF Dataset (TGIF) benchmarks, the grid shrinks to 4×32×32. A GPT-style transformer — up to 20 layers, hidden size up to 1,024, depending on the dataset — is then trained autoregressively over that grid using spatio-temporal position encodings, so the model knows not just where in a frame a token sits but which frame it belongs to.

The token grid is the whole trick: once a clip is a few thousand codes instead of millions of pixels, an off-the-shelf GPT prior can run over it. PNG ↓

The result, on BAIR, was an FVD (Fréchet Video Distance, the metric Evaluating and Monetizing Motion covers in full) of 103.3 — matching TrIVD-GAN-FP’s 103.3 and beating DVD-GAN-FP’s 109.8, the two strongest adversarial baselines available at the time. A likelihood-trained, discrete-token model had reached parity with the best GANs on a standard video benchmark, using an architecture with none of adversarial training’s instability. VideoGPT did not solve the data problem, though: it trained one model per dataset, from scratch, on datasets ranging from thousands to low hundreds of thousands of clips, with no path to leveraging the much larger corpora of captioned still images that image generation was already exploiting.

CogVideo (Hong et al., 2022) answers that data problem directly, thirteen months later, by refusing to train from scratch at all. Rather than learning spatial structure — what an object looks like, how a scene is composed — from video data alone, CogVideo starts from CogView2, an already-trained 6-billion-parameter text-to-image transformer, and freezes every one of its parameters. At each transformer layer, a new temporal attention channel is added alongside the frozen spatial channel, and only that new channel is trained. The two channels’ outputs are mixed by a learned, per-channel gate, so the model can lean on CogView2’s frozen spatial knowledge for what each frame should look like while learning, from scratch but with far fewer new parameters, how frames should move. The resulting model — 9.4 billion parameters total, 6 billion of them frozen — trained on 5.4 million captioned video clips at 160×160 resolution (later upsampled to 480×480), generating 32-frame clips. In a 90-evaluator human preference study on UCF-101-derived prompts, evaluators picked CogVideo’s output as best 49.53% of the time, against 15.42% for VideoGPT and 5.6% for TGANv2 — the largest reported margin between a token-based video model and both an earlier token-based model and a GAN baseline.

Six of CogVideo's 9.4 billion parameters never see a gradient. Only the temporal channel and the gate that blends it with frozen spatial knowledge are trained. PNG ↓

CogVideo’s second contribution, multi-frame-rate hierarchical training, targets the weak text-video alignment in scraped caption data: a caption like “a dog runs across a yard” is compatible with hundreds of different frame rates, and training on a mismatched one teaches the model that the caption predicts the wrong amount of motion per frame. CogVideo tags each training clip with the lowest frame-rate token for which at least five frames can be sampled from the clip, generates a small set of low-frame-rate keyframes first, and then recursively interpolates additional frames at higher frame rates in a second stage — factoring the sequential-generation cost into a short, coarse pass followed by cheap infilling, rather than one long autoregressive pass at the final frame rate.

Phenaki (Villegas et al., 2022) addresses the third constraint — a fixed-size assumption that cannot handle real, arbitrary-length video — with a redesigned tokenizer and a different generation mechanism entirely. Its tokenizer, C-ViViT (a causal variant of the Video Vision Transformer), uses causal attention along the time axis only: encoding frame $t$ can attend to frames $1$ through $t$ , never to frames after it. That one constraint means the tokenizer’s input length is not baked into its weights — it can encode a 5-frame clip or a 500-frame clip through the identical causal mechanism, and, because the encoding of frame $t$ never depends on frames beyond it, the model can also encode single captioned still images as one-frame “videos,” which is how Phenaki’s training mixes roughly 15 million 8-frames-per-second video-text pairs with about 50 million image-text pairs and, further, about 400 million image-text pairs from LAION-400M — supplying by far the largest fraction of training signal from data that was never a video at all.

Rather than predicting tokens one at a time left to right the way VideoGPT and CogVideo do, Phenaki generates a clip’s token grid with a bidirectional masked transformer, conditioned on text embeddings: during training, a random fraction of the token grid is masked out and the model is trained to fill in the missing tokens from the ones left visible and the prompt, in any order, rather than only from tokens strictly before them in a fixed scan. At generation time this lets the model refill the entire grid in a handful of parallel iterative rounds — the paper uses 12 to 48 — rather than one sequential step per token. Because C-ViViT’s causal structure lets a new clip start from the last frames of a previous one, Phenaki can also extend a video autoregressively at the clip level, re-encoding the tail of what it just generated as the seed for the next clip, and re-conditioning on a new sentence each time. Trained on 1.4-second clips, the resulting model generates coherent video minutes long from a sequence of prompts that changes partway through — the “variable length” and “open domain” of the paper’s title, and, per the paper’s own framing, the first published system to generate video from a sequence of prompts that changes over the course of the clip rather than a single fixed caption.

The loop, not the transformer, is what makes the video open-ended: each pass hands its own last frames back in as the next segment's starting point. PNG ↓

On Kinetics-600 video prediction, Phenaki reports an FVD of 36.4 — a large improvement over VideoGPT’s 103.3 on the easier BAIR benchmark a year earlier, though the two numbers are not measured on the same dataset and are not directly comparable as a single ranking. On Moments-in-Time, the C-ViViT tokenizer alone reconstructs video at an FVD of 65.78 while using roughly 40% fewer tokens per clip than a comparable per-frame tokenizer, which is the efficiency gain that makes minutes-long generation computationally reachable at all.

The Mathematics

VideoGPT and CogVideo both train an exact-likelihood autoregressive prior over a discrete token grid, which is the same chain-rule factorization Pixels One at a Time introduces for a 1D pixel sequence, extended here to a 3D grid indexed by time as well as by the two spatial axes:

p(z) = \prod_{t=1}^{T} \prod_{i=1}^{H \times W} p\big(z_{t,i} \mid z_{<t}, \, z_{t,<i}\big)

$z$ is the full grid of discrete video tokens the tokenizer produced for one clip; $T$ is how many time steps the grid has, and $H \times W$ is how many token positions sit inside each time step. $z_{t,i}$ is one specific token, at time step $t$ and spatial position $i$ within that time step. The factorization says: predict every token in time step 1, in raster order, before moving to time step 2, and within any time step condition on everything already generated — every earlier time step in full ( $z_{<t}$ ), plus everything already generated within the current time step ( $z_{t,<i}$ ). This is the same chain-rule identity used for pixels, with no new approximation — the only change is what counts as “earlier”: time, then within-frame position. What it buys you is exactly what VQ-VAE bought for images: because $z$ has orders of magnitude fewer entries than the raw video it was compressed from, the number of sequential steps a transformer needs to sample one clip drops from millions of pixel values to thousands of tokens.

Symbol	Meaning
$z$	The full grid of discrete tokens for one video clip
$T$	Number of time steps in the token grid
$H \times W$	Number of token positions per time step
$z_{t,i}$	The token at time step $t$ , spatial position $i$
$z_{<t}$	All tokens from every time step before $t$
$z_{t,<i}$	Tokens already generated earlier within time step $t$

CogVideo’s departure from a plain autoregressive prior is architectural, not a change to that factorization: at every transformer layer, a frozen spatial attention channel inherited from CogView2 is mixed with a newly trained temporal channel through a learned gate,

\tilde{x} = \alpha \odot \mathrm{Attn}_{\mathrm{base}}\big(\mathrm{LN}(x)\big) + (1-\alpha) \odot \mathrm{Attn}_{\mathrm{plus}}\big(\mathrm{LN}(x)\big), \qquad \alpha = \sigma(a)

$x$ is the layer’s input, and $\mathrm{LN}$ is layer normalization. $\mathrm{Attn}_{\mathrm{base}}$ is CogView2’s original attention block, carried over with its weights frozen; $\mathrm{Attn}_{\mathrm{plus}}$ is a new attention block, initialized from scratch and trained on video, that learns temporal relationships between frames. $\alpha$ is a learned vector, squeezed into the range $(0,1)$ by the sigmoid function $\sigma$ from a raw trainable parameter $a$ , and $\odot$ is elementwise multiplication — so each dimension of the layer’s output gets its own learned blend of “old spatial knowledge” versus “new temporal knowledge.” What it buys you: 6 billion of CogVideo’s 9.4 billion parameters never need a training gradient at all, because they already encode what CogView2 learned about how objects and scenes look; the video-specific learning problem shrinks to whatever $\mathrm{Attn}_{\mathrm{plus}}$ and $\alpha$ need to capture about motion, which is a far smaller and more data-efficient problem than learning spatial appearance and motion together from a much smaller video corpus.

Symbol	Meaning
$x$	Input to the transformer layer
$\mathrm{LN}(\cdot)$	Layer normalization
$\mathrm{Attn}_{\mathrm{base}}$	Frozen spatial attention channel, inherited from CogView2
$\mathrm{Attn}_{\mathrm{plus}}$	Newly trained temporal attention channel
$\alpha$	Learned per-dimension mixing gate, in $(0,1)$
$a$	Raw trainable parameter feeding the gate
$\sigma(\cdot)$	Sigmoid function
$\tilde{x}$	The layer’s mixed attention output

Phenaki’s generator abandons the chain-rule factorization above entirely in favor of masking a random subset of the token grid and training the model to recover it from everything else and the text prompt:

\mathcal{L}_{\mathrm{mask}} = -\, \mathbb{E}_{z, \gamma}\left[ \sum_{i \in m(\gamma)} \log p_\theta\big(z_i \mid z_{\bar{m}(\gamma)}, c\big) \right], \qquad \gamma \sim \mathrm{Uniform}(0, 1)

For a given clip’s token grid $z$ and a sampled masking ratio $\gamma$ between 0 and 1, $m(\gamma)$ is the random set of token positions that get hidden, and $\bar{m}(\gamma)$ is the complement — the positions left visible. $c$ is the text prompt’s embedding, and $p_\theta$ is the transformer’s predicted probability, with parameters $\theta$ , for a hidden token given the visible ones and the prompt. The loss is the negative log-probability the model assigns to the true value of every hidden token, averaged over many random draws of which tokens got hidden. What it buys you is the opposite trade from the autoregressive factorization above: instead of one exact, ordered factorization that must be sampled strictly in sequence, the model learns to fill in any subset of missing tokens from any other subset, which at generation time lets it start from an entirely masked grid and fill it in over a small, fixed number of parallel rounds — 12 to 48 in Phenaki’s reported setup — rather than one sequential pass per token. It is the same speed-versus-exactness trade Pixels One at a Time describes for pixel-by-pixel sampling, resolved in the opposite direction: give up the exact chain-rule factorization to collapse thousands of sequential steps into dozens of parallel ones.

Symbol	Meaning
$z$	The clip’s full token grid
$\gamma$	Sampled masking ratio, drawn uniformly from 0 to 1
$m(\gamma)$	The set of masked, hidden token positions
$\bar{m}(\gamma)$	The set of visible, unmasked token positions
$z_i$	A single hidden token being predicted
$c$	The text prompt’s embedding
$\theta$	The transformer’s trainable parameters
$p_\theta(\cdot)$	The model’s predicted probability for a hidden token

What It Enabled

The three papers trace a clean progression: solve the tokenization problem for video, then solve the data problem by inheriting an image model, then solve the fixed-length problem by making the tokenizer causal and the generator non-sequential.

	VideoGPT, 2021	CogVideo, 2022	Phenaki, 2022
Tokenizer	3D-convolutional VQ-VAE, axial attention	Inherits CogView2’s discrete image tokenizer	C-ViViT, causal in time
Generation mechanism	Autoregressive GPT prior	Autoregressive, dual-channel attention	Bidirectional masked transformer
Parameters	Not disclosed at a headline figure	9.4B total, 6B frozen from CogView2	1.8B (0.9B variant for direct comparison)
Training data	Per-dataset, thousands to ~100K clips	5.4M captioned video clips	~15M video-text pairs, ~450M image-text pairs
Max reported clip length	16 frames	32 frames	Minutes, via autoregressive clip extension
Reported quality signal	FVD 103.3 on BAIR	49.53% human preference vs. 15.42% for VideoGPT, on UCF-101	FVD 36.4 on Kinetics-600

Two lineages fork from this chapter. The first stayed with tokens and transformers: CogVideo’s direct successor, CogVideoX, returned in 2024 with a 3D variational autoencoder and an expert transformer, covered in Open Video Models, and the broader idea of tokenizing a signal into a compact grid before running a transformer over it resurfaces again, in continuous rather than discrete form, in the “spacetime patches” behind Sora, covered in Sora and Spacetime Patches. The second lineage left tokens behind: within months of Phenaki, Video Diffusion Models applied the denoising process already dominant in image generation directly to video, and for the next two years diffusion, not autoregressive or masked token prediction, became the field’s default approach — largely because a diffusion model sidesteps both the strict sequential-sampling cost of VideoGPT and CogVideo’s autoregressive priors and the separate tokenizer-training step that any token-based approach requires.

Key Papers

Paper	Authors	Year	Link	Why it matters
VideoGPT: Video Generation using VQ-VAE and Transformers	Yan et al.	2021	arXiv:2104.10157	First to point VQ-VAE tokenization plus a GPT-style prior at video; matched GAN-era FVD on BAIR without adversarial training
CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformers	Hong et al.	2022	arXiv:2205.15868	Inherited a frozen pretrained text-to-image transformer via dual-channel attention, solving the video-data scarcity problem
Phenaki: Variable Length Video Generation From Open Domain Textual Description	Villegas et al.	2022	arXiv:2210.02399	Causal C-ViViT tokenizer plus a bidirectional masked transformer generated minutes-long video from a changing prompt sequence

PM Lens

What I take from this chapter is a lesson about where to spend a scarce dataset. CogVideo’s real contribution was not a bigger transformer — it was recognizing that 5.4 million captioned videos is a small dataset by the standards of image-text pretraining, and that freezing 6 of 9.4 billion parameters and training only a temporal add-on got more out of that small dataset than training everything from scratch would have. I have seen the same call show up in roadmap reviews for any team building a model on top of a scarcer, more expensive modality than text or images: video, audio, sensor data, proprietary transaction logs. The question worth asking before committing a training budget is not “how much data do we have” in isolation, it is “how much of what this model needs to learn can we inherit from a model already trained on an adjacent, more abundant modality, and how little do we actually need to learn from scratch.” Phenaki’s answer went further still — training on roughly 30 times more image-text pairs than video-text pairs — which is the same bet taken to its logical end.

Search