Video Models Become Game Engines

It is February 2024. Text-to-video models can already produce several plausible seconds of footage from a written prompt, but every one of them generates in a single direction: a fixed prompt goes in, a fixed clip comes out, and nothing a viewer does while it plays changes a single pixel. Two papers published six months apart — one from Google DeepMind, one from a two-person team inside Google — take the same tokenizers and denoising networks this book has already introduced and add one new input: an action, supplied frame by frame, from whoever is watching. The result is a video model a person can walk around inside.

Where We Were

The idea of learning a simulator instead of hand-coding one is not new to this chapter. Chapter 3 traces it back to March 2018, when Ha and Schmidhuber’s World Models compressed each frame of a racing game into a small latent vector and trained a recurrent network to dream up the next one, well enough that a policy trained entirely inside the dream could then drive the real game. Hafner and collaborators spent the following two years turning that proof of concept into a repeatable recipe — PlaNet, then Dreamer — that beat model-free reinforcement learning on continuous-control benchmarks by planning inside a learned latent model rather than the real environment.

Two things were true of every model in that lineage, and both were limits rather than choices. First, they were cheap: a convolutional VAE and a small recurrent network, trained in an hour or so on a single GPU. Second, they were built for control, not for looking at. Their latent codes were compressed enough to plan with, not to render anything a human would mistake for real footage, and the environments they modeled — CarRacing, a handful of Atari games, DeepMind Control Suite tasks — were toy problems chosen because a lightweight model could master them, not because anyone expected to watch the result.

Meanwhile, the video-generation side of this book had been solving the opposite half of the problem. Diffusion-based video models had learned to tokenize and denoise their way to genuinely convincing footage — first with cascaded pixel-space diffusion, then with the latent and space-time architectures covered in Chapters 5, 6, and 9, culminating in OpenAI’s Sora, announced the same month as the first paper in this chapter. Sora could render a minute of coherent, physically plausible video from a paragraph of text. It could not be steered once generation started; there was no notion of a player, a keyboard, or a frame-by-frame decision that changed the outcome. Fidelity and interactivity had been developed on separate tracks, at different scales, for six years, and nobody had put them in the same model.

The Breakthrough

Genie (Bruce et al., 2024), posted to arXiv in February 2024 by a team at Google DeepMind, is built from three pieces trained jointly, none of which ever sees an action label during training. A spatiotemporal video tokenizer (ST-ViViT, roughly 200M parameters) compresses each frame into discrete codes drawn from a codebook of 1,024 entries — the same discrete-codebook idea Chapter 5 of Image Generation introduces for VQ-VAE, applied here to video. A latent action model (LAM, roughly 300M parameters) looks at every pair of consecutive frames and infers, without ever being told what a “jump” or “move left” is, a single discrete code from a codebook of just 8 entries that best explains the transition between them. A dynamics model — a 10.1-billion-parameter decoder-only transformer, by far the largest of the three components — then learns to predict the tokens of the next frame given all previous frame tokens and all previous inferred actions.

Genie was trained on a dataset the paper calls Platformers: 30,000 hours of curated footage drawn from an initial pool of 55 million publicly available internet videos, filtered down to 6.8 million 16-second clips at 10 frames per second and 160×90 resolution. Training the full dynamics model used 256 TPU v5p chips over 125,000 steps, consuming roughly 942 billion tokens and an estimated 6.6 × 10²² floating-point operations for that component alone. At inference time, a user supplies one of the 8 latent-action codes on every step and Genie renders the frame that code implies — a genuinely novel result, since nothing in training ever saw a labeled action, and the same recipe transfers unchanged from platform games to robotics footage the paper also tests it on. It is also, by the paper’s own account, not real time: Genie ran at roughly 1 frame per second at publication, explicitly flagged as a target for future speedups rather than a solved problem.

GameNGen (Valevski et al., 2024), posted six months later in August 2024, solves the speed half of the problem that Genie leaves open, at the cost of generality: it targets one specific game, DOOM, rather than an open-ended corpus of internet video. Training runs in two phases. First, a Proximal Policy Optimization agent — the same PPO algorithm Chapter 13 of Language Models covers for reward-model optimization, here applied to gameplay reward instead of a learned preference model — plays DOOM for 10 million environment steps, and every frame and action produced during both training and evaluation is recorded, yielding roughly 900 million frames of training data. Second, a diffusion model built by repurposing Stable Diffusion 1.4 — the latent-diffusion architecture Chapter 13 of Image Generation covers — is trained to predict the next frame’s latent conditioned on a window of the past 64 frames and the actions that produced them, roughly 3.2 seconds of history at the model’s operating rate.

Two engineering choices make GameNGen usable rather than merely accurate. The first is noise augmentation: during training, the past frames used as conditioning context are deliberately corrupted with Gaussian noise at a randomly sampled level (up to 0.7, drawn from one of 10 discrete buckets), and the network is told what level was used. Without this, a model trained only ever to condition on perfectly clean context frames drifts and degrades once it has to condition on its own slightly-imperfect earlier outputs — exactly the failure mode that running any model on its own generations, thousands of times in a row, invites. The second is a small fix at the end of the pipeline: the pretrained Stable Diffusion autoencoder produces visible artifacts on game-specific detail, particularly the heads-up-display bar at the bottom of the screen, so the paper fine-tunes just the decoder half of that autoencoder with a plain reconstruction loss to clean those artifacts up. The training run for the generative model used 128 TPU v5e chips; a single TPU handles inference. At just 4 DDIM sampling steps — the deterministic accelerated sampler Chapter 9 of Image Generation covers — the model runs at 20 frames per second on that single chip, with the paper reporting no measurable quality loss relative to 20 or more steps. Next-frame prediction reaches a PSNR of 29.4, comparable to lossy JPEG compression at quality settings of 20–30, and in a blind test, ten human raters shown short real-DOOM and GameNGen clips side by side picked the real game only 58% of the time for 1.6-second clips and 60% of the time for 3.2-second clips — both barely above the 50% chance floor.

Two more systems, published either side of GameNGen, show the same idea generalizing in different directions. DIAMOND (Alonso et al., 2024), posted in May 2024 and later a NeurIPS Spotlight, argues against Genie’s own design choice: rather than tokenizing frames into a discrete codebook before modeling their dynamics, DIAMOND runs diffusion directly in pixel space, on the grounds that discrete tokenizers can throw away small visual details — a single bullet or a distant enemy — that matter enormously for the reward signal a reinforcement-learning agent is trying to maximize, even though they barely move the reconstruction loss a tokenizer is trained against. An agent trained purely inside a DIAMOND world model reaches a mean human-normalized score of 1.46 on the Atari 100k benchmark, 46% above human performance and a new best for agents trained without ever touching the real environment beyond 100,000 frames; the same recipe, applied to a fixed 87-hour dataset of human CS:GO gameplay rather than an online RL loop, produces a playable, if not real-time-trained, world model of a completely different genre of game.

Oasis (Decart and Etched, 2024), released as a technical report and public browser demo on October 31, 2024, is the first of these systems built explicitly as a product rather than a research result. Trained on Minecraft gameplay footage, Oasis pairs a transformer-based spatial autoencoder with a transformer-based latent diffusion backbone, generating each frame autoregressively conditioned on what came before and on the player’s keyboard input — the same DiT design Chapter 19 of Image Generation introduces for image diffusion, adapted to a causal, frame-by-frame setting, plus a dynamic-noising scheme at inference that varies the noise level across the diffusion pass to limit error accumulation over long autoregressive runs. The publicly released checkpoint has 500 million parameters (a larger model powered the live demo); it renders a playable, first-person Minecraft-like world at 20 frames per second, which the team describes as needing a new frame roughly every 40 milliseconds to stay real time — inside the roughly 50-millisecond-per-frame budget a 20-fps target allows. There is no game engine underneath it: every frame, including the on-screen inventory and lighting, is a diffusion sample.

The Mathematics

Genie’s dynamics model is trained the same way any autoregressive sequence model is — the factorization Chapter 5 of Image Generation already covers for pixels — except that the conditioning variable at every step is an action the model itself invented during training:

$$p_\theta\big(z_1, \dots, z_T \mid \hat a_1, \dots, \hat a_{T-1}\big) = \prod_{t=1}^{T} p_\theta\big(z_t \mid z_{<t}, \; \hat a_{<t}\big)$$

Symbol	Meaning
$z_t$	Discrete tokenizer codes for frame $t$ (codebook size 1,024)
$\hat a_t$	Inferred latent action between frames $t$ and $t{+}1$ (codebook size 8)
$\theta$	Dynamics-model parameters (10.1B)
$T$	Number of frames in the training clip

GameNGen trains its denoising network on the same forward-noising process Chapter 8 of Image Generation derives in full, but targets a velocity reparameterization rather than the added noise directly, with one addition aimed squarely at surviving thousands of consecutive autoregressive steps:

$$\mathcal{L}(\theta) = \mathbb{E}_{x_0, \, \epsilon, \, k, \, k_c}\Big[\, \big\| v^{(k)} - v_\theta\big(x_0^{(k)}, \, k, \, \tilde c^{(k_c)}, \, k_c\big) \big\|_2^2 \,\Big], \qquad v^{(k)} = \sqrt{\bar\alpha_k}\,\epsilon - \sqrt{1-\bar\alpha_k}\,x_0$$

Symbol	Meaning
$x_0$	Clean target frame latent
$k$	Diffusion noise level applied to the target
$\epsilon$	True noise added by the forward process
$v^{(k)}, v_\theta$	Velocity target and the network’s predicted velocity
$\tilde c^{(k_c)}$	Past frames and actions, corrupted with noise level $k_c$
$k_c$	Sampled context noise level, given to the network as an embedding

What It Enabled

The four systems in this chapter split cleanly along a fidelity-versus-generality axis: the more open-ended the domain, the further from real time the result.

System	Public	Trained on	Headline result
Genie	Feb 2024	30,000 hrs of internet platformer video	10.7B params, 8-way latent action control, ~1 FPS
DIAMOND	May 2024	Online RL (Atari) plus 87 hrs of CS:GO footage	1.46 mean human-normalized score on Atari 100k
GameNGen	Aug 2024	900M PPO-agent DOOM frames	20 FPS on one TPU, PSNR 29.4, near-chance human detection
Oasis	Oct 2024	Minecraft gameplay footage	500M params, 20 FPS, ~40 ms/frame

That split maps directly onto where the field went next. The narrow, single-game route that GameNGen and Oasis took — pick one environment, throw enough recorded gameplay at a diffusion model, and optimize hard for frame-rate — is the more direct path to a shippable interactive product, and it is also the path Chapter 14 picks up when it covers the step-distillation and causal-decoder techniques that push video generation toward real time more generally. The broad, action-agnostic route Genie took — learn a controllable simulator from unlabeled video at internet scale, with no notion of “the game” baked in — is the one Chapter 16 follows forward, through Genie’s own second and third generations and a wave of foundation-scale world models aimed at robotics and embodied agents rather than entertainment.

Key Papers

Paper	Authors	Year	Link	Why it matters
Genie: Generative Interactive Environments	Bruce et al.	2024	arXiv:2402.15391	First foundation-scale, action-controllable world model learned from unlabeled internet video, with no action labels at training time
Diffusion Models Are Real-Time Game Engines	Valevski, Leviathan, Arar, and Fruchter	2024	arXiv:2408.14837	Shows a fine-tuned Stable Diffusion checkpoint can run a full game at 20 FPS on one chip, indistinguishable from the real thing to most human raters
Diffusion for World Modeling: Visual Details Matter in Atari	Alonso et al.	2024	arXiv:2405.12399	Argues pixel-space diffusion beats discrete-token world models for reinforcement learning because small visual details carry reward-relevant information
Oasis: A Universe in a Transformer	Decart and Etched	2024	oasis-model.github.io	First publicly playable real-time diffusion world model shipped as a product, with an open-sourced 500M-parameter checkpoint