World Models as Foundation Models

It is January 2025. A video model that can render a playable game world, demonstrated in 2024, is a proof of concept; it is not yet infrastructure anyone can build a robot or a self-driving stack on top of. Over the following eight months, three separate labs converge on the same ambition from three different directions — generate the world, predict it without ever rendering it, or let a person step inside it in real time — and all three start calling the result a foundation model, borrowing the term language models had already made mean “pretrain once, adapt everywhere.”

Where We Were

By late 2024, the world-model concept this book has tracked since 2018 (Predicting the Next Frame) had just acquired a new demonstration of scale. Genie (2024-02) had shown that a video generative model, trained on nothing but unlabeled internet video, could learn to render a 2D playable world frame by frame from a single starting image and a latent action a human never labeled. GameNGen (2024-08) had shown a diffusion model rendering DOOM in real time, interactively, well enough that human raters struggled to tell it apart from the real game engine. Both results, detailed in full in Chapter 15, proved that a large enough video model could stand in for a game engine over short horizons in a narrow, previously-seen domain.

What neither result answered was whether the same recipe could scale past games into something a roboticist or an autonomous-vehicle team would actually deploy. A game world tolerates visual glitches and short memory; a robot arm deciding whether to grasp a cup, or a self-driving stack rehearsing an unusual intersection, needs a simulator whose errors are bounded and whose behavior generalizes to scenes it never saw during training. That gap — from “renders a game plausibly” to “usable as pretrained infrastructure for embodied decision-making” — is where the field pointed itself as 2024 closed. Google DeepMind’s own next step, Genie 2 (Google DeepMind, 2024), announced that December, already pushed in this direction: from a single prompt image it generated action-controllable 3D environments, sustaining consistency for up to a minute where most of its demonstrated examples still lasted only 10–20 seconds, and DeepMind used it directly to generate fresh training environments for SIMA, an existing generalist game-playing agent — a v0 of “world model as training infrastructure” rather than “world model as demo.”

Three 2024 demos prove the trick scales; three 2025 labs each package it as reusable infrastructure, from a different direction. PNG ↓

Three questions were still open going into 2025: could a world model be packaged as reusable pretrained infrastructure, the way a foundation language model is, rather than a bespoke demo per lab? Did a world model actually need to generate pixels at all, if all a downstream robot policy needs is to predict what will happen next? And could interactivity — a human or agent actually acting inside the generated world, not just watching it — be pushed from a research curiosity to something closer to real time? 2025 answered each question with a different model.

The Breakthrough

NVIDIA answered the infrastructure question first. Cosmos World Foundation Model Platform for Physical AI (NVIDIA, 2025), posted to arXiv in January 2025, proposes a world foundation model (WFM) as an explicit analogue to a pretrained language model: train one large model on a huge, curated corpus of general-purpose video, then post-train — fine-tune — that same backbone into specialized world models for a given robot, a given camera rig, or a given driving domain, rather than training each application from scratch. NVIDIA released two families of pretrained WFMs under this platform, both trained on roughly 100 million video clips curated from about 20 million hours of raw video, on a cluster of 10,000 H100 GPUs over three months. A diffusion-based family (Cosmos-Predict1, in 7B and 14B sizes) tokenizes video into continuous latents at an 8×8×8 compression ratio and denoises them with a diffusion transformer — the same patchify-and-attend recipe Sora demonstrated for video, reused here as infrastructure rather than a demo. An autoregressive-based family (4B, 5B, 12B, and 13B sizes) instead tokenizes video into discrete codes at 8×16×16 compression and predicts them left to right, the same factorization Chapter 5 of Image Generation teaches. NVIDIA released both families open-weight under a commercial-use license, and the paper demonstrates post-training the base WFMs into three specialized applications: camera-controllable navigation through a generated scene, robotic-manipulation video prediction conditioned on an action sequence, and multi-view video generation for autonomous-driving simulation.

Meta answered the pixels question by rejecting the premise. V-JEPA 2 (Assran et al., 2025), posted in June 2025, is a joint-embedding predictive architecture — it never learns to render a video frame at all. Instead it learns to predict the latent embedding of a masked or future patch of video directly from the embeddings of the patches it can already see, an approach its authors argue avoids wasting model capacity on the unpredictable, often visually irrelevant texture that a pixel-space predictor is forced to model. Pretrained self-supervised on more than a million hours of internet video plus a million images, with a vision-transformer encoder scaled from 300 million parameters (ViT-L) up to roughly one billion (ViT-g), V-JEPA 2 reaches 77.3% top-1 accuracy on the Something-Something v2 motion-understanding benchmark and 39.7 recall-at-5 on Epic-Kitchens-100 action anticipation — a 44% relative improvement over prior task-specific methods on that benchmark. Meta then post-trained an action-conditioned variant, V-JEPA 2-AC, on fewer than 62 hours of unlabeled robot-arm video from the DROID dataset, and deployed it zero-shot — with no task-specific reward, no environment-specific fine-tuning, and no new data collected in the target labs — on real Franka robot arms in two different labs, for pick-and-place manipulation.

Google DeepMind answered the interactivity question. Genie 3 (Google DeepMind, 2025), announced in August 2025, generates a navigable environment from a text prompt at 720p and 24 frames per second, sustains visual consistency for several minutes with a working memory reaching back roughly one minute, and — the feature its predecessor lacked — supports real-time interaction: a person can move through the generated world with sub-second latency as it is generated, and can issue “promptable world events” mid-scene, such as changing the weather or introducing a new object, that the model incorporates into the ongoing generation rather than requiring a fresh prompt. DeepMind released Genie 3 as a limited research preview to a small cohort of researchers and creators rather than a public product.

Three answers to the same ambition. Two converge on the same downstream use; only Genie 3 is itself a place an agent can act. PNG ↓

DeepMind’s own Genie 3 announcement frames world models generally as “a key stepping stone on the path to AGI,” and Cosmos borrows “foundation model” directly from the language-model literature. Neither claim is a benchmark result. What is actually demonstrated, as of 2025, is: a world can be generated and interacted with in real time for a few minutes before consistency degrades (Genie 3); a robot policy can be planned in latent space with no task-specific training, on two objects, in two labs (V-JEPA 2-AC); and a pretrained video backbone can be post-trained into narrower robotics and driving applications (Cosmos). Each is a genuine, measured result. None of them establishes general embodied intelligence, and this chapter states only the measured result, not the framing built on top of it.

The Mathematics

Cosmos’s diffusion and autoregressive objectives reuse machinery this book already teaches in full — the DDPM loss and diffusion transformer backbone owned by Diffusion Foundations and Flow Matching and Transformer Backbones, and the autoregressive factorization owned by Pixels, One at a Time — applied to a video tokenizer rather than derived anew. What is genuinely new to this chapter is the latent-space prediction objective that lets V-JEPA 2 learn without ever generating a pixel:

\mathcal{L}_{\mathrm{JEPA}} = \big\lVert P_\phi\big(\Delta y,\, E_\theta(x)\big) - \mathrm{sg}\big(E_{\bar\theta}(y)\big) \big\rVert_1

$x$ is the video the model can see — the visible context patches. $E_\theta$ is a context encoder, a vision transformer, that turns $x$ into an embedding. $y$ is a target the model cannot see: patches that were masked out, or a genuinely future frame. $E_{\bar\theta}$ is a second encoder, a slowly-updated exponential moving average of $E_\theta$ ‘s own weights, that embeds the target; $\mathrm{sg}(\cdot)$ , stop-gradient, blocks any learning signal from flowing back through it, which is what stops the whole system from collapsing to a trivial constant embedding that predicts itself perfectly. $P_\phi$ is a predictor network that takes the context embedding and an offset $\Delta y$ — which patches, or how far into the future — and predicts what the target’s embedding should be. The loss is the L1 distance between that prediction and the real target embedding. What this buys: the model never has to reconstruct pixel-level texture it cannot actually predict — which way leaves rustle, the exact grain of a carpet — it only has to predict the more compressed, more predictable representation of what happens next, which is the property V-JEPA 2’s authors argue makes the resulting embeddings more directly useful for downstream planning than a generative video model’s pixels are.

Symbol	Meaning
$x$	visible context patches shown to the model
$y$	target patches to predict — masked-out or drawn from a future frame
$E_\theta$	context encoder, a vision transformer
$E_{\bar\theta}$	target encoder, an exponential moving average of $E_\theta$ ‘s weights
$P_\phi$	predictor network, maps the context embedding plus an offset to a predicted target embedding
$\Delta y$	offset specifying which patches or which future timestep to predict
$\mathrm{sg}(\cdot)$	stop-gradient, blocks gradient flow into the target encoder

That same latent space is where V-JEPA 2-AC plans. Given a goal — an image of what the robot should achieve — it searches over candidate action sequences with the cross-entropy method, the same search PlaNet used to plan directly in a learned latent space seven years earlier (Chapter 3), scored by distance to the goal in latent space rather than by any hand-designed reward:

a^{*}_{1:T} = \operatorname*{arg\,min}_{a_{1:T}} \big\lVert P\big(a_{1:T},\, s_k,\, z_k\big) - z_g \big\rVert_1

$a_{1:T}$ is one candidate sequence of actions over a planning horizon of length $T$ . $s_k$ and $z_k$ are the robot’s current state and its latent embedding, from the same encoder $E_\theta$ trained above. $z_g$ is the latent embedding of the goal image — wherever the robot is trying to get to. $P$ is the action-conditioned predictor, rolling the latent forward under a candidate action sequence to imagine where it would end up. The cross-entropy method samples many candidate sequences, keeps the ones whose imagined final latent lands closest to $z_g$ , refits its sampling distribution around those elites, and repeats — the same iterative search Chapter 3 describes for PlaNet. What this buys: because the objective is just “get close to the goal’s embedding,” any task that can be specified as a goal image defines its own objective, with no hand-engineered reward function and no per-task fine-tuning — the same trained predictor plans pick-and-place for a cup or a box, in a lab it never saw during training.

Symbol	Meaning
$a_{1:T}$	candidate action sequence over the planning horizon
$s_k$ , $z_k$	current robot state and its latent embedding
$z_g$	latent embedding of the goal image
$P$	action-conditioned predictor, rolls latent dynamics forward under candidate actions
$T$	planning horizon length

Cosmos’s disclosed training footprint — 10,000 H100 GPUs running for three months to pretrain a family of models up to 14B parameters on 20 million hours of raw video — is a concrete, recent data point for the compute side of the co-evolution argument The Co-evolution Thesis makes across the whole book: a world model general enough to be sold as reusable infrastructure, rather than a narrow research demo, is only reachable once a cluster of that size is both available and financeable. V-JEPA 2’s result cuts the other way and is just as telling for hardware economics: because it never renders pixels, planning one action costs 16 seconds of latent-space search, against roughly four minutes per action for an equivalent Cosmos-based video-generation planner on the same robot task — a difference that decides whether a full manipulation trajectory finishes in minutes or in over an hour of compute.

What It Enabled

The three approaches are not strictly competitors — each optimizes for a different downstream use — but 2025 was the first year the field could compare them directly against a common ambition: usable, general-purpose infrastructure for embodied AI, rather than a single demonstrated trick.

	Cosmos	V-JEPA 2 / V-JEPA 2-AC	Genie 3
Generates pixels	Yes — diffusion or autoregressive video	No — predicts latent embeddings only	Yes — interactive video, real time
Largest disclosed scale	14B (diffusion), 13B (autoregressive)	~1B encoder (ViT-g)	Undisclosed
Pretraining data	~100M clips from ~20M hours of video	1M+ hours of video, 1M images	Undisclosed
Primary downstream use	Post-trained for robotics, driving, camera control	Zero-shot robot planning from a goal image	Real-time interactive environments, agent training
Release	Open-weight, commercial license	Open-weight (Hugging Face, June 2025)	Limited research preview

	Genie, Feb 2024	Genie 2, Dec 2024	Genie 3, Aug 2025
Scene dimensionality	2D	3D	3D
Consistency duration	Short, single scene	Up to 1 minute (most examples 10–20s)	Several minutes, ~1 minute working memory
Interaction	Latent action, offline generation	Offline generation from one image	Real time, sub-second latency, promptable world events
Public access	Research paper (owned by Chapter 15)	Research announcement	Research preview; folded into Google’s consumer Project Genie prototype starting January 2026, gated behind the AI Ultra subscription tier

Two threads pick up directly from here. The question of whether a generated world holds together not just in time but in three-dimensional space — camera moves that don’t warp geometry, objects that stay put from a different viewing angle — is the subject of The Third Dimension, where NeRF-style and Gaussian-splatting representations tackle 3D consistency directly rather than hoping a video model learns it implicitly. And the question this chapter leaves open — how do you score a world model’s physical plausibility, and who pays for the GPU-hours any of these three approaches burns — is taken up in Evaluating and Monetizing Motion.

Every later chapter's question traces back through the same 2024 proof that a video model could stand in for a game engine. PNG ↓

Key Papers

Paper	Authors	Year	Link	Why it matters
Cosmos World Foundation Model Platform for Physical AI	NVIDIA	2025	arXiv:2501.03575	Frames a world model as pretrained, post-trainable infrastructure for robotics and driving, open-weight at up to 14B parameters
V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning	Assran et al.	2025	arXiv:2506.09985	Shows a world model that never renders pixels can still plan zero-shot robot manipulation from a goal image
Genie 2: A large-scale foundation world model	Google DeepMind	2024	deepmind.google	Extends Genie from 2D to consistent 3D worlds generated from a single image, used to train an existing agent
Genie 3: A new frontier for world models	Google DeepMind	2025	deepmind.google	Adds real-time interaction and promptable world events at 24fps and 720p, DeepMind’s first real-time world model

PM Lens

The number I keep coming back to from this chapter is 16 seconds versus 4 minutes — V-JEPA 2-AC planning an action in latent space against a Cosmos-style planner rendering pixels to do the same job. That is not a benchmark curiosity; it is the difference between a robot finishing a pick-and-place task in minutes or in over an hour of compute, and it is exactly the kind of number that should show up in a build-versus-buy comparison before either model gets anywhere near a robotics roadmap. It also maps cleanly onto how available each approach actually is today: Cosmos and V-JEPA 2 are both open-weight, downloadable, and license-cleared for commercial post-training right now; Genie 3 is a gated research preview that only reached any paying consumer at all through Google’s AI Ultra tier and its Project Genie prototype, starting January 2026, more than a year after Cosmos shipped its weights. For a PM scoping embodied AI, availability is the filter to apply before capability.

Search