Sora and Spacetime Patches

It is February 2024. Every video diffusion model shipped so far, from the joint-training cascades of 2022 to Stable Video Diffusion’s open weights three months earlier, still generates a fixed handful of frames at a resolution chosen at training time. OpenAI announces a model that does neither: Sora, which turns video into the same kind of patch tokens a diffusion transformer already knows how to denoise for still images, trained jointly on images and clips of whatever duration, resolution, and aspect ratio they arrive in, generating up to a minute of video from a single prompt.

Where We Were

By early 2024, video generation had two working but separate lineages, and neither had adopted the architecture that had already taken over image diffusion. The first lineage tokenized video into a discrete grid and predicted it with an autoregressive or masked transformer — VideoGPT, CogVideo, and Phenaki, covered in Tokens for Video — trading exact likelihood for a training-time tokenizer and, in Phenaki’s case, a variable-length grid. The second lineage, which had become the field’s default through 2023, kept a convolutional U-Net at the center and either trained it end-to-end on space-time volumes or bolted trainable temporal layers onto a frozen image U-Net, the approach Video LDM, AnimateDiff, and Stable Video Diffusion took to reach open, shippable weights (Latent Video Diffusion). Stable Video Diffusion’s own released variants generated 14 or 25 frames from one conditioning image, at a resolution and duration fixed by training — a ceiling any U-Net inherits from its convolutional downsampling path, which is built around one spatial resolution and has no native notion of a variable-length time axis at all.

Image generation, meanwhile, had already answered a closely related architectural question and moved on from it. Scalable diffusion models with transformers (Peebles & Xie, 2022) — DiT — had shown, fourteen months earlier, that a latent-diffusion model’s U-Net could be replaced outright with a plain transformer: cut the noised latent into fixed-size patches the way a Vision Transformer cuts an image, run a stack of transformer blocks over the resulting token sequence, and reassemble the output into a latent afterward. DiT-XL/2 reached a state-of-the-art FID of 2.27 on class-conditional ImageNet at 256×256, and, more importantly for what came next, models with more Gflops — from more depth, more width, or smaller patches producing more tokens — consistently produced lower FID, the compute-scaling curve diffusion had not clearly had before (Flow Matching and Transformer Backbones). That result was published and reproduced for still images through 2023. Nobody had yet pointed a patch-and-transformer diffusion model at video, where the patch grid would need a third axis — time — and where a training run would need to swallow clips of whatever length and shape they actually came in, rather than a single resolution chosen in advance.

The bolt-on U-Net lineage kept shipping for two years while DiT's patch recipe sat unused for video. Sora is the point where the two threads finally meet. PNG ↓

Two constraints, not one, stood between DiT’s patchify-and-attend recipe and a working video model. The compute one was obvious: a minute of video has orders of magnitude more raw values than a single image, so whatever compresses it into patches has to compress far harder than an image autoencoder does. The data one was less obvious but just as binding: a fixed-resolution, fixed-duration training set forces every video into the same crop and the same clip length regardless of what it actually shows, discarding whatever composition and framing the original video had. Sora’s announcement addressed both at once, and did so as a single design decision rather than two separate fixes.

The Breakthrough

OpenAI’s technical report, Video generation models as world simulators (OpenAI, 2024), states its core method plainly: Sora turns visual data into patches “by first compressing videos into a lower-dimensional latent space, and subsequently decomposing the representation into spacetime patches.” A dedicated video compression network reduces a raw clip’s spatial and temporal dimensions into a compressed latent, the video analogue of the perceptual autoencoder that Latent Diffusion and Stable Diffusion uses for images; that compressed latent is then cut into patches that span both space and time at once, rather than space alone. The report draws the analogy directly to language: patches play the role for Sora that tokens play for a large language model, and a single still image is simply a video with a time axis of length one, tokenized by the identical procedure. Because the report explicitly withholds model and implementation details — “model and implementation details are not included in this report” — this chapter states only what OpenAI disclosed about the method and its behavior, not a reconstructed architecture.

That patch-based representation is what makes variable-shape training possible. Rather than resizing or center-cropping every clip to one resolution, Sora trains on data at its native size, and the report states that “training on videos at their native aspect ratios improves composition and framing” relative to training exclusively on square crops, the norm in most earlier video generation work. The same native-size training pays off at sampling time: a single model can be sampled at widescreen 1920×1080, vertical 1080×1920, and every aspect ratio between the two, rather than needing a separate model or a separate finetune per output shape. Text conditioning is handled the same way DALL·E 3 handled image captions: a dedicated video captioner, trained to produce detailed descriptions, recaptions the training corpus, and a large language model expands short user prompts into the more detailed captions the model was trained against before they reach the diffusion transformer.

The same six steps run whether the input is one still image or a minute-long clip, because nothing in the pipeline assumes a fixed resolution or duration. PNG ↓

The report is direct about what kind of model sits at the center of that pipeline: “Sora is a diffusion model; given input noisy patches (and conditioning information like text prompts), it’s trained to predict the original ‘clean’ patches,” and “Sora is a diffusion transformer” specifically — DiT’s patchify-and-attend recipe, with the patch grid now spanning time as well as space, and no architectural ceiling on how many patches a training clip contributes. The report’s headline scaling claim reuses DiT’s own finding rather than introducing a new one: side-by-side samples at a base compute budget, four times that budget, and thirty-two times that budget show sample quality improving markedly as compute increases, the same Gflops-versus-quality curve DiT had already demonstrated for still images, now shown to hold for video. On the strength of that curve, the largest disclosed model, Sora, generates video up to a minute long at high definition, and OpenAI frames the result as evidence “that scaling video generation models is a promising path towards building general purpose simulators of the physical world” — a claim in the same lineage as the learned-simulator framing Predicting the Next Frame traces back to 2018, now attempted at a scale no earlier world-model paper had reached.

The report backs that framing with specific disclosed capabilities. Sora, the report states, “can generate multiple shots of the same character in a single sample, maintaining their appearance throughout the video,” a multi-shot consistency earlier video models had not demonstrated. It shows partial object permanence — persisting people, animals, and objects through occlusion or when they briefly leave the frame, though the report is explicit that this holds “often, though not always.” And it extends past photorealistic video into simulated processes: “Sora is also able to simulate artificial processes — one example is video games. Sora can simultaneously control the player in Minecraft with a basic policy while also rendering the world and its dynamics in high fidelity” — an emergent capability nobody trained the model to have directly, arrived at purely from scale, that anticipates the action-controllable and game-engine framings taken up later in this book (World Models Become Game Engines, World Models as Foundation Models).

The Mathematics

DiT’s patchify-and-attend mechanism and the underlying diffusion training objective are taught in full in Diffusion Foundations and Flow Matching and Transformer Backbones; nothing about the attention mechanism or the noising process changes here. What Sora’s patch tokenization adds, on top of that shared machinery, is a third axis inside the patch grid itself:

N = \frac{T'}{p_t} \cdot \frac{H'}{p_h} \cdot \frac{W'}{p_w}, \qquad x_i = \mathrm{flatten}\Big(z\big[t_i\!:\!t_i+p_t,\; h_i\!:\!h_i+p_h,\; w_i\!:\!w_i+p_w\big]\Big)

$z$ is the compressed spacetime latent the video compression network produces, a tensor with a time extent $T'$ , a height $H'$ , a width $W'$ , and some number of channels — the video analogue of the single-frame latent $\mathcal{E}(x)$ that image latent diffusion compresses down to. $p_t$ , $p_h$ , and $p_w$ are the patch dimensions along time, height, and width; cutting $z$ into non-overlapping blocks of that size and flattening each block into a vector $x_i$ produces a sequence of $N$ patch tokens, exactly the way DiT flattens 2D image patches, with one more axis available to cut along. What this buys the model: because $N$ is computed from whatever $T'$ , $H'$ , and $W'$ a given clip happens to have, a single patchify-and-attend network handles a one-frame still image, a four-second clip, and a minute-long clip with the same operation and the same trained weights — there is no fixed spatial or temporal grid baked into the architecture the way there is in a convolutional U-Net.

Symbol	Meaning
$z$	the compressed spacetime latent, output of the video compression network
$T', H', W'$	the latent’s extent along time, height, and width
$p_t, p_h, p_w$	the patch size along each of those three axes
$N$	the number of spacetime patch tokens produced
$x_i$	one flattened patch, the $i$ -th token in the sequence

The diffusion transformer’s training target is the one the technical report states directly — the network predicts the clean patches, not the noise that was added to them:

\mathcal{L} = \mathbb{E}_{x_0,\, t,\, c}\Big[\big\lVert x_0 - f_\theta(z_t, t, c) \big\rVert_2^{2}\Big]

$x_0$ is the true, clean sequence of spacetime patch tokens for a training clip or image; $z_t$ is that same sequence after $t$ steps of the forward noising process taught in Diffusion Foundations; $c$ is the conditioning vector built from the recaptioned or GPT-expanded text prompt; and $f_\theta$ is the diffusion transformer itself, predicting a full, clean patch sequence directly rather than predicting the noise $\epsilon$ that a standard DDPM loss targets. This is a parameterization choice, not a new kind of diffusion process — regressing onto $x_0$ instead of $\epsilon$ is an established alternative already covered where the DDPM loss is introduced — but pairing it with a variable-length sequence of spacetime patches is what lets the same loss, and the same network, train on a single image and an entire minute-long clip without any change of shape.

Symbol	Meaning
$x_0$	the true, clean spacetime patch sequence
$z_t$	that sequence after $t$ steps of forward noising
$c$	the text-conditioning vector, from a recaptioned or expanded prompt
$f_\theta$	the diffusion transformer, parameterized by $\theta$
$t$	the noising timestep

A patch grid that spans time as well as space grows the token sequence a transformer attends over by roughly the number of temporal patches, on top of whatever spatial patches already contribute — a minute-long clip can plausibly reach a sequence length in the tens of thousands of tokens once decomposed into spacetime patches. OpenAI’s own headline result, that sample quality improves markedly from base compute to 4x to 32x, is a direct restatement of the hardware-model co-evolution thesis: a model architecture that turns additional accelerator compute into predictably better output only pays off once that compute is actually available and affordable at datacenter scale, the argument Hardware, Model Co-evolution makes in full for the GPU generations that made runs like this financeable.

What It Enabled

Sora did not replace the bolt-on-U-Net lineage overnight — it published a method and a set of qualitative results, not a public model, and OpenAI’s own report withheld the training data, compute budget, and parameter count. What it changed was which architecture the field started building on.

	Bolt-on U-Net video diffusion, 2022–2023	Sora, spacetime-patch diffusion transformer, 2024
Backbone	Convolutional U-Net, with or without added temporal layers	Plain transformer over a spacetime patch token sequence, no U-Net
Training shape	Fixed resolution and frame count per training run	Native duration, resolution, and aspect ratio, images and video trained jointly
Max disclosed duration	Stable Video Diffusion: 14 or 25 frames per clip	Up to one minute
Max disclosed resolution	Video LDM text-to-video: up to 1280×2048	1920×1080 widescreen down to 1080×1920 vertical, and formats between
Public weights	Stable Video Diffusion released openly, November 2023	Announced February 2024 as a research report; public product release, December 2024

Disclosed capability	Disclosed limitation
Multiple shots in one generated video that persist character identity and visual style	Does not accurately model the physics of many basic interactions, such as glass shattering
Partial object permanence through occlusion or frames left and re-entered	Object permanence holds often, though not always, and can fail without an obvious cause
Simulating a digital process — controlling a Minecraft-like player while rendering the world at high fidelity	Confuses spatial detail, such as left and right, and struggles with precise descriptions of events unfolding over time

The lineage that follows splits in the same way DiT’s own lineage split for images: an open-weights race to reproduce the recipe, and a race to ship it as a product. CogVideoX and HunyuanVideo, released later in 2024, adopt a spacetime-patch diffusion transformer over a 3D video autoencoder as open weights, the direct technical descendants of the method Sora’s report described without releasing (Open Video Models). Google’s Veo, Runway’s Gen-3 Alpha, and Kling all launched within months of Sora’s announcement, each entering a commercial race that Sora had effectively opened before it was itself available to the public (The Commercial Race). Sora’s own public release did not arrive until December 2024, ten months after the announcement — long enough for several competitors to ship a public product first. And the one-minute ceiling itself becomes the next open problem, since neither spacetime patches nor the diffusion objective say anything about how a clip that long stays temporally consistent scene to scene, the question taken up directly in Long Video and Consistency.

Sora is not an endpoint. Every later chapter in this book is one consequence of moving patches from images to video. PNG ↓

Key Papers

Paper	Authors	Year	Link	Why it matters
Scalable Diffusion Models with Transformers	Peebles & Xie	2022	arXiv:2212.09748	Replaced the U-Net with a patchify-and-attend transformer for image diffusion and showed FID improving predictably with compute — the architecture and the scaling claim Sora reuses for video
Video generation models as world simulators	OpenAI	2024	openai.com	Extended patch tokenization to a time axis, trained a diffusion transformer jointly on variable-duration, variable-resolution video and images, and reported the same compute-scaling behavior holding for video

PM Lens

The detail I keep coming back to from this chapter is the ten-month gap between Sora’s February 2024 announcement and its December 2024 public release — a research report with no product behind it, for most of a year, while Veo, Runway Gen-3, and Kling shipped to paying customers in the meantime. That gap is the real lesson for anyone scoping a generative-video roadmap off a technical report rather than a shipped API: a scaling curve and a handful of qualitative samples tell you the ceiling is real, they do not tell you when, or at what price, you can build against it. Today the practical version of Sora is priced work: OpenAI’s Sora 2 API bills video generation by the second, on the order of a dime a second at 720p and several times that for the higher-resolution Pro tier — numbers worth checking against whatever your product actually needs before a scaling claim from February turns into a line item in December.

Search