The Current Frontier

It is April 2024. Diffusion transformers own image generation so completely that the autoregressive pixel models covered in Chapter 5 read as a historical detour — exact likelihoods, sequential sampling, a promising idea that lost to a faster one. A paper claiming a purely autoregressive model now beats the best diffusion transformer on ImageNet barely looks plausible on its own terms. A month later, a second paper drops the distinction between “language model” and “image model” altogether, training one transformer on interleaved text and image tokens with no separate generator at all. Within a year, a version of that idea is running inside a chat product used by hundreds of millions of people.

Where We Were

By early 2024, the image-generation stack had converged on a specific recipe. Latent diffusion, taught in Chapter 13, compressed images into a smaller continuous space before denoising. The diffusion transformer (DiT), owned by Chapter 19, replaced the U-Net backbone that Chapter 13’s models had used, and flow matching gave that backbone a straighter, more efficient training target. Stable Diffusion XL had shipped in July 2023; Stable Diffusion 3 followed in March 2024 on the rectified-flow-transformer recipe. Every major open and closed text-to-image system — SD3, and five months later FLUX.1 — was, underneath, a transformer denoising a compressed latent, sampled over some number of iterative steps.

Autoregressive image generation had not disappeared, but it had drifted to the margins of that story. Muse (Chang et al., 2023), published in January 2023, took a third path: rather than denoise a latent or sample pixels one at a time, it trained a transformer to predict randomly masked image tokens in parallel, conditioned on a frozen large language model’s text embedding — the masked-modeling training task also used, for text, in Chapter 6. Muse reported that its 900M-parameter model reached an FID of 6.06 on CC3M, competitive with contemporary diffusion models, using only 24 parallel decoding steps for its base transformer against the hundreds or thousands of sequential evaluations a raster-order autoregressive model of the same resolution would need. It was a real result, and it arrived in the same month DALL-E 2 and Stable Diffusion were still absorbing most of the field’s attention; masked-token generation stayed a minority approach through 2023 while diffusion’s own speed problem was being solved from the inside, by the samplers of Chapter 9 and the distillation techniques of Chapter 18.

The two colours interleave rather than take turns: the field never settled on one family before the next challenge to it arrived. PNG ↓

Two open questions sat underneath this consensus. First: autoregressive image models had lost on speed, not on principle — could a different factorization of the autoregressive problem, rather than a different training paradigm, close that gap? Second, and more structural: every system described above still separated “the model that understands language” from “the model that produces pixels,” even when both were transformers, wired together through an embedding or a cross-attention layer. Was that separation necessary, or was it an artifact of history — GPT-style language models and diffusion image models having been invented by mostly separate research communities that later learned to interoperate, rather than a single team designing one model for both jobs from the start? By late 2024 and into 2025, both questions had answers, and a third development showed what happens when those answers get shipped as a single consumer product.

The Breakthrough

Visual autoregressive modeling (Tian et al., 2024) — VAR — kept autoregression but changed what gets predicted at each step. Standard autoregressive image models, from PixelCNN through the transformer-over-VQ-tokens design behind DALL-E, predict one token at a time in a raster scan, each token conditioned on every token before it in that scan order — the factorization Chapter 5 teaches in full. VAR replaces that ordering with what its authors call “next-scale prediction”: an image is first encoded, by a multi-scale VQVAE built around the residual quantization idea and the discrete-codebook mechanism of Chapter 5, into a small stack of token maps at successively higher resolutions, from a coarse handful of tokens up to a full-resolution grid. The autoregressive transformer then predicts one entire token map at a time, conditioned on every coarser map that came before it — not one token conditioned on every earlier token. Within a single scale, every token is generated in parallel, because none of them condition on each other; only the scales themselves are ordered. On the ImageNet 256×256 benchmark, VAR improved FID over its own autoregressive baseline from 18.65 to 1.73 and Inception Score from 80.4 to 350.2, with roughly a 20x faster inference speed than that raster-order baseline. The paper’s headline comparison was against diffusion, not against other autoregressive models: a 2-billion-parameter VAR model reached FID 1.73 on the same benchmark, surpassing DiT-XL/2’s reported 2.27 and larger diffusion transformers besides, while needing about a quarter as many training epochs (350 versus DiT’s 1,400). Scaling VAR’s model size further produced power-law relationships between compute and loss with correlation coefficients near −0.998 — the same kind of clean scaling curve that had, by then, become the expected signature of a well-behaved autoregressive recipe in language modeling.

Only the handful of scales are ordered; every token within one scale is produced in parallel, which is where the 20x speedup over raster order comes from. PNG ↓

Chameleon (Chameleon Team, 2024), published a month after VAR, attacked the second question directly. Rather than pairing a language model with a separate image generator, Chameleon converts images into discrete tokens — a custom tokenizer encodes a 512×512 image into 1,024 tokens drawn from an 8,192-entry codebook — and interleaves those image tokens with ordinary text tokens into one combined sequence, in whatever order a document naturally mixes text and pictures. A single transformer is then trained on that mixed-modal sequence with the same next-token, cross-entropy objective any text-only language model uses, so the model has no notion of “the image branch” or “the text branch”: every token, image or text, is predicted the same way, by the same weights. Meta trained two sizes, 7B and 34B parameters, on roughly 9.2 trillion tokens split across text-only, paired text-image, and interleaved documents. Training a mixed-modal transformer at that scale surfaced an instability specific to this setting — described in the Mathematics section below — that had not needed solving in a text-only or an image-only model. Once stabilized, the 34B model’s mixed-modal, long-form generations were preferred by human raters over Google’s Gemini Pro 60.4% of the time and over GPT-4V 51.6% of the time on a held-out evaluation of interleaved text-and-image responses, while matching Llama 2 and staying competitive with Mixtral 8x7B on text-only benchmarks.

The clearest evidence that both ideas — a non-raster autoregressive factorization, and a single model spanning text and images — had left the research literature and reached production came ten months later. On March 25, 2025, OpenAI shipped native image generation inside GPT-4o, describing the underlying design as “an autoregressive prior with a powerful decoder”: a transformer first produces a sequence of image tokens autoregressively, the same way it would produce a sequence of text tokens, and a diffusion-based decoder then converts that token sequence into final pixels. It is neither a pure autoregressive model in VAR’s sense nor a pure latent-diffusion model in Chapter 13’s sense, but a pipeline that puts autoregression where Chameleon put it — as the mechanism that ties image generation to the model’s own language understanding — and keeps diffusion, described in full in Chapter 8, for the final image-quality step. Because generation happens “natively” inside GPT-4o rather than through a call to a separate image model, the same conversation that requested an image could also revise it, and the model’s general world knowledge and in-context understanding of an uploaded reference image carried directly into what it drew — capabilities OpenAI highlighted specifically as consequences of not routing image generation through a separate system. The feature became the default image generator in ChatGPT for Plus, Pro, Team, and Free users at launch, and shipped as an API model, gpt-image-1, on April 23, 2025.

Chameleon removes the seam between text and image inside one model; GPT-4o keeps diffusion for the final pixels but puts an autoregressive, language-model-native step in front of it. PNG ↓

The Mathematics

VAR’s core departure from standard autoregression is a change of what gets factorized. Where an ordinary autoregressive image model factorizes a joint distribution over individual tokens (Chapter 5’s $p(x) = \prod_{i=1}^{n} p(x_i \mid x_1,\dots,x_{i-1})$ ), VAR factorizes over entire token maps:

p(r_1, r_2, \dots, r_K) = \prod_{k=1}^{K} p\big(r_k \mid r_1, r_2, \dots, r_{k-1}\big)

Each $r_k$ is not a single token but a complete grid of tokens representing the image at scale $k$ — a coarse thumbnail’s worth of tokens for small $k$ , a full-resolution grid for $k = K$ . The equation says the model predicts the entire map $r_k$ conditioned on every coarser map before it, and — because nothing inside $r_k$ depends on anything else inside $r_k$ — every token within that one map can be generated in parallel. What it buys you: the number of sequential steps to generate an image drops from one per token (potentially tens of thousands, at full resolution) to one per scale, $K$ , which for VAR’s configurations is on the order of ten — the direct mechanism behind the paper’s reported 20x inference speedup over a raster-order autoregressive baseline of comparable size.

Symbol	Meaning
$r_k$	the complete token map at scale $k$ , produced in parallel
$K$	the total number of scales, from coarsest to full resolution
$p(r_k \mid r_1,\dots,r_{k-1})$	the model’s prediction for scale $k$ , conditioned on all coarser scales

VAR’s tokens themselves come from the multi-scale VQVAE’s residual quantization, which reuses the nearest-codebook-vector lookup taught in Chapter 5 at each of the $K$ scales rather than once at a single resolution; VAR does not change that lookup, only the number of times it is applied and the factorization built on top of it.

Muse’s training objective is a masked, rather than autoregressive, cross-entropy loss over discrete image tokens:

\mathcal{L}_{\text{mask}} = -\,\mathbb{E}_{z,\,M}\left[\sum_{i \in M} \log p_\theta\big(z_i \mid z_{\bar M}, c\big)\right]

Symbol	Meaning
$z$	the discrete image token grid for a training example
$M$ , $\bar M$	the randomly masked positions, and the unmasked remainder
$z_i$	the true token at masked position $i$
$c$	the text conditioning vector, from a frozen pretrained language model
$p_\theta$	the model’s predicted distribution over token values

Muse samples with iterative parallel decoding at inference and applies classifier-free guidance, the mechanism Chapter 10 owns, with a guidance scale that increases across decoding steps rather than staying fixed — a scheduling detail specific to Muse’s sampler, not a change to the guidance mechanism itself.

Chameleon’s training objective is the ordinary autoregressive cross-entropy loss over a combined text-and-image vocabulary — the same factorization Chapter 5 teaches, applied token-by-token across a sequence that happens to mix two modalities. What is new is the stabilization term Chameleon’s authors needed to add to train that objective at 34B-parameter scale without the loss diverging:

\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{CE}} + \sigma_z \log^2 Z, \qquad \sigma_z = 10^{-5}

$\mathcal{L}_{\text{CE}}$ is the standard next-token cross-entropy loss. $Z$ is the softmax normalizer computed at the model’s output layer — the sum, over every possible next token, of the exponentiated logits — and in a very large mixed-modal vocabulary spanning both text and image tokens, $Z$ tended to drift to extreme values during training, a failure mode Chameleon’s authors call logit drift. Adding a small penalty on $\log^2 Z$ , weighted by $\sigma_z$ , discourages that drift without meaningfully changing what the main cross-entropy loss is teaching the model. What it buys you: a mixed-modal transformer that trains stably at tens of billions of parameters and trillions of tokens, a regime where, without this term, the authors report training would otherwise diverge — a problem that a text-only or image-only transformer of the same size did not exhibit in the same way, because it did not have two very differently distributed token types sharing one softmax.

Symbol	Meaning
$\mathcal{L}_{\text{total}}$	the full training loss actually optimized
$\mathcal{L}_{\text{CE}}$	standard autoregressive cross-entropy over the combined vocabulary
$Z$	the softmax normalizer at the output layer
$\sigma_z$	the fixed weight on the z-loss penalty term

OpenAI has not published a paper describing gpt-image-1’s training objective or architecture in equation form; the public description — an autoregressive prior feeding a diffusion decoder — is architectural rather than mathematical, and no further detail is verifiable from public sources.

What It Enabled

The four systems in this chapter did not converge on one winning recipe; they demonstrated that diffusion’s dominance of image generation was a contingent outcome of the early 2020s, not a structural necessity, and each opened a different downstream direction.

Approach	Representative model	Image representation	Generation mechanism	Reported ImageNet 256 FID	Relative sampling cost
Diffusion transformer	DiT-XL/2 (Chapter 19)	continuous latent patches	iterative denoising	2.27	many steps, each a full network pass
Next-scale autoregressive	VAR-d30, 2B params	multi-scale VQ tokens	parallel per-scale, sequential across ~10 scales	1.73	~20x fewer sequential steps than raster AR
Masked parallel	Muse, 3B params	VQGAN tokens	24-step parallel unmasking (base model)	not reported on ImageNet; 7.88 zero-shot on COCO	far fewer steps than raster AR or early diffusion
Early-fusion autoregressive	Chameleon, 34B params	interleaved text/image tokens	single-stream next-token prediction	not benchmarked; general-purpose model	standard LLM-style decoding
Hybrid AR + diffusion	GPT-4o native image generation	learned tokens, architecture not disclosed	autoregressive prior, then diffusion decoder	not published	roughly one minute per image, by OpenAI’s own account

VAR’s result reopened a question the field had treated as settled since Chapter 9 and Chapter 19: whether diffusion’s speed problem was inherent to the diffusion formulation specifically, or a symptom of forcing generation through many small sequential steps in general — a symptom next-scale autoregression sidesteps from a different direction than distillation, covered in Chapter 18, sidesteps it. Chameleon’s early-fusion recipe became one of the reference architectures behind the wave of natively multimodal frontier language models — GPT-4o and Gemini among them — covered from the language-model side in Language Models, Chapter 18; this chapter and that one describe the same architectural bet from opposite ends, image generation as a byproduct of a language model versus a language model that happens to also generate images.

None of this chapter's three systems descends from another; each recombines earlier chapters' mechanisms differently, and two of the three point forward to a different later chapter. PNG ↓

The other consequence is one this book has not had to confront squarely until now. When a single conversational product can generate photorealistic images on request, revise them across turns, and render arbitrary embedded text, the question of whether a given image is a photograph or a synthesis stops being a niche concern for forensic researchers and becomes a mainstream product and policy question — the subject taken up directly in Chapter 21.

Key Papers

Paper	Authors	Year	Link	Why it matters
Muse: Text-To-Image Generation via Masked Generative Transformers	Chang et al.	2023	arXiv:2301.00704	Showed a non-diffusion, non-raster-autoregressive path to competitive text-to-image quality via masked parallel token prediction
Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction	Tian et al.	2024	arXiv:2404.02905	Replaced next-token with next-scale prediction, making an autoregressive model beat DiT on ImageNet FID and inference speed
Chameleon: Mixed-Modal Early-Fusion Foundation Models	Chameleon Team	2024	arXiv:2405.09818	Trained one transformer on interleaved text and image tokens with a single next-token objective, at 34B-parameter scale
Introducing 4o Image Generation	OpenAI	2025	openai.com/index/introducing-4o-image-generation	Shipped an autoregressive-prior-plus-diffusion-decoder design as the default ChatGPT image generator, closing the loop between research and mainstream product

PM Lens

This is the first chapter in the book where the state of the art is a product I use the same week I’m evaluating it, not a paper I read months after launch. When OpenAI priced gpt-image-1’s API in April 2025 at roughly $0.02, $0.07, and $0.19 per image for low, medium, and high quality — and told developers to expect it to take up to a minute per image, against the few seconds a well-tuned diffusion pipeline delivers — that was a legible product trade-off: pay more, wait longer, get an image that follows a complicated instruction and renders your logo’s text correctly, versus a cheaper, faster model that is worse at both. I don’t read “autoregressive versus diffusion” as a purity contest anymore; I read it as two cost curves a roadmap has to pick between, feature by feature, the same way a team picks a sampler in Chapter 9’s terms. The Chameleon-style bet — one model, no separate image tool — is the one I’d watch longest, because it changes what “adding image generation to a chat product” costs to build at all.

Search