The Year Images Went Mainstream

It is April 2022. Over the next ten weeks, OpenAI and two teams at Google each publish a text-to-image system better than anything published before it, and each bets on a different mechanism to get there: hierarchical diffusion over a shared text-image embedding space, cascaded diffusion driven by a giant frozen language model, and autoregression pushed to twenty billion parameters. Three months later, a small independent lab running its product out of a Discord server opens a paid subscription, and by the end of the year, turning a sentence into a photorealistic image is something anyone with ten dollars a month can do, not just a research demo run on a cluster.

Where We Were

By early 2022, two competing recipes for text-to-image generation already existed, and both had visible ceilings. CLIP (Radford et al., January 2021) — covered in Chapter 11 — had shown that a contrastive objective trained on 400 million image-text pairs could learn a joint embedding space where a caption and its matching picture land near each other, but CLIP itself generates nothing; it only scores. DALL-E (Ramesh et al., February 2021), covered along with GLIDE in Chapter 12, turned text-to-image into a sequence problem: encode an image as a grid of discrete VQ-VAE tokens (the codebook mechanism Chapter 5 owns), concatenate a BPE-tokenized caption in front of them, and train an autoregressive transformer to predict the image tokens one at a time. It worked, and it generalized to prompts far outside its training distribution, but its samples were visibly softer and less coherent than what dedicated GANs could produce for the categories those GANs specialized in.

GLIDE (Nichol et al., December 2021) — also Chapter 12’s subject — took the other branch: condition a diffusion model directly on a text embedding and steer it with classifier-free guidance, the mechanism Chapter 10 develops in full. In its own head-to-head comparison against a DALL-E-style autoregressive baseline, GLIDE’s raters preferred its photorealism and caption match, at the cost of a diffusion model that still ran at a modest 3.5 billion parameters and a base resolution of 64x64, upsampled afterward. Neither DALL-E nor GLIDE answered three questions that were now on the table simultaneously. Could CLIP’s embedding space, so far used only to score how well a caption and an image matched, be repurposed as a generative interface in its own right? Did scaling the text encoder matter more than scaling the image model itself? And could autoregression, the approach GLIDE’s own comparison had just found wanting, close the gap on diffusion if pushed far enough past DALL-E’s scale? Three teams, working in parallel and largely without knowledge of each other’s unpublished results, answered all three within the same quarter — and none of the three answers was the obvious one.

Three separate tools, none of them a finished text-to-image system on its own. April 2022 is when three teams bet on three different ways to combine them. PNG ↓

The Breakthrough

“Hierarchical Text-Conditional Image Generation with CLIP Latents” (Ramesh et al., 2022) — the paper the field would come to call unCLIP, and the architecture behind DALL-E 2 — answered the first question directly: yes, CLIP’s embedding space can generate. The system splits image generation into two learned stages instead of one. A prior takes a caption’s CLIP text embedding and learns to predict what that caption’s CLIP image embedding would look like, without ever seeing the image itself. A decoder, a modified GLIDE, then takes that predicted image embedding — plus the original caption — and runs the reverse diffusion process to actually paint a picture consistent with it. The paper tried both an autoregressive and a diffusion-based prior and found the diffusion prior both cheaper to sample and better at matching captions; the decoder starts at 64x64 and is upsampled to 256×256 and 1024×1024 by two further diffusion models.

unCLIP's two networks: a prior that guesses a plausible CLIP image embedding from the caption, and a decoder that paints an image consistent with it. Sampling the prior again yields a different, still-matching image. PNG ↓

The system totals roughly 5.5 billion parameters across its four networks — a 1B-parameter prior, a 3.5B-parameter decoder, and two upsamplers together accounting for about 1B more — and it reached a zero-shot FID of 10.39 on the MS-COCO benchmark, ahead of GLIDE’s 12.24. The bigger payoff was qualitative: because the prior only constrains an image to a region of CLIP’s embedding space rather than to one exact picture, sampling the prior multiple times for the same caption produces genuinely different images that still match the caption, rather than minor pixel-level variations. Against GLIDE in human evaluation, raters preferred unCLIP’s samples for diversity 70.5% of the time, while photorealism (48.9%) and caption similarity (45.3%) came in close to a coin flip — the paper’s own framing was that it traded a sliver of fidelity for a large gain in variety, not that it dominated on every axis.

“Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding” (Saharia et al., 2022) — Imagen — answered the second question, and the answer was not what most of the field expected. Rather than training a text encoder from scratch on image-caption pairs the way CLIP and GLIDE did, Imagen freezes a large language model pretrained only on text — T5-XXL, 4.6 billion parameters — and feeds its embeddings into a cascade of three diffusion models: a 2-billion-parameter base model generating a 64x64 image, then a 600-million-parameter model upsampling to 256×256, then a 400-million-parameter model upsampling to 1024×1024. The paper’s central finding, stated as an explicit ablation rather than a hunch, was that increasing the frozen text encoder’s size improved both image fidelity and image-text alignment more than increasing the diffusion model’s size did — text understanding, not image modeling, was the bottleneck.

Imagen also had to solve a problem the earlier two systems mostly avoided: classifier-free guidance at a high enough weight to sharpen images noticeably tends to push pixel values outside the range the network was trained on, producing saturated, over-contrasted samples. Imagen’s fix, dynamic thresholding, clips and rescales the diffusion model’s predicted clean image at every sampling step relative to how extreme its own pixel values are — letting the model use much larger guidance weights than static clipping would tolerate. On a new benchmark it introduced for the purpose, DrawBench — around 200 prompts spanning categories like counting, spatial relationships, and unusual color combinations — human raters preferred Imagen’s samples over DALL-E 2’s, GLIDE’s, and Latent Diffusion’s on both fidelity and caption alignment. On MS-COCO, Imagen’s zero-shot FID-30K of 7.27 beat every other model tested, including DALL-E 2’s 10.39 and Make-A-Scene’s 7.55 — the latter trained directly on COCO, which Imagen was not.

“Scaling Autoregressive Models for Content-Rich Text-to-Image Generation” (Yu et al., 2022) — Parti, short for Pathways Autoregressive Text-to-Image — answered the third question by refusing to abandon autoregression and instead scaling it the way GPT-3 had scaled language modeling. Parti tokenizes images with ViT-VQGAN into a grid of discrete codes from an 8,192-entry codebook — 1,024 tokens for a 256×256 image, against 65,536 raw pixels for the same picture — then trains an encoder-decoder transformer to treat text-to-image generation as a translation task: encode the caption, decode a sequence of image tokens. The factorization itself is the same next-token mechanism Chapter 5 already covers; Parti’s contribution was pure scale, applied to a sequence short enough that scaling it stayed tractable. Trained at four sizes — 350M, 750M, 3B, and 20B parameters — zero-shot COCO FID fell from 14.10 to 10.71 to 8.10 to 7.23 as parameters rose, a smooth, monotonic curve with no sign of flattening at 20B. Fine-tuned on COCO, the 20B model reached an FID of 3.22. Parti introduced its own benchmark, PartiPrompts (P2), over 1,600 English prompts built specifically to be hard — abstract concepts, world knowledge, unusual perspectives — and on it, human raters preferred the 20B model’s images to the 3B model’s 62.7% of the time on caption alignment, evidence that the scaling curve kept paying off well past the point most autoregressive image models had previously been pushed.

The Mathematics

unCLIP’s prior-decoder split rests on one factorization of the text-conditional image distribution. Let $x$ be the image, $y$ the caption, and $z_i$ the CLIP image embedding of $x$ — a quantity computed deterministically from $x$ once $x$ exists, using the CLIP image encoder Chapter 11 covers.

P(x \mid y) = P(x, z_i \mid y) = P(x \mid z_i, y)\, P(z_i \mid y)

The first equality holds because $z_i$ contributes no new information once $x$ is fixed — it is just a function of $x$ , so adding it to the conditioning changes nothing about the probability of $x$ itself. The second equality is the ordinary chain rule, but it is the one that matters: it says a caption’s image can be generated in two separable steps, first guessing a plausible CLIP image embedding $P(z_i \mid y)$ from the caption alone, then generating an actual image $P(x \mid z_i, y)$ that matches both that embedding and the caption. What it buys you: the two steps can be trained as two separate networks — the prior and the decoder — and because the prior only pins down a region of embedding space rather than one exact image, sampling it repeatedly for the same $y$ yields different valid $z_i$ values, and therefore different valid images, without sacrificing caption relevance.

Symbol	Meaning
$x$	the generated image
$y$	the input caption
$z_i$	the CLIP image embedding of $x$
$P(z_i \mid y)$	the prior — predicts an image embedding from a caption
$P(x \mid z_i, y)$	the decoder — generates an image from an embedding and a caption

Imagen’s dynamic thresholding modifies the diffusion sampler’s predicted clean-image estimate $\hat{x}_0$ at every step, rather than the noise prediction guidance itself modifies (the guidance formula is Chapter 10’s subject).

s = \mathrm{percentile}\big(|\hat{x}_0|,\, p\big), \qquad \hat{x}_0 \leftarrow \frac{\mathrm{clip}(\hat{x}_0,\, -s,\, s)}{s} \ \text{ when } s > 1

$\hat{x}_0$ is the diffusion model’s current best guess, at some intermediate sampling step, of what the fully denoised image will look like — and at a high classifier-free guidance weight, some of its pixel values can drift well outside the $[-1, 1]$ range the network was trained on, which is what produces the washed-out, over-saturated look guidance pushed too far tends to cause. Rather than fix a hard clipping bound in advance, $s$ is set dynamically, per step, to a chosen high percentile $p$ of the current prediction’s own absolute pixel values; only if that percentile exceeds 1 does the clip-and-rescale actually trigger. What it buys you: guidance weights large enough to meaningfully sharpen an image, without the saturation artifacts a fixed threshold would let through once the network’s own predictions start pushing past the range it was ever trained to produce.

Symbol	Meaning
$\hat{x}_0$	the model’s current predicted clean image at a sampling step
$s$	the dynamic threshold, recomputed every step
$p$	the percentile used to set $s$
$\mathrm{clip}(\cdot, -s, s)$	clamps pixel values to the range $[-s, s]$

What It Enabled

The three systems answered different questions and left a legible trade space behind them.

	DALL-E 2 / unCLIP	Imagen	Parti
Core mechanism	Diffusion prior + diffusion decoder over CLIP embedding	Frozen T5-XXL + cascaded diffusion	Autoregressive transformer over VQ tokens
Parameters	~5.5B total	2B base + 4.6B frozen text encoder	20B (largest variant)
Zero-shot COCO FID-30K	10.39	7.27	7.23
Benchmark introduced	—	DrawBench (~200 prompts)	PartiPrompts (1,600+ prompts)
Distinctive strength	Image variation from one caption	Text-image alignment, counting, spatial relations	Monotonic gains from scale, no plateau at 20B

None of the three approaches disappeared afterward, but they did not carry equal weight going forward either. Imagen’s frozen-large-language-model recipe and dynamic thresholding became close to standard practice; its cascaded-diffusion structure and unCLIP’s embedding-conditioned decoder both fed directly into how Chapter 13’s subject, Stable Diffusion, was built and released as open weights four months later, in August 2022. Parti’s autoregressive bet looked, for a couple of years, like the road not taken next to diffusion’s dominance — until Chapter 20 picks the thread back up with next-scale and native-multimodal autoregressive image generation resuming Parti’s argument that scale, applied to the right architecture, keeps paying off. The guidance and conditioning machinery all three systems depended on is refined further in Chapter 15, and the FID numbers this chapter leans on throughout get their own scrutiny — including the metric’s known blind spots — in Chapter 21.

Two 2021 systems seed three 2022 answers, and two of those three converge four months later on one open-weights model anyone can run. PNG ↓

None of the three papers dwelt on it, but 2022 was also the year the provenance question stopped being theoretical. The C2PA coalition, formed in February 2021 specifically to standardize content-credential metadata, had been built for a world that did not yet have consumer-grade photorealistic text-to-image at scale; by mid-2022 it did. Chapter 21 covers the watermarking, memorization, and disclosure debates this triggered in full — the point worth flagging here is only that the timing was not a coincidence. A capability jump this sharp, arriving inside ten weeks across three independent labs, is exactly the kind of discontinuity that outruns the governance built for the previous state of the art.

The same quarter also produced a product story that ran alongside the papers rather than through them. Midjourney, a small independent lab that had run an invite-only Discord server since March 2022, opened public open beta on July 12, 2022, with four paid tiers starting at a $10-per-month Basic plan. OpenAI expanded DALL-E 2 access in July 2022, introducing a credit system alongside the wider waitlist rollout — roughly $0.03 per image at retail — then dropped the waitlist entirely that September and, once the API opened in November, priced access at about $0.02 per image — while Stable Diffusion’s August 2022 open-weights release meant anyone with a consumer GPU could run a comparable pipeline for the cost of electricity. By the close of 2022, text-to-image generation had gone from three arXiv papers to a live market with visibly different price points and access models attached to each approach.

Key Papers

Paper	Authors	Year	Link	Why it matters
Hierarchical Text-Conditional Image Generation with CLIP Latents	Ramesh et al.	2022	arXiv:2204.06125	Introduces unCLIP: a diffusion prior over CLIP embeddings plus a diffusion decoder, behind DALL-E 2
Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding	Saharia et al.	2022	arXiv:2205.11487	Imagen; shows a frozen large language model matters more than a larger diffusion model, plus dynamic thresholding
Scaling Autoregressive Models for Content-Rich Text-to-Image Generation	Yu et al.	2022	arXiv:2206.10789	Parti; scales autoregressive text-to-image to 20B parameters with a smooth, unplateaued FID curve

PM Lens

Three teams answered three different product questions at once, and the market ended up sorting itself along those same lines. Imagen’s bet — spend the parameter budget on understanding the prompt, not on the picture — is the one that quietly won inside most production text-to-image systems since. Parti’s bet looked dead for a couple of years and then came back inside the current wave of native-multimodal image models. And Midjourney, the one team here without a paper, won the actual consumer market in 2022 with a $10-a-month Discord bot while two much better-funded labs were still deciding how to price an API. The lesson I keep relearning from this quarter: shipping a rough interface to real users a full quarter before your API is “ready” can matter more than which paper’s FID number is lowest. Benchmarks tell you which lab a researcher should read next; they rarely tell you which product a customer will pay for first.

Search