DALL-E and GLIDE

It is February 2021. A twelve-billion-parameter transformer, trained to treat a caption and an image as one sequence of tokens, produces recognizable illustrations of things that have never existed — an armchair shaped like an avocado, a snail made of a harp — from nothing but a sentence. Ten months later, a different team at the same lab answers the same brief with the opposite architecture: no tokens, no autoregression, just a diffusion model learning to denoise its way toward whatever a sentence describes. The fork in the road this chapter documents did not stay forked for long, but for one year the field ran both experiments at once.

Where We Were

By early 2021, three separate research threads had each solved part of the text-to-image problem without solving all of it. GAN-based text conditioning, covered in First Words to Pictures, had gotten as far as StackGAN’s 256×256 stacked generators (December 2016) and AttnGAN’s per-word attention (November 2017), but adversarial training’s instability at scale kept a hard ceiling on how far anyone had pushed a GAN-based pipeline toward open-ended, unconstrained captions — the datasets that worked best were narrow ones like bird and flower photographs, not the full breadth of what a person might type.

A second thread had nothing to do with images at all. Van den Oord et al.’s VQ-VAE (November 2017) had shown that an image could be compressed into a small grid of discrete codes drawn from a learned codebook, and that an autoregressive prior — the same chain-rule factorization a language model uses — could then be trained over that grid instead of over raw pixels, cutting the number of sequential sampling steps by roughly fifty-fold for a typical configuration (Pixels One at a Time). That result mattered for a specific reason: once an image is a sequence of discrete tokens, generating one is no longer a computer-vision problem. It is a language-modeling problem, and by 2020 language modeling had a proven scaling recipe — GPT-3, a 175-billion-parameter transformer trained on internet text, had just demonstrated that making an autoregressive transformer bigger and feeding it more data reliably made it better, with no architectural changes required.

A third thread supplied a missing tool rather than a missing architecture. CLIP, released in January 2021, trained an image encoder and a text encoder jointly with a contrastive objective on hundreds of millions of image-caption pairs scraped from the internet, producing a shared embedding space in which a matching image and caption land close together and a mismatched pair lands far apart (CLIP: Bridging Text and Image). CLIP was not built to generate anything. It was built to score — to answer “how well does this image match this caption” for any image and any caption, without task-specific fine-tuning. A model with that scoring ability, released the same month as a wave of new generative work, was exactly the missing piece both halves of this chapter would reach for, in two different ways.

Diffusion itself, meanwhile, had just proven it could produce images at all without a discriminator: DDPM matched GAN-level quality on unconditional CIFAR-10 generation in June 2020 (Diffusion Foundations), but as of early 2021 nobody had yet pointed a diffusion model at open-ended natural-language captions.

Two backbones reach the same brief at the same lab within the same year, with CLIP as a shared tool neither one needed to exist. PNG ↓

The question the field had not yet answered was architectural, not conceptual: everyone agreed that text-to-image generation at GPT-3 scale was worth attempting, and that some way of scoring or conditioning on CLIP’s shared embedding space would help. What nobody knew in early 2021 was which generative backbone — the discrete-token autoregressive transformer that language modeling had already proven at scale, or the denoising diffusion process that had just matched GANs without a discriminator — would carry that scale furthest.

The Breakthrough

DALL-E (Ramesh et al., 2021) answered by reusing the VQ-VAE playbook from Pixels One at a Time almost unchanged, at a much larger scale. Training happened in two stages. Stage one trained a discrete variational autoencoder (dVAE) — the same discrete-codebook idea as VQ-VAE, optimized here with a Gumbel-softmax relaxation rather than a straight-through estimator — to compress a 256×256 RGB image into a 32×32 grid of tokens, each token one of 8,192 codebook entries. Stage two froze that dVAE and trained a 12-billion-parameter, GPT-3-styled decoder-only transformer — 64 self-attention layers, using row, column, and convolutional sparse-attention patterns across its layers to keep the cost of attending over a long sequence manageable — on a single concatenated stream of up to 256 BPE-encoded text tokens followed by the 1,024 image tokens from stage one, 1,280 tokens total. The transformer’s only job, exactly as in any autoregressive language model, was to predict each token in that stream from everything before it. Training data was 250 million text-image pairs collected from the internet.

Generation ran that transformer forward from a caption, sampling image tokens one at a time, then decoded the sampled 32×32 grid back into pixels through the dVAE’s decoder. Because sampling is stochastic, the same caption could produce many different completions, and DALL-E’s paper reported results generated with reranking: draw up to 512 candidate images per caption, then use CLIP — released by the same lab one month earlier — to score how well each candidate matched the caption, and keep the top-scoring ones. Nothing about the transformer itself required CLIP; CLIP’s role was purely a post-hoc filter, picking the best of many autoregressive rolls of the dice.

Text-to-image becomes a language modelling problem once an image has a token vocabulary. CLIP only reranks afterward; it never trains the transformer. PNG ↓

The result was a model that had never been trained on MS-COCO’s captions at all, evaluated in a genuinely zero-shot setting, reaching an FID on that benchmark within two points of the best prior approach trained specifically for it. Against DF-GAN, the strongest contemporary GAN baseline in the paper’s own comparison — a direct descendant of the StackGAN and AttnGAN lineage from First Words to Pictures — human evaluators picked DALL-E’s samples as more realistic 90% of the time and as the better caption match 93% of the time. The autoregressive, discrete-token recipe that language modeling had already proven at scale worked for images too, once an image had a token vocabulary to be autoregressive over.

GLIDE (Nichol et al., 2021), submitted ten months later by an overlapping set of authors at the same lab, asked whether the same problem could be solved without tokenizing the image at all. Its name, spelled out in the paper, is Guided Language to Image Diffusion for Generation and Editing, and its architecture is the ADM-style guided diffusion U-Net from Guidance — with one substitution: instead of conditioning on a fixed label $y$ from a thousand-class vocabulary, GLIDE conditions on an arbitrary caption. A transformer text encoder (24 residual blocks, width 2048) reads the caption; its final token’s embedding is used the way a class embedding is used elsewhere in the ADM architecture, and the full sequence of token embeddings from its last layer is made available to the diffusion U-Net as extra context at every attention layer, letting the denoising network draw on individual words rather than only a single pooled vector — the same instinct AttnGAN’s word-region attention captured for GANs in First Words to Pictures, now running inside a denoising network rather than a generator. (The formal cross-attention mechanism that later diffusion models use for text conditioning is developed in full in Latent Diffusion and Stable Diffusion; GLIDE’s version is an earlier, less formalized instance of the same idea.) The base model has 3.5 billion parameters — 2.3 billion in the diffusion U-Net, 1.2 billion in the text transformer — trained for 2.5 million iterations at batch size 2,048, plus a separate 1.5-billion-parameter diffusion model that upsamples the base model’s output from 64×64 to 256×256.

GLIDE swaps DALL-E's discrete tokens for a denoising loop conditioned on the same text encoder's output twice over, once pooled and once per word. PNG ↓

The paper’s central experiment compared two ways of steering that diffusion process, both descended from the guidance mechanisms taught in full in Guidance: classifier-free guidance, whose formula that chapter already develops, and a text-native adaptation of classifier guidance that substitutes a CLIP alignment score for a classifier’s log-probability — the new mechanism this paper contributes, developed with its own equation below. Human evaluators preferred classifier-free guidance’s outputs over CLIP guidance’s on both photorealism and caption match, and GLIDE shipped with classifier-free guidance as its default — the paper’s authors note this finding surprised them, since CLIP guidance seemed the more direct way to bring CLIP’s text-image alignment signal into the sampling loop. Against DALL-E itself, at matched 256×256 resolution, human evaluators preferred GLIDE’s samples for photorealism 87% of the time and for caption similarity 69% of the time — a result the paper frames explicitly against parameter count: “GLIDE was trained with roughly the same training compute as DALL-E but with a much smaller model (3.5 billion vs. 12 billion parameters).” The diffusion recipe, with guidance, beat the autoregressive recipe at roughly a quarter of the parameters, for similar training compute.

GLIDE also demonstrated an editing capability neither DALL-E nor any prior text-to-image GAN offered as a first-class feature: fine-tuning the same diffusion model for text-driven inpainting, by erasing random regions of training images and feeding the model the remaining pixels plus a mask channel as additional conditioning, alongside the caption. The resulting model could take a real photograph, an erased region, and a text prompt, and fill in only that region consistently with the prompt and the surrounding pixels — the direct ancestor of the editing techniques covered in Editing. For public release, OpenAI trained and shipped a smaller, 300-million-parameter model, GLIDE (filtered), on a newly gathered dataset of several hundred million images — deliberately kept disjoint from the dataset used to train the base model, DALL-E, and CLIP — with images of people, violent content, and hate symbols filtered out before training, rather than releasing the full 3.5-billion-parameter model — an early instance of a safety trade-off, shipping a weaker model rather than the strongest one trained, that recurs throughout this book’s later chapters on evaluation and provenance.

The Mathematics

DALL-E’s two-stage training is justified by a single evidence lower bound over the joint distribution of an image $x$ , its caption $y$ , and the dVAE’s discrete image tokens $z$ , factorized as $p_{\theta,\psi}(x,y,z) = p_\theta(x \mid y,z)\,p_\psi(y,z)$ . Notation follows Ramesh et al. (2021); the general evidence-lower-bound idea this instantiates is taught in full in Variational Autoencoders.

\ln p_{\theta,\psi}(x,y) \;\ge\; \mathbb{E}_{z \sim q_\phi(z \mid x)}\Big[\ln p_\theta(x \mid y,z) \;-\; \beta\, D_{\mathrm{KL}}\big(q_\phi(y,z \mid x)\, \|\, p_\psi(y,z)\big)\Big]

$q_\phi(z \mid x)$ is the dVAE encoder: given a real image $x$ , it produces a distribution over the 32×32 grid of discrete tokens $z$ . $p_\theta(x \mid y,z)$ is the dVAE decoder, reconstructing an image from those tokens (the caption $y$ plays no role in reconstruction, only in the second term). $p_\psi(y,z)$ is the transformer from stage two: its job is to model the joint distribution over captions and image tokens. Because $y$ and $x$ are both given, observed data rather than something being inferred, $q_\phi(y,z \mid x)$ in the KL term is the same encoder distribution as $q_\phi(z \mid x)$ with $y$ carried along unchanged — the notation makes both terms line up as distributions over the same pair $(y,z)$ . $\beta$ weights how strongly the encoder’s token assignments are pulled toward what the transformer already models; the bound is only exact at $\beta=1$ , but Ramesh et al. found $\beta = 6.6$ gave better codebook usage in practice. What this equation buys you: a principled reason to train the dVAE first and the transformer second rather than jointly end to end — stage one optimizes the reconstruction term with a fixed, uninformative prior over $z$ , and stage two, with the dVAE now frozen, optimizes the KL term by training $p_\psi$ to match the token distribution $q_\phi$ already produces.

Symbol	Meaning
$x$	The original RGB image
$y$	The caption, as BPE text tokens
$z$	The dVAE’s discrete image tokens, a 32×32 grid over an 8,192-entry codebook
$q_\phi(z \mid x)$	The dVAE encoder
$p_\theta(x \mid y,z)$	The dVAE decoder, reconstructing $x$ from tokens
$p_\psi(y,z)$	The stage-two transformer’s joint model over captions and tokens
$\beta$	Weight on the KL term; $6.6$ in practice, though the bound is exact only at $\beta=1$

GLIDE’s CLIP-guided sampling is the text-conditioned, contrastive-score version of the classifier guidance formula taught in full in Guidance; the only change is what supplies the gradient. Where classifier guidance uses a trained classifier’s log-probability of a class, GLIDE substitutes the dot product between a CLIP image embedding and a CLIP text embedding:

\hat{\mu}_\theta(x_t \mid c) = \mu_\theta(x_t \mid c) + s\,\Sigma_\theta(x_t \mid c)\,\nabla_{x_t}\big(f(x_t) \cdot g(c)\big)

$\mu_\theta(x_t \mid c)$ and $\Sigma_\theta(x_t \mid c)$ are the diffusion model’s own predicted mean and variance for the reverse step at noise level $t$ , conditioned on the caption $c$ — the same quantities Diffusion Foundations introduces. $f(x_t)$ is CLIP’s image encoder applied to the noisy image $x_t$ , and $g(c)$ is CLIP’s text encoder applied to the caption; their dot product $f(x_t) \cdot g(c)$ is exactly the alignment score CLIP is trained to produce, taught in full in CLIP: Bridging Text and Image. Taking the gradient of that score with respect to $x_t$ and adding it to the predicted mean, scaled by $s$ , nudges the sampling step toward images that CLIP scores as better matching the caption — structurally identical to classifier guidance’s gradient of a class log-probability, with a contrastive alignment score standing in for a classifier’s output. What this equation buys you: a way to steer diffusion sampling toward open-ended text, using a scoring model that was never trained to guide anything, rather than a classifier restricted to a fixed label set. The catch the paper reports directly: ordinary CLIP, trained only on clean images, scores noisy $x_t$ poorly, because a noisy intermediate sample is out of distribution for it — GLIDE had to train its own noised CLIP variant on images corrupted the same way the diffusion process corrupts them, to get a usable gradient at every step.

Symbol	Meaning
$x_t$	The noisy image at diffusion step $t$
$c$	The caption, the conditioning signal
$\mu_\theta, \Sigma_\theta$	The diffusion model’s predicted reverse-step mean and variance
$f(\cdot)$	CLIP’s image encoder
$g(\cdot)$	CLIP’s text encoder
$s$	The guidance scale
$\nabla_{x_t}$	Gradient with respect to the noisy image, computed by backpropagating through $f$

What It Enabled

The two papers left the field with a working comparison between backbones at similar training compute, not just similar ambition:

	DALL-E, Feb 2021	GLIDE, Dec 2021	Prior GAN lineage (DF-GAN)
Generative mechanism	Autoregressive transformer over dVAE tokens	Guided diffusion U-Net	Adversarial generator/discriminator
Parameters	12 billion	3.5 billion base, plus 1.5 billion upsampler	Tens of millions, typical for the era
Training data	250 million text-image pairs	Same 250 million pairs as DALL-E; filtered public release trained on a separate, disjoint set of several hundred million images	Narrow, curated caption datasets
Conditioning signal	Caption tokens, concatenated into the input sequence	Caption, via a transformer encoder feeding the U-Net	Pooled or per-word text embeddings
Editing / inpainting support	Not a first-class feature	Yes, via dedicated fine-tuning	Not a first-class feature
Human preference vs. DALL-E (photorealism / caption match)	—	87% / 69% preferred	—
Human preference vs. DF-GAN (photorealism / caption match)	90% / 93% preferred	—	—

Neither backbone won outright. DALL-E demonstrated that the token-plus-transformer recipe language modeling had already validated at GPT-3 scale transferred directly to images once VQ-VAE-style tokenization existed to carry it. GLIDE demonstrated that a smaller, guided diffusion model, trained on comparable compute, produced samples people preferred over DALL-E’s on both axes tested — and that diffusion, unlike the autoregressive route, came with a natural mechanism for text-driven editing built from the same training objective. Both results reached the same lab within the same year, and both fed directly into what followed: DALL-E 2, covered in The Year Images Went Mainstream, replaced DALL-E’s autoregressive backbone with a diffusion model conditioned on CLIP image embeddings — abandoning the architecture of its own predecessor in favor of GLIDE’s — while keeping CLIP-based reranking’s underlying instinct that a contrastive scorer is useful at generation time, not just for retrieval. Imagen, released the same year, followed GLIDE’s guided-diffusion structure with a much larger frozen text encoder in place of GLIDE’s jointly trained one. And running the same guided-diffusion recipe inside a pretrained autoencoder’s compressed latent space, rather than raw pixel space, is what made Stable Diffusion’s open-weights release affordable to train and run at consumer scale, covered in Latent Diffusion and Stable Diffusion.

Guided diffusion outgrows its own sibling within the year. The autoregressive branch does not disappear, but it waits until chapter 20 to return. PNG ↓

The autoregressive branch did not disappear after this year — it returned with a different token scheme in later work covered in The Current Frontier — but for the rest of this book’s Part II, the guided-diffusion recipe GLIDE demonstrated is the one every major shipped text-to-image system builds on.

DALL-E’s 12-billion-parameter decoder-only transformer could not be trained on a single accelerator’s memory, and its sparse row, column, and convolutional attention patterns across 64 layers existed specifically to keep the cost of attending over a 1,280-token sequence tractable at that width — the same model-parallel training problem GPT-3 had already forced the field to solve at even larger scale, covered in Model Parallelism. GLIDE’s smaller, 3.5-billion-parameter diffusion U-Net fit more comfortably on contemporary hardware, but paid for guidance with a doubled or tripled forward pass at every one of its sampling steps instead of a wider model at training time — the same training-compute-versus-inference-compute trade this book’s guidance chapter develops in full.

Key Papers

Paper	Authors	Year	Link	Why it matters
Zero-Shot Text-to-Image Generation	Ramesh, Pavlov, Goh, Gray, Voss, Radford, Chen, Sutskever	2021	arXiv:2102.12092	Scaled the VQ-VAE-plus-autoregressive-transformer recipe to 12 billion parameters and 250 million text-image pairs; introduced CLIP reranking of sampled candidates
GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models	Nichol, Dhariwal, Ramesh, Shyam, Mishkin, McGrew, Sutskever, Chen	2021	arXiv:2112.10741	Applied guided diffusion to open-ended text conditioning; showed classifier-free guidance preferred over CLIP guidance; preferred over DALL-E in human evaluation at under a third of the parameters; added text-driven inpainting

PM Lens

The number I keep coming back to from this chapter is 3.5 billion versus 12 billion, at roughly matched training compute, with the smaller model winning the human preference test. It is a direct, dated counterexample to the reflex that a bigger model is automatically the better bet, and it is worth having on hand whenever a vendor pitch leans on parameter count as the headline spec. The second habit this chapter set for me is checking which generative backbone a product actually uses before assuming continuity with its name: “DALL-E” the product went from an autoregressive transformer to a diffusion model between its first and second versions, a total architecture swap the branding never advertised, and the same caution applies to any tool whose vendor keeps the marketing name stable while quietly replacing what runs underneath it. I ask engineering the backbone question now before I write a roadmap slide that assumes last year’s constraints still hold this year.

Search