Guidance

It is May 2021. Diffusion models can already match the likelihoods of other generative approaches, and faster samplers have cut the cost of drawing from them, but on the benchmark that matters most for images — class-conditional synthesis on ImageNet — they still look soft and generic next to the sharpest GANs. Over the following fourteen months, two papers hand diffusion a dial: turn it one way for fidelity, the other for diversity, and for the first time a diffusion model beats a GAN on its own turf.

Where We Were

By mid-2020, diffusion had a working denoising objective. DDPM (Ho et al., 2020) trains a network $\epsilon_\theta$ to predict the noise added to an image at each step of a fixed forward corruption process, then runs that prediction in reverse to generate samples — the mechanism lives in Chapter 8. Sampling was slow, requiring on the order of a thousand steps, until DDIM and the score-based SDE view later that year showed the same trained models could be run with far fewer, better-chosen steps; that reformulation is Chapter 9’s subject. By early 2021, diffusion models were fast enough and theoretically grounded enough to be judged seriously against the field’s reigning architecture for high-fidelity image synthesis: the GAN.

The incumbent set a high bar. BigGAN-deep (Brock et al., September 2018) — a class-conditional GAN scaled up on ImageNet, covered in Chapter 6 — reached an FID of 6.02 at 128×128 resolution, 6.95 at 256×256, and 8.43 at 512×512. Its samples were the sharpest and most texturally detailed the field had produced, at the cost of the training instability and mode collapse that motivated Chapter 4. A plain class-conditional diffusion model, by contrast, was stable to train and covered the data distribution more completely, but its samples looked soft: given only a class label, the model did not lean on that label hard enough, drifting toward generic, low-detail images of the requested class rather than confident, specific ones. The gap was not really “diffusion versus GANs” — it was one narrower failure, and it had a name once someone went looking for it: conditioning that was too weak.

Diffusion spends three years becoming stable and fast without yet becoming sharp. The gap the coral card names, weak conditioning, is exactly what guidance closes next. PNG ↓

The Breakthrough

“Diffusion Models Beat GANs on Image Synthesis” (Dhariwal & Nichol, 2021), submitted to arXiv in May 2021, closed the gap with two changes: an improved U-Net (more attention heads, timestep-conditioned normalization, and a handful of tuning choices this chapter does not need to re-derive) and a new sampling technique, classifier guidance. The insight behind it is that a diffusion model’s noise prediction is already equivalent to a score — a gradient of a log-density — at every step (Chapter 9 develops the score view in full), and gradients of different log-densities simply add. So train a second network, a classifier $p_\phi(y \mid x_t)$ that recognizes the target class $y$ from a partially noised image $x_t$ at every noise level the diffusion process visits, and at each sampling step nudge the predicted noise in the direction that increases that classifier’s confidence — scaled by a hyperparameter $s$ .

Every one of a few hundred sampling steps repeats this same loop: ask the classifier again, nudge the prediction, step once more. PNG ↓

The effect was decisive. Sampling 250 steps on ImageNet at a guidance scale of $s=1.0$ , FID at 256×256 fell from 10.94 unguided to 4.59 guided — enough to pass BigGAN-deep’s 6.95. At 128×128 the guided model reached 2.97 against BigGAN-deep’s 6.02, and at 512×512, 7.72 against 8.43. Pushing $s$ above 1.0 traded further: sharper, more class-typical images at some cost to how much of the class’s diversity any single sample showed — the paper describes this as trading recall for precision. Classifier guidance was diffusion’s first credible, quantified route past the GAN benchmark, and it made a second point that mattered as much as the numbers: guidance strength was now a knob a practitioner could turn at sampling time, not a property fixed when the model was trained.

The knob had a structural cost. Classifier guidance needed a second network trained from scratch on the diffusion process’s own noise schedule — an ordinary ImageNet classifier, trained only on clean images, has never seen anything as noisy as an early-step $x_t$ . The paper builds this classifier cheaply, by reusing the diffusion U-Net’s own downsampling trunk and terminating it early with an attention-pooling head, but reuse of architecture did not mean reuse of training: the classifier still needed its own training run, on the same noised images, before it could be used. Every sampling step then paid for a classifier forward pass and a backward pass through it, on top of the diffusion model’s own forward pass. And the technique could only condition on whatever discrete labels the classifier had learned — a thousand ImageNet classes, not the open-ended text prompts the field wanted within the year.

“Classifier-Free Diffusion Guidance” (Ho & Salimans, 2022), submitted in July 2022, removed the second network by removing the classifier’s job rather than just its cost. During training, the diffusion model is shown its conditioning $c$ — a class label, or later a text embedding — with some probability, and shown a fixed null token with the remaining probability, so one network learns both a conditional prediction $\epsilon_\theta(x_t, c)$ and, for free, an unconditional prediction $\epsilon_\theta(x_t)$ . At sampling time both predictions come from that same network, and the guided estimate is an extrapolation away from the unconditional prediction and toward the conditional one — no classifier, no second network, no extra labeled data.

One set of weights learns both jobs through conditioning dropout, so sampling can extrapolate between them with no second network. PNG ↓

The Mathematics

Classifier guidance and classifier-free guidance both modify the same object: the noise, or score, the network predicts at each denoising step, developed in full in Chapter 8. They arrive there by different routes.

Both formulas below lean on a fact Chapter 9 establishes in full: a diffusion model’s noise prediction and the score of the noisy-data distribution are the same object up to a known rescaling, $\epsilon_\theta(x_t, t) \approx -\sqrt{1-\bar\alpha_t}\,\nabla_{x_t}\log p_\theta(x_t)$ . That correspondence is what lets a gradient — something classifiers and score-based samplers speak fluently — be added directly to a noise prediction, rather than requiring the two formulas to be reconciled through some separate change of variables.

Classifier guidance rests on the fact that gradients of log-probabilities add when probabilities multiply, then substitutes that sum into the noise prediction the sampler actually uses:

\begin{aligned} \nabla_{x_t} \log\big(p_\theta(x_t)\,p_\phi(y \mid x_t)\big) &= \nabla_{x_t} \log p_\theta(x_t) + \nabla_{x_t} \log p_\phi(y \mid x_t) \\ \hat{\epsilon} &= \epsilon_\theta(x_t, t) - s\sqrt{1-\bar\alpha_t}\,\nabla_{x_t} \log p_\phi(y \mid x_t) \end{aligned}

The first line says that the score of the joint density “this is a plausible image, and it looks like class $y$ ” splits cleanly into two independent pieces: the diffusion model’s own score $\nabla_{x_t} \log p_\theta(x_t)$ , which says how to make $x_t$ look more like a real image, plus the classifier’s score $\nabla_{x_t} \log p_\phi(y \mid x_t)$ , which says how to make it look more like class $y$ . The second line turns that sum into something the sampler can use directly: $\epsilon_\theta(x_t, t)$ is the diffusion model’s ordinary noise prediction, and subtracting the scaled classifier gradient — scaled by $s$ and by the same $\sqrt{1-\bar\alpha_t}$ factor that relates noise predictions to scores at step $t$ — steers the denoising step toward images the classifier is confident about. What it buys you: a second, independently trained signal that pulls samples toward a specific class, with $s$ as an explicit fidelity-versus-diversity dial.

Symbol	Meaning
$x_t$	the noisy image at diffusion step $t$
$p_\theta(x_t)$	the diffusion model’s implicit density over noisy images
$p_\phi(y \mid x_t)$	a classifier trained to recognize class $y$ from a noisy $x_t$
$y$	the target class label
$\hat{\epsilon}$	the guided noise prediction actually used to take the sampling step
$\epsilon_\theta(x_t, t)$	the diffusion model’s unguided noise prediction
$s$	the guidance scale, a tunable hyperparameter
$\bar\alpha_t$	the cumulative signal-retention factor of the forward noising process at step $t$

Classifier-free guidance keeps the same one-model-only setup and skips the classifier entirely, extrapolating between the network’s conditional and unconditional predictions:

\tilde{\epsilon}_\theta(x_t, c) = (1+w)\,\epsilon_\theta(x_t, c) - w\,\epsilon_\theta(x_t)

$\epsilon_\theta(x_t, c)$ is the network’s noise prediction when it is told the conditioning $c$ , and $\epsilon_\theta(x_t)$ is the same network’s prediction when it is given the null token instead — both come from one set of weights, because conditioning dropout during training taught the network to do both jobs. The guided estimate $\tilde{\epsilon}_\theta$ does not just use the conditional prediction; it overshoots past it, in the direction away from the unconditional prediction, by a factor controlled by $w$ . At $w=0$ this is the ordinary conditional model; larger $w$ pushes samples further from “a generic image” and further toward “unmistakably an image of $c$ .” What it buys you: the same fidelity-versus-diversity dial as classifier guidance, with no second network, no extra training data, and any conditioning signal the base model already understands.

Symbol	Meaning
$\tilde{\epsilon}_\theta$	the guided noise prediction used at sampling time
$\epsilon_\theta(x_t, c)$	the network’s prediction conditioned on $c$
$\epsilon_\theta(x_t)$	the same network’s prediction with null conditioning
$c$	the conditioning signal — a class label, later a text embedding
$w$	the guidance weight, a tunable hyperparameter
$x_t$	the noisy image at diffusion step $t$

The two formulas are the same idea in different clothing. The CFG paper shows that the difference between its conditional and unconditional predictions, $\epsilon_\theta(x_t, c) - \epsilon_\theta(x_t)$ , plays the same role — up to the $\sqrt{1-\bar\alpha_t}$ scaling in the equation above — as $\nabla_{x_t} \log p(c \mid x_t)$ , the classifier gradient classifier guidance needed a trained network to supply. Classifier-free guidance did not remove that gradient from the math; it observed that a model trained both conditionally and unconditionally already contains it, for the price of one extra forward pass instead of one extra network.

The paper’s own ImageNet 64×64 ablation, with conditioning dropped 10% of the time during training ( $p_{\text{uncond}}=0.1$ ), shows the trade-off directly:

Guidance weight $w$	FID	Inception Score
0.0	1.80	53.71
0.1	1.55	66.11
0.3	3.03	92.8
1.0	12.6	170.1

FID — Fréchet Inception Distance, the field’s standard distributional-similarity metric, introduced with the TTUR paper and covered in Chapter 21 — bottoms out around a modest guidance weight and then rises again, while Inception Score, a rougher proxy for how confidently classifiable and sharp individual samples look, keeps climbing. That crossing point is the guidance dial made numeric: somewhere past it, “more typical of the class” stops meaning “closer to the real data distribution” and starts meaning “less diverse.”

What It Enabled

The clearest before-and-after is the comparison the ADM paper set out to win:

Model	128×128 FID	256×256 FID	512×512 FID
BigGAN-deep, 2018	6.02	6.95	8.43
Diffusion, unguided	5.91	10.94	23.24
Diffusion, classifier guidance, $s=1.0$	2.97	4.59	7.72

Guidance, not the architecture change alone, is what turned “roughly competitive” into “clearly ahead,” and it did so at every resolution tested, with the margin widening as resolution grew.

Classifier guidance itself saw limited further use once the classifier-free alternative existed: it needed a second network retrained for every new noise schedule or resolution, and it could only condition on a fixed, labeled vocabulary. Classifier-free guidance conditions on anything the base model already conditions on, which is what let it travel. GLIDE (December 2021) tested both a CLIP-based classifier and classifier-free guidance for text-to-image generation and found the classifier-free variant preferred by human raters — that comparison, and the text-conditioning setup itself, is Chapter 12’s subject. From GLIDE forward, conditioning dropout and a user-facing guidance-scale control became close to standard: DALL-E 2 and Imagen in 2022 (Chapter 14), Stable Diffusion the same year (Chapter 13), and the rectified-flow and diffusion-transformer models that followed (Chapter 19) all trained with conditioning dropout and exposed some form of the guidance weight to the people using them.

	Classifier guidance, 2021	Classifier-free guidance, 2022
Extra network at sampling time	yes, a noise-aware classifier	no
Extra compute per step	classifier forward and backward pass	one extra diffusion forward pass
Conditioning signal	fixed label set the classifier was trained on	anything the base model already conditions on
Change to diffusion training	none	conditioning dropout
Long-run adoption	limited to labeled, class-conditional settings	GLIDE, DALL-E 2, Imagen, Stable Diffusion, and later models

The diversity cost is not a footnote. A guidance weight tuned for the sharpest-looking single sample is, by construction, a weight tuned to under-represent whatever modes of a class or prompt are less typical — a model guided hard toward “confident and class-typical” will sample the frequent, stereotypical version of a concept more often than the training data’s own variety would justify. That trade-off, first made explicit and measurable here, is an early instance of the representation and evaluation questions Chapter 21 takes up in full once guidance is standard equipment on production text-to-image systems rather than a research ablation.

Once the classifier was removed, the same guidance dial travelled to every major text-to-image system that followed. PNG ↓

Key Papers

Paper	Authors	Year	Link	Why it matters
Denoising Diffusion Probabilistic Models	Ho et al.	2020	arXiv:2006.11239	Establishes the noise-prediction network $\epsilon_\theta$ that both guidance formulas modify
Diffusion Models Beat GANs on Image Synthesis	Dhariwal & Nichol	2021	arXiv:2105.05233	Introduces classifier guidance; first diffusion model to beat a strong GAN benchmark on ImageNet FID
Classifier-Free Diffusion Guidance	Ho & Salimans	2022	arXiv:2207.12598	Removes the auxiliary classifier via conditioning dropout; becomes the guidance mechanism behind nearly all later text-to-image diffusion models

PM Lens

The guidance scale is the one piece of diffusion-model math that ships as a literal slider. Every Stable Diffusion interface I have used, and the APIs built on top of it, exposes a CFG scale control, and complaints about “washed out” or “over-saturated, plastic-looking” outputs are almost always a guidance value pushed too high, not a broken model. That is worth knowing before promising a team that a fine-tune will fix an aesthetics complaint — half the time the fix is turning a number down, not retraining anything. The deeper pattern generalizes past images. “Spend more inference compute for a controllable amount of extra quality” showed up here first, as two forward passes per diffusion step, years before the same trade became the headline story of reasoning-model test-time compute. It is worth asking, of any new “quality knob” a vendor pitches, whether it is genuinely new or this same trade wearing a different name.

Search