Control

It is February 2023. Stable Diffusion has been public for six months, and a text prompt has proven to be a blunt instrument: it steers subject matter, style, and rough composition, but it cannot pin an exact pose, trace a specific building’s outline, or hand the model a reference face and ask for that face. Six days apart, two papers propose the same fix from different angles — freeze the pretrained model and teach something small to read a second, non-text signal — and by August a third extends the idea from spatial layout to reference imagery itself.

Where We Were

By early 2023, text-to-image diffusion had two proven ways to narrow a generation beyond what a prompt alone could specify, and both had a structural limit. The first was personalization: Textual Inversion and DreamBooth, both from August 2022 (Chapter 16 covers them in full), taught a model a new subject or style by fine-tuning on a handful of images of that one thing. This worked, but it produced a new set of weights — or a new embedding — per concept, and it answered a different question than “put this exact shape where I specify.” A model fine-tuned to recognize a particular dog does not thereby gain any way to place that dog in a pose traced from a reference photograph.

The second was training-free editing. SDEdit (Meng et al., August 2021) injects a starting image partway into the forward noising process and lets the reverse process denoise it back out, nudging a rough sketch toward a photorealistic image; InstructPix2Pix (November 2022) turns natural-language instructions into edits of an existing image. Both live in Chapter 17, which owns both methods in full. Neither gives a model a literal, pixel-aligned map to follow — SDEdit’s fidelity to the input degrades quickly as the amount of injected noise grows, and instruction-following edits describe a change in words rather than specify one in coordinates. Classifier-free guidance (Chapter 10) sharpens how strongly a model listens to its conditioning, but it cannot manufacture conditioning richer than a prompt in the first place, and the cross-attention conditioning mechanism Latent Diffusion built into the U-Net (Chapter 13) was designed for text tokens, not for a depth map or a pose skeleton.

What was missing was a general slot for a second, spatial or visual input — one that any pretrained text-to-image model could accept without a full retraining run for every new concept, and without the base model’s own weights being disturbed enough to risk the visual quality that run of pretraining had bought.

Personalization and instruction-editing both arrived first; neither one could pin an exact shape. Two teams close that gap independently, six days apart. PNG ↓

The Breakthrough

“Adding Conditional Control to Text-to-Image Diffusion Models” (Zhang, Rao & Agrawala, 2023), submitted February 10, 2023, calls its method ControlNet. The idea is to keep every weight of the pretrained diffusion model exactly as it is, clone the block that needs to see the new conditioning signal into a second, trainable copy, and connect the two through a purpose-built kind of layer. The locked copy keeps whatever the base model learned from its original training data; the trainable copy learns only how to fold a new condition — a Canny edge map, a depth map, a human-pose skeleton, a semantic segmentation mask, an M-LSD line map, an HED soft-edge map, or a raw user sketch — into that pathway.

The connector is what makes this trainable without wrecking the frozen model on day one. A “zero convolution” is an ordinary 1×1 convolution whose weight and bias are initialized to zero. At the very first training step, a zero convolution outputs exactly zero, so the trainable copy’s contribution to the network is nothing at all — the combined model behaves identically to the untouched base model. As training proceeds, the zero convolution’s weights move away from zero gradually, letting the conditioning signal in a little at a time rather than all at once with whatever random, untrained noise a normally initialized layer would inject. The paper reports that this produces a “sudden convergence phenomenon”: the model does not gradually improve at following the conditioning image so much as abruptly start succeeding, typically in under ten thousand optimization steps.

Both zero convolutions start at exactly zero, so the combined block behaves identically to the untouched base model on the first training step. PNG ↓

ControlNet’s trainable copy is a full duplicate of Stable Diffusion’s encoding blocks and middle block — a substantial amount of new capacity, and Hugging Face’s own accounting of a packaged ControlNet checkpoint puts the added weight at roughly 700 million parameters on top of Stable Diffusion 1.5’s own 860-million-parameter U-Net. That size bought reliability — a lighter “ControlNet-lite” ablation the paper also tested scored worse in its own user study — but it also meant every new condition doubled the memory and compute footprint of a large fraction of the base model.

“T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models” (Mou et al., 2023), submitted six days later on February 16, 2023, takes the same starting premise — freeze the base model, learn something small — to a different architectural conclusion. Rather than cloning any part of the U-Net, T2I-Adapter is a separate, lightweight network that extracts features from the conditioning image at several resolutions and injects them into the frozen U-Net’s encoder at matching scales. The full adapter has around 77 million parameters and roughly 300 MB of storage; the paper also reports two compressed variants, an 18-million-parameter “adapter-small” and a 5-million-parameter “adapter-tiny,” trading some quality for a smaller footprint. Because nothing in the base U-Net is duplicated, several adapters trained independently — a sketch adapter and a color adapter, say, or depth and human pose together — can be combined at inference time by weighted summation of their injected features, without retraining either one, though the paper notes that balancing the combination weights when stacking adapters still needs manual tuning. T2I-Adapter’s training data was correspondingly modest: 164,000 images from COCO17 for the sketch adapter, 164,000 from COCO-Stuff for semantic segmentation, and 600,000 image-text pairs from LAION-Aesthetics for other conditions — a small fraction of the scale a from-scratch model would need, because the frozen base model still supplies everything about general image synthesis.

ControlNet’s own numbers show the cost of its design choice directly: on an A100 GPU, adding a ControlNet to a Stable Diffusion training run costs about 23% more GPU memory and 34% more time per iteration, and the paper trains its depth-conditioned model on 200,000 samples over five days on a single consumer RTX 3090 Ti. T2I-Adapter’s 77-million-parameter design and IP-Adapter’s 22-million-parameter design both chase the same goal from the other direction: keep the frozen base model’s memory footprint untouched and make the trainable addition small enough to fit the same consumer card the community was already using for inference. It is the image-generation instance of a trade-off Chapter 14 of the engine-room book develops in full for language models — freeze a large pretrained network, train a small module instead of the whole thing, and get most of the benefit of full fine-tuning for a fraction of the memory.

“IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models” (Ye et al., 2023), submitted August 13, 2023, changes what the second input is. ControlNet and T2I-Adapter both condition on spatial structure — a map that says where edges, depth discontinuities, or joints fall. IP-Adapter instead accepts a reference image and asks the model to draw on its content or style, the way a text prompt draws on a word’s meaning — “an image is worth a thousand words,” as the paper puts it, where prompt engineering struggles to describe a specific face, an exact texture, or a particular composition in words at all. Rather than cloning any part of the U-Net or building a separate feature-extraction network, IP-Adapter adds one new pair of projection matrices to every cross-attention layer already inside the frozen U-Net, so the reference image gets its own cross-attention pathway running in parallel with the existing text cross-attention, and the two are summed. The reference image itself is encoded with a frozen CLIP image encoder — the same contrastive embedding space Chapter 11 covers — so the new pathway has a well-defined, pretrained representation to attend over from the start rather than needing to learn one from raw pixels. Only those new projection matrices train; the paper reports 22 million trainable parameters in total, smaller than either prior method, trained on roughly 10 million image-text pairs drawn from LAION-2B and COYO-700M, on a single machine with eight V100 GPUs for one million steps at a batch size of eight per GPU.

Only the image branch's keys and values are new, 22 million parameters against the frozen base model. Text and image conditioning add together rather than compete. PNG ↓

The Mathematics

ControlNet’s core mechanism is the zero-convolution block itself. Given a frozen block $\mathcal{F}(\cdot; \Theta)$ from the pretrained network, a feature map $x$ , and a conditioning input $c$ , the block’s output is:

y_c = \mathcal{F}(x; \Theta) + \mathcal{Z}\big(\mathcal{F}(x + \mathcal{Z}(c; \Theta_{z1}); \Theta_c); \Theta_{z2}\big)

$\mathcal{F}(x; \Theta)$ is just the original, frozen block doing what it always did — the locked copy, contributing $y = \mathcal{F}(x; \Theta)$ on its own with no awareness of $c$ . The second term is the whole new pathway: the conditioning $c$ first passes through a zero convolution $\mathcal{Z}(\cdot; \Theta_{z1})$ , gets added to $x$ , and runs through $\mathcal{F}$ again — this time using a separate, trainable set of weights $\Theta_c$ cloned from $\Theta$ — before passing through a second zero convolution $\mathcal{Z}(\cdot; \Theta_{z2})$ and being added to the locked output. Because both zero convolutions start at exactly zero weight and bias, the entire second term is zero at initialization, so $y_c = y$ on the first training step: the frozen model’s behavior is perfectly preserved until training has actually taught the new pathway something. What it buys you: a way to add an arbitrarily large new capability to a pretrained network without any risk of the random initialization of that addition damaging the network before training begins.

Symbol	Meaning
$x$	the input feature map to this block
$c$	the conditioning image, e.g. an edge map or pose skeleton
$\mathcal{F}(\cdot; \Theta)$	the original pretrained block, kept frozen
$\Theta$	the frozen weights of the locked copy
$\Theta_c$	the trainable weights of the cloned copy, initialized from $\Theta$
$\mathcal{Z}(\cdot; \Theta_{z1}), \mathcal{Z}(\cdot; \Theta_{z2})$	zero convolutions, weight and bias initialized to zero
$y_c$	the block’s output once the trainable pathway is added

Training itself uses the same denoising objective Chapter 8 establishes, extended with a second conditioning argument alongside the text prompt:

\mathcal{L} = \mathbb{E}_{z_0,\, t,\, c_t,\, c_f,\, \epsilon \sim \mathcal{N}(0,1)}\Big[\big\Vert \epsilon - \epsilon_\theta(z_t, t, c_t, c_f) \big\Vert_2^2\Big]

This is the ordinary DDPM noise-prediction loss — the network $\epsilon_\theta$ is trained to predict the noise $\epsilon$ that was added to produce the noisy latent $z_t$ at timestep $t$ , exactly as in Chapter 8 — with one change: the network now sees two conditioning signals instead of one, the text prompt $c_t$ it already understood and the new spatial condition $c_f$ ControlNet adds. Because $\Theta$ itself never updates, all of the adaptation this loss drives happens in the small set of new weights, which is what keeps the pretrained model’s general image quality intact while it learns to respect $c_f$ . What it buys you: the entire diffusion training machinery this book already develops, reused without modification, just with one extra argument threaded through it.

Symbol	Meaning
$z_0, z_t$	the clean and noised latent at timestep $t$
$t$	the diffusion timestep
$c_t$	the text conditioning, unchanged from the base model
$c_f$	the new spatial conditioning image
$\epsilon$	the true noise added by the forward process
$\epsilon_\theta$	the network’s noise prediction, now a function of both conditions

IP-Adapter’s decoupled cross-attention keeps the base model’s existing text cross-attention pathway untouched and runs a second one beside it for the reference image, then sums the two:

\begin{aligned} \mathbf{Q} &= \mathbf{Z}\mathbf{W}_q, \qquad \mathbf{K} = \mathbf{c}_t\mathbf{W}_k, \qquad \mathbf{V} = \mathbf{c}_t\mathbf{W}_v, \qquad \mathbf{K}' = \mathbf{c}_i\mathbf{W}'_k, \qquad \mathbf{V}' = \mathbf{c}_i\mathbf{W}'_v \\ \mathbf{Z}^{\text{new}} &= \mathrm{Softmax}\!\left(\frac{\mathbf{Q}\mathbf{K}^\top}{\sqrt{d}}\right)\mathbf{V} + \mathrm{Softmax}\!\left(\frac{\mathbf{Q}(\mathbf{K}')^\top}{\sqrt{d}}\right)\mathbf{V}' \end{aligned}

The query $\mathbf{Q}$ — the U-Net’s own hidden state, asking “what should I attend to here” — is shared between both branches, exactly as the query, key, and value mechanism works in the transformer’s attention (Chapter 5 of the language-models book). The first branch is the ordinary text cross-attention the base model already had: keys and values $\mathbf{K}, \mathbf{V}$ come from the text embedding $\mathbf{c}_t$ through the original, frozen weights $\mathbf{W}_k, \mathbf{W}_v$ . The second branch is entirely new: keys and values $\mathbf{K}', \mathbf{V}'$ come from the reference image’s CLIP embedding $\mathbf{c}_i$ , through a new pair of weight matrices $\mathbf{W}'_k, \mathbf{W}'_v$ that are initialized as copies of $\mathbf{W}_k, \mathbf{W}_v$ and are the only weights this method trains. The two attention outputs are simply added, so the model attends to text and image conditioning in parallel rather than one replacing or crowding out the other. What it buys you: image-prompt conditioning that composes with any text prompt already understood by the frozen model, for the cost of two small matrices per cross-attention layer instead of a cloned network or a separate feature-injection pipeline.

Symbol	Meaning
$\mathbf{Z}$	the U-Net’s hidden state at this layer
$\mathbf{c}_t$	the text embedding, unchanged conditioning source
$\mathbf{c}_i$	the reference image’s CLIP embedding
$\mathbf{W}_q, \mathbf{W}_k, \mathbf{W}_v$	the frozen text cross-attention weights
$\mathbf{W}'_k, \mathbf{W}'_v$	the new, trainable image cross-attention weights
$\mathbf{Z}^{\text{new}}$	the combined output, text and image attention summed
$d$	the key dimension, used to scale the dot product

What It Enabled

ControlNet’s own user study, comparing sketch-conditioned generation against two prior approaches on a 20-sketch, 12-rater panel scored one to five, shows how large the gap to the previous state of the art was:

Method	Quality (1–5)	Fidelity (1–5)
PITI	1.10 ± 0.05	1.02 ± 0.01
Sketch-Guided Diffusion, $\beta=3.2$	2.52 ± 0.44	3.28 ± 0.72
ControlNet-lite (ablation)	3.93 ± 0.59	4.09 ± 0.46
ControlNet	4.22 ± 0.43	4.28 ± 0.45

The same paper’s depth-conditioning comparison against Stable Diffusion v2’s own purpose-built depth-to-image model found raters correct only 0.52 ± 0.17 of the time at telling the two apart — indistinguishable from chance, despite ControlNet’s depth model being trained by a much smaller team on far less compute than an official model release.

The three methods this chapter covers do not simply compete; they occupy different points on a cost-versus-scope trade-off that the field kept using after 2023:

	ControlNet	T2I-Adapter	IP-Adapter
Submitted	Feb 10, 2023	Feb 16, 2023	Aug 13, 2023
Conditioning type	Spatial maps — edges, depth, pose, segmentation	Spatial maps — sketch, depth, keypose, segmentation	Reference image content and style
Mechanism	Cloned encoder blocks, zero convolutions	Separate lightweight feature-injection network	New cross-attention branch per layer
Base model modified	No — frozen; new copy runs alongside	No — frozen; adapter runs alongside	No — frozen; new weights only
Added trainable parameters	≈ 700M added to SD 1.5’s 860M U-Net	≈ 77M (18M / 5M compressed variants)	22M
Training data	Per-condition, up to roughly 1M pairs	164K–600K images per condition	≈ 10M pairs, LAION-2B and COYO-700M
Composability	One ControlNet per condition, stackable	Multiple adapters, weighted-sum combination	Combines with text prompts and with ControlNet

None of the three needed the base model’s own training data or the compute of the original Stable Diffusion run — a point that matters as much as the architecture. Because the pretrained weights never move, a single ControlNet or adapter, trained once, works with any fine-tune, merge, or style checkpoint sharing the same base architecture, and multiple adapters can run at once. That composability is what let downstream tooling — the open-source interfaces that grew up around Stable Diffusion through 2023 — treat “add a ControlNet” and “add an IP-Adapter” as generic, stackable operations rather than one-off integrations, in the same way Chapter 16’s LoRA fine-tunes became generic, stackable additions to a base checkpoint. A workflow combining a DreamBooth-trained subject, a LoRA style, a ControlNet pose map, and an IP-Adapter reference face — four separately trained, separately frozen additions layered on one base model — became a routine construction rather than a research result.

The limits were the ones already visible in the numbers above: a ControlNet checkpoint is heavier than the model it controls by nearly the size of the original U-Net, IP-Adapter’s reference-image conditioning competes with the text prompt for the model’s attention in ways that still need per-application tuning of a balance weight, and multi-adapter combinations inherited T2I-Adapter’s own caveat that stacking conditions well is a manual tuning problem, not something guaranteed by the architecture. Later diffusion-transformer backbones (Chapter 19) would need their own answer to the same question, since a zero-convolution block cloned from a U-Net does not transplant unchanged onto an architecture with no U-Net to clone.

All three adapters converge on the same composable pattern; only the diffusion-transformer backbones two chapters later break the pattern's core assumption. PNG ↓

Key Papers

Paper	Authors	Year	Link	Why it matters
Adding Conditional Control to Text-to-Image Diffusion Models	Zhang, Rao & Agrawala	2023	arXiv:2302.05543	Introduces ControlNet: locked and trainable copies connected by zero convolutions, letting a frozen diffusion model learn precise spatial conditioning without disturbing its pretrained weights
T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models	Mou et al.	2023	arXiv:2302.08453	Achieves comparable spatial control with a far smaller, separate adapter network and demonstrates composable, weighted-sum combination of multiple conditions
IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models	Ye et al.	2023	arXiv:2308.06721	Extends adapter-based control from spatial maps to reference images via a decoupled cross-attention branch, with only 22M trainable parameters

PM Lens

The lesson I keep coming back to from this trio is that “control” and “quality” turned out to be separable problems, and separable problems can be solved by separate, small, swappable pieces instead of one bigger model. When I evaluate an image or video vendor’s roadmap now, “can I hand it a pose, a depth map, or a reference face and have it obeyed” is a different question from “how good are its unconstrained samples,” and a team that has only solved the second one is not close to shipping a production creative tool — most real workflows, from ad creative to product photography, need the first far more than they need another half-point of benchmark quality. It is also a cost question I have had to explain to non-technical stakeholders: a ControlNet-class add-on can be nearly as large as the base model it controls, while an IP-Adapter-class add-on is a rounding error, and that difference shows up directly in inference latency and GPU spend at scale, not just in a research paper’s parameter count table.

Search