Editing

It is August 2021. Diffusion models can generate a photorealistic image from noise, but almost nothing yet lets a user hand one an existing photo and ask for a change. Over the following fifteen months, three papers turn diffusion into an editor rather than only a generator — first by treating a real image as a partly-denoised starting point, then by treating a diffusion model’s own attention maps as a steering wheel, and finally by baking both tricks into a single model that takes a written instruction and edits in one forward pass.

Where We Were

By the middle of 2021, two separate toolkits existed for changing pixels that already existed, and neither one was built for the job. The GAN toolkit, covered in the StyleGAN lineage of Chapter 6, edited by walking through a generator’s latent space: invert a real photo into a latent code with an encoder or an optimization loop, move that code along a direction that corresponds to the attribute being changed (older, smiling, a different hairstyle), and decode. The approach worked well for faces and other domains a single GAN had been trained on, but every step of it was fragile. Inversion was lossy — a photo pushed backward through a generator’s forward mapping rarely round-trips exactly — and finding a usable edit direction for anything beyond the handful of attributes researchers had already characterized meant more training, more labeled data, or both.

The diffusion toolkit, by contrast, barely had an editing story at all. DDPM (Ho et al., 2020) and the faster samplers built on it, covered in Chapter 8 and Chapter 9, described how to turn noise into an image, not how to turn one image into a related one. DDIM (Song et al., 2020), covered in the same Chapter 9, had shown by October 2020 that the reverse process could be run deterministically, which is what would later make it invertible. CLIP (Radford et al., 2021), covered in Chapter 11, had just shown that a single model could score how well an image matched a text description — a measurement diffusion models would soon lean on heavily for guidance, but a measurement is not yet an editing method. The one diffusion-adjacent editing technique in circulation, mask-based inpainting, worked by re-generating only the pixels inside a user-drawn mask while holding the rest of the image fixed at every denoising step — effective for filling a hole, but useless for changing something without first telling the model exactly which pixels to touch. By August 2021, StyleGAN-style latent walks, DDPM, DDIM, and CLIP were each in place, but nothing yet combined them into a general, mask-free way to edit a real photo with diffusion.

What was missing was a mechanism that used a pretrained diffusion model as a prior for editing without demanding a mask, a labeled edit-direction dataset, or a from-scratch retraining for every new kind of change. Diffusion’s own machinery — a well-defined forward noising process, and DDIM’s proof (covered in Chapter 9) that the reverse process could be run deterministically — turned out to already contain the ingredients.

The Breakthrough

SDEdit (Meng et al., 2021), submitted to arXiv in August 2021, made the first move, and it required no new training at all. The insight was to stop treating the forward noising process as something that only happens during training, and start treating it as an editing tool: take a real image — or a rough user stroke painting, or two images crudely composited together — add a partial amount of noise to it, stopping partway through the forward process instead of running it all the way to pure noise, then run the ordinary reverse process, using a pretrained diffusion model, from that partial-noise point back down to a clean image. The added noise erases exactly enough of the input’s low-level detail that the reverse process has room to make it look like a plausible photograph, while leaving enough of the input’s coarse structure intact that the diffusion model has no reason to depart from it. One knob, how much noise to add, directly trades one property for the other.

SDEdit’s editing mechanism needed a pretrained generative diffusion prior and nothing else: no mask, no classifier, no per-image optimization loop, and no task-specific loss function of the kind GAN-based editing methods required for each new application. The same mechanism handled stroke-based image synthesis, editing a real photograph, and blending a composited region into its surroundings, because all three are the same operation — partial noise, then denoise — applied to a different kind of input.

“Prompt-to-Prompt Image Editing with Cross Attention Control” (Hertz et al., 2022), submitted in August 2022, attacked a different failure mode. Text-to-image diffusion models like Imagen (Chapter 14 covers Imagen’s text conditioning in full) are so sensitive to their prompt that changing one word — “a photo of a bicycle” to “a photo of a car” — tends to regenerate an almost entirely different image, not the same scene with only the requested object swapped. State-of-the-art text-driven editing at the time worked around this by asking the user to supply a spatial mask marking exactly where the edit should apply, which sacrificed the point of editing by text in the first place: the user still had to specify a region by hand.

Prompt-to-Prompt’s authors traced the sensitivity to a specific place inside the model: the cross-attention layers that fuse the text prompt with the image being generated at every denoising step. These are the same query-key-value attention layers Chapter 5 of the language-models book develops in general; a latent diffusion U-Net reuses that mechanism unchanged, with image features supplying the queries and text-token embeddings supplying the keys and values. The resulting attention map, one per text token per denoising step, records which pixels each word in the prompt is currently influencing — effectively a live, per-step layout dictated by the prompt. The authors’ finding was that this map is close to fixed early in generation and mostly just gets refined afterward, which means an edited prompt’s spatial layout can be forced to match the original prompt’s layout simply by injecting the original run’s attention maps into the edited run for the early denoising steps, then letting the edited prompt’s own attention take over. No mask is drawn; the original image’s own attention pattern acts as the mask.

“InstructPix2Pix: Learning to Follow Image Editing Instructions” (Brooks, Holynski, and Efros, 2022), submitted in November 2022, folded both prior tricks into one product-shaped model. Rather than asking a user to write two prompts (before and after) or tune a noise-level knob, InstructPix2Pix took a single input image and a plain-language instruction — “make it winter,” “turn the horse into a zebra” — and edited in a single forward pass, with no mask, no inversion, and no per-image optimization at inference time. Its contribution was not a new editing mechanism at the diffusion-sampling level; it was a way to manufacture enough training data to distill the previous two tricks into a directly instructable model. The authors first fine-tuned GPT-3 (Brown et al., 2020) on a small, hand-written set of captions paired with editing instructions and edited captions, then used the fine-tuned GPT-3 to generate several hundred thousand new instruction-and-caption-pair triples at scale. Each caption pair was then turned into a before-and-after image pair using Stable Diffusion together with Prompt-to-Prompt’s attention injection — exactly the mechanism described above — filtering with a CLIP-based similarity check to discard pairs where the “after” image had drifted too far from the “before” image or the instruction. That process produced 454,445 example triples of an input image, an editing instruction, and an edited image, entirely synthetically, with no human-edited photograph anywhere in the pipeline.

A conditional latent diffusion model, initialized from a pretrained Stable Diffusion checkpoint, was then fine-tuned on that synthetic dataset to predict an edited image directly from an input image and an instruction — no intermediate attention-map surgery, no noise-level tuning, just a learned mapping from an image and an instruction to an edited image. What Prompt-to-Prompt did by manipulating a diffusion model’s internals at every sampling step, InstructPix2Pix reproduced by training a new model to imitate the outputs of that manipulation, once, up front, and then never needing it again at inference time.

The Mathematics

SDEdit poses editing on top of the continuous-time variance-exploding SDE that Chapter 9 develops in full, where noise is added without shrinking the signal — unlike DDPM’s variance-preserving process in Chapter 8, which rescales the image at every step. Given a guide image $x^{(g)}$ and a chosen intermediate time $t_0$, the guide is corrupted to exactly the noise level the SDE reaches at $t_0$:

$$x^{(g)}(t_0) = x^{(g)} + \sigma(t_0)\,\epsilon, \qquad \epsilon \sim \mathcal{N}(0,\mathbf{I})$$

Symbol	Meaning
$x^{(g)}$	the guide image: a real photo, a stroke painting, or a composite
$x^{(g)}(t_0)$	the guide after partial noising, at time $t_0$
$t_0$	the chosen intermediate noise level, the editing chapter’s one tunable knob
$\sigma(t_0)$	the noise scale the forward SDE has reached by time $t_0$
$\epsilon$	standard Gaussian noise

Increasing $t_0$ buys realism at faithfulness’s expense, and SDEdit’s authors bound exactly how much faithfulness is given up, with high probability, as a function of $t_0$ alone:

$$\Pr\Big[\big\|x^{(g)} - \mathrm{SDEdit}(x^{(g)};t_0,\theta)\big\|_2^2 \le \sigma^2(t_0)\big(C\sigma^2(t_0) + d + 2\sqrt{-d\log\delta} - 2\log\delta\big)\Big] \ge 1-\delta$$

Symbol	Meaning
$x^{(g)}$	the original guide image
$\mathrm{SDEdit}(x^{(g)};t_0,\theta)$	the edited output, as a function of the guide, the noise level, and the model’s weights $\theta$
$\sigma^2(t_0)$	the noise variance injected up to time $t_0$
$C$	a constant depending on the score model’s smoothness
$d$	the dimensionality of the image
$\delta$	the failure probability of the bound

Prompt-to-Prompt’s cross-attention maps, $M_t$ for the original prompt and $M_t^{*}$ for the edited prompt at denoising step $t$, are the ordinary softmax attention output Chapter 5 of the language-models book defines — this chapter does not re-derive that formula, only what is done with its result. Word-swap editing is a single piecewise rule for which map the sampler actually uses at each step:

$$\mathrm{Edit}(M_t, M_t^{*}, t) = \begin{cases} M_t^{*} & t < \tau \\[2pt] M_t & t \ge \tau \end{cases}$$

Symbol	Meaning
$M_t$	the cross-attention map at step $t$ under the original prompt
$M_t^{*}$	the cross-attention map at step $t$ under the edited prompt
$t$	the current denoising step
$\tau$	the step threshold after which the edited prompt’s own attention is used

A second operation, attention re-weighting, changes how strongly a single word is expressed without swapping it for another word at all:

$$\big(\mathrm{Edit}(M_t, M_t^{*}, t)\big)_{i,j} = \begin{cases} c \cdot (M_t)_{i,j} & j = j^{*} \\[2pt] (M_t)_{i,j} & \text{otherwise} \end{cases}$$

Symbol	Meaning
$c$	the re-weighting factor applied to one token’s attention
$i$	a spatial location in the image
$j$	a text token index
$j^{*}$	the specific token being re-weighted
$(M_t)_{i,j}$	attention weight from pixel $i$ to token $j$ at step $t$

InstructPix2Pix trains a single network to predict noise from three things — a noisy latent, an input image, and a text instruction — and extends classifier-free guidance, covered in full in Chapter 10, from one conditioning signal to two independently weighted ones:

$$\tilde\epsilon_\theta(z_t, c_I, c_T) = \epsilon_\theta(z_t,\varnothing,\varnothing) + s_I\big(\epsilon_\theta(z_t,c_I,\varnothing) - \epsilon_\theta(z_t,\varnothing,\varnothing)\big) + s_T\big(\epsilon_\theta(z_t,c_I,c_T) - \epsilon_\theta(z_t,c_I,\varnothing)\big)$$

Symbol	Meaning
$\tilde\epsilon_\theta$	the guided noise prediction used at sampling time
$z_t$	the noisy latent at diffusion step $t$
$c_I$	the input-image conditioning
$c_T$	the text-instruction conditioning
$s_I$	the image-guidance scale
$s_T$	the instruction-guidance scale
$\varnothing$	the null conditioning signal used during dropout training

What It Enabled

The three methods differ sharply in what they cost a user and what they cost a system builder:

	SDEdit, 2021	Prompt-to-Prompt, 2022	InstructPix2Pix, 2022
Edit signal	stroke, patch, or composite	edited text prompt (swap or re-weight)	plain-language instruction
Mask required	no (optional for compositing)	no	no
New training required	none — uses a pretrained model as-is	none — uses a pretrained model as-is	one fine-tuning run, on synthetic data
Real-photo editing	direct, via partial noising	needs DDIM inversion first (Chapter 9)	direct, feed-forward
Cost per edit at inference	one partial reverse pass	two parallel denoising runs (original and edited prompt)	one forward pass, no extra runs

A second table places the same three methods against the numbers each paper reported:

Metric	Value	Source
SDEdit realism preference vs. GAN baselines (human study)	98.09%	SDEdit paper
SDEdit stroke faithfulness, L2 vs. an encoder-based GAN-inversion baseline (human strokes)	32.55 vs. 53.76	SDEdit paper
InstructPix2Pix training set size	454,445 image/instruction/edit triples	InstructPix2Pix paper
InstructPix2Pix training compute	8×40GB A100 GPUs, 10,000 steps, 25.5 hours	InstructPix2Pix paper
InstructPix2Pix inference time	roughly 9 seconds per edit on one A100	InstructPix2Pix paper

Mask-based inpainting, the technique this chapter’s editing toolkit initially had to work around, did not disappear — it specialized. GLIDE (December 2021), covered in Chapter 12, shipped a text-conditioned inpainting mode that re-generates only the pixels inside a supplied mask at every denoising step, holding the rest of the image fixed; SDEdit’s own compositing mode does something similar without requiring the mask to be pixel-perfect, blending a pasted region into its surroundings by partially renoising across the seam. Inpainting and the mask-free methods this chapter centers on became two branches of the same family rather than competitors: a mask says exactly where to edit and lets the model focus its generative capacity there, while SDEdit, Prompt-to-Prompt, and InstructPix2Pix each found a different way to let the model infer where to edit on its own.

The lineage from here runs in two directions. Attention-level control generalized past prompt editing into spatial control more broadly — supplying a pose, a depth map, or an edge map as an explicit conditioning signal, the subject of Chapter 15. Personalization — teaching a model a specific subject or style rather than editing a specific image, via Chapter 16’s DreamBooth and LoRA — solved an adjacent but distinct problem: not “change this photo” but “generate new photos of this exact dog.” InstructPix2Pix’s distillation pattern — use a slow, training-free technique to manufacture data, then train a fast, feed-forward model that imitates it — recurs directly in the step-distillation methods of Chapter 18, which distill a slow sampler rather than a slow editing pipeline, but for the same underlying reason: pay an expensive setup cost once, at training time, so every later inference call is cheap. Instruction-based editing itself became a standard capability of later production systems, up to and including gpt-image-1’s native, ChatGPT-integrated editing (March 2025), covered in Chapter 20.

Key Papers

Paper	Authors	Year	Link	Why it matters
SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations	Meng et al.	2021	arXiv:2108.01073	First mask-free, training-free diffusion editing method; partial noising plus the existing reverse process is the whole mechanism
Prompt-to-Prompt Image Editing with Cross Attention Control	Hertz et al.	2022	arXiv:2208.01626	Shows cross-attention maps encode a text-to-image model’s implicit layout, and that injecting them controls edits without a mask
InstructPix2Pix: Learning to Follow Image Editing Instructions	Brooks, Holynski, and Efros	2022	arXiv:2211.09800	Distills Prompt-to-Prompt-generated synthetic data into a single feed-forward model that follows plain-language edit instructions

PM Lens