Latent Diffusion and Stable Diffusion

It is December 2021. Diffusion models have spent seven months proving they can beat generative adversarial networks on image fidelity, but that win came at pixel-space prices: a guided diffusion model trains and samples every one of up to a thousand steps across the full resolution of the image, a compute bill only a handful of labs can pay. A team at LMU Munich’s Computer Vision and Learning group — several of them the same authors who, a year earlier, had taught a generative adversarial network to compress images into a small discrete codebook — publishes a paper that puts the two ideas together: stop running the diffusion process on pixels, and run it instead inside a much smaller, pre-learned compressed space.

Where We Were

By late 2021, diffusion had won the argument on quality. Denoising Diffusion Probabilistic Models, or DDPM (Diffusion Foundations), had matched GAN-level fidelity on small images in June 2020 with a training objective that needed no adversarial network at all, and in May 2021 Diffusion Models Beat GANs on Image Synthesis, or ADM, pushed that lead onto ImageNet-scale, high-resolution generation using classifier guidance (Guidance). Both results shared the same structural cost: the forward-noising and reverse-denoising process ran directly on pixels, at the image’s full spatial resolution, and it ran that way at every single one of the model’s diffusion steps. A U-Net large enough to denoise a 256×256 image competitively had to do that work up to a thousand times to produce one sample, and training a model of that size on that objective was, by the field’s own account, a project measured in hundreds of GPU-days on institutional hardware — out of reach for anyone without a research lab’s compute budget.

A second, separate lineage had spent the same years attacking a different problem: not how to generate an image, but how to represent one compactly enough that a much smaller model could work with it at all. Neural Discrete Representation Learning, or VQ-VAE, had shown in November 2017 that an image could be compressed into a small spatial grid of discrete codes drawn from a learned vocabulary, and that an autoregressive prior trained over that short grid needed far fewer sequential steps than one trained over raw pixels (Pixels One at a Time). In December 2020, Esser, Rombach, and Ommer refined that recipe substantially. “Taming Transformers for High-Resolution Image Synthesis” (Esser et al., 2020) — the paper that named the result VQGAN — kept VQ-VAE’s codebook mechanism but replaced its plain pixel-reconstruction loss with a perceptual loss and a patch-based adversarial objective, and replaced its PixelCNN prior with a GPT-2-style transformer. The result was a compressed latent space whose decoded images looked sharp and detailed rather than smoothed-over, at compression factors aggressive enough — a downsampling factor of 16 was typical for face datasets, with the paper exploring a range from 2 to 64 depending on the target resolution — that the transformer prior, run in a sliding-window fashion across overlapping patches, could synthesize images in the megapixel range.

Compression and generation ran on separate clocks for four years. Nobody asked whether VQGAN's latent space could host a diffusion model until this paper did. PNG ↓

By the end of 2021, both lineages were mature on their own terms, but nobody had asked the question that connects them: VQGAN’s compressed latent space had been built to host an autoregressive transformer — could it host a diffusion model instead, and inherit diffusion’s training stability and sample diversity without inheriting pixel-space diffusion’s compute bill? Text-guided diffusion was also just arriving that same month with GLIDE (DALL-E and GLIDE), but GLIDE ran its guided diffusion process directly in pixel space, at the same full-resolution cost ADM had already established. The gap between “diffusion is the better generative model” and “diffusion is a generative model most teams can actually afford to train and run” was still wide open.

The Breakthrough

Rombach, Blattmann, Lorenz, Esser, and Ommer’s “High-Resolution Image Synthesis with Latent Diffusion Models” (Rombach et al., 2021) answered that question by splitting image generation into two stages trained separately. The first stage is a perceptual-compression autoencoder, trained once, using essentially the same recipe VQGAN had introduced a year earlier: an encoder $\mathcal{E}$ and decoder $D_{\text{dec}}$ trained jointly with a perceptual loss and a patch-based adversarial loss, so that the compressed representation discards only the pixel-level detail a human viewer would not miss, not the semantic content of the image. The second stage is a diffusion model — the same forward-noising, learned-reverse-process machinery taught in full in the DDPM chapter — but trained entirely inside the small spatial grid the first stage produces, never touching a raw pixel.

The paper reports an ablation across downsampling factors $f \in \{1, 2, 4, 8, 16, 32\}$ and finds $f=4$ and $f=8$ the best trade-off: aggressive enough compression to make the diffusion U-Net’s job cheap at every one of its sampling steps, without compressing away enough detail to visibly hurt reconstruction quality. For a 512×512 image, an $f=8$ autoencoder produces a $64 \times 64$ latent grid — a $64\times$ reduction in the number of spatial positions the diffusion U-Net’s convolutions and attention layers have to process, at every one of the (up to) thousand steps a sampler runs, not just once. Because the autoencoder is trained once and reused for every downstream diffusion model, that compute saving compounds: the same latent space can host an unconditional model, a class-conditional model, or a text-conditional model, and each one pays the pixel-space cost only twice, at encode and decode, rather than at every diffusion step.

That reuse depended on a second contribution: a general mechanism for injecting an arbitrary conditioning signal into the latent U-Net via cross-attention, rather than building a separate architecture for each conditioning type. A domain-specific encoder — a text transformer, a semantic-layout encoder, anything appropriate to the modality — maps the conditioning input into an intermediate representation, and the U-Net’s own intermediate features attend to that representation using the same scaled dot-product attention mechanism taught in full in The Transformer; what changes is only where the keys and values come from. That single mechanism is what let one architecture serve class-conditional generation, layout-to-image, super-resolution, inpainting, and text-to-image without a bespoke redesign for each task.

Only the encoder and decoder ever touch raw pixels, and each pays that cost once. Every one of the sampler's up-to-a-thousand steps runs on the small grid instead. PNG ↓

The paper’s own text-to-image demonstration, trained on LAION-400M with a from-scratch BERT-tokenized transformer as the domain encoder, was a research artifact rather than a product. The product arrived eight months later. On August 22, 2022, Stability AI, in collaboration with the original CompVis group, RunwayML, and LAION, released Stable Diffusion’s weights publicly under the Creative ML OpenRAIL-M license — a permissive license allowing commercial and non-commercial use — following a beta that had already run through more than 10,000 Discord testers and a researcher-only release of the preceding checkpoint. Stable Diffusion kept the LDM recipe’s $f=8$ downsampling and cross-attention conditioning, but swapped the paper’s from-scratch text transformer for a frozen CLIP ViT-L/14 text encoder (CLIP: Bridging Text and Image), giving the model a ready-made, already-aligned text-image embedding space rather than one it had to learn from scratch. The released v1.4 checkpoint paired an 860M-parameter U-Net with that 123M-parameter text encoder and an autoencoder decoder of roughly 84M parameters, and Stability AI’s own announcement stated the model would run on consumer GPU hardware using about 6.9 GB of VRAM — a machine an individual developer could own, not a datacenter allocation.

Eleven months later, the same core team returned with a scaled-up successor. “SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis” (Podell et al., 2023) kept the latent-diffusion architecture but roughly tripled the U-Net to 2.6B parameters, added a second text encoder (OpenCLIP ViT-bigG alongside CLIP ViT-L, concatenating their outputs into a 2048-dimensional conditioning context against Stable Diffusion 1.x’s 768), introduced explicit conditioning on image size and crop parameters so the model could be trained on images of varying aspect ratio without wasted padding, and added an optional refiner model that applies a short noise-and-denoise pass to a base model’s output latents to sharpen fine detail in faces and backgrounds.

The Mathematics

The autoencoder’s training objective builds directly on VQ-VAE’s codebook mechanism — the quantization lookup and straight-through estimator taught in full in Pixels One at a Time — but adds the two terms that make VQGAN’s reconstructions look sharp rather than smoothed:

\mathcal{L}_{\text{VQGAN}} = \mathcal{L}_{\text{perc}}(x, \hat{x}) \;+\; \mathcal{L}_{\text{VQ}} \;+\; \lambda \Big[\log D(x) + \log\big(1 - D(\hat{x})\big)\Big]

$x$ is the original image and $\hat x = D_{\text{dec}}(z_q)$ its reconstruction from the quantized latent. $\mathcal{L}_{\text{perc}}(x, \hat{x})$ compares $x$ and $\hat{x}$ not pixel-by-pixel but in the feature space of a pretrained image classifier — penalizing perceptually meaningful differences (missing texture, blurred edges) rather than exact pixel values, which is why VQGAN reconstructions look sharp where a plain L2 loss would look averaged and soft. $\mathcal{L}_{\text{VQ}}$ is the codebook and commitment loss already taught in the VQ-VAE chapter, unchanged. The bracketed term is a patch-based discriminator’s ordinary adversarial loss: $D$ is trained to tell real patches of $x$ from reconstructed patches of $\hat{x}$ , and the encoder-decoder pair is trained to fool it, exactly as in a GAN (Generative Adversarial Networks), except the discriminator judges local patches rather than the whole image. $\lambda$ weights how much the adversarial signal contributes relative to the other two terms. What this buys the rest of the chapter: a compressed latent space whose decoded images are visually convincing at high compression factors — the exact property a diffusion model needs from the space it is about to be trained inside.

Symbol	Meaning
$x$ , $\hat{x}$	the original image and its reconstruction from the quantized latent
$\mathcal{L}_{\text{perc}}$	perceptual loss, compared in a pretrained classifier’s feature space
$\mathcal{L}_{\text{VQ}}$	codebook and commitment loss from VQ-VAE, Chapter 5
$D$	patch-based discriminator, real versus reconstructed
$\lambda$	weight on the adversarial term

With the autoencoder trained and frozen, the diffusion model’s training objective is the same simple noise-prediction regression taught in full in Diffusion Foundations, with one substitution: the data the diffusion process noises and denoises is the encoder’s latent, not the image itself.

L_{\text{LDM}} := \mathbb{E}_{z=\mathcal{E}(x),\, t,\, \epsilon \sim \mathcal{N}(0,\mathbf{I})}\Big[\big\lVert \epsilon - \epsilon_\theta(z_t, t) \big\rVert_2^2\Big]

$\mathcal{E}(x)$ is the frozen encoder’s output for image $x$ — a small grid of numbers, not pixels. $z_t$ is that latent after $t$ steps of the same closed-form forward-noising process DDPM uses, just applied to $z$ instead of $x$ . $\epsilon_\theta(z_t, t)$ is the U-Net’s prediction of the noise that was added, and the loss penalizes the squared distance between that prediction and the true noise $\epsilon$ . Every symbol on the right is identical in kind to DDPM’s loss; the only change is that the U-Net’s convolutions and attention layers now operate on a grid that is $f^2$ times smaller in spatial extent than the original image, at every one of the sampler’s steps, which is the entire source of latent diffusion’s compute saving over pixel-space diffusion.

Symbol	Meaning
$\mathcal{E}(x)$ , $z$	the frozen perceptual encoder and its output latent for image $x$
$z_t$	the latent after $t$ steps of forward noising
$\epsilon_\theta(z_t, t)$	the U-Net’s predicted noise, given the noisy latent and timestep
$\theta$	the trainable parameters of the latent-space U-Net

Conditioning is added through cross-attention layers spliced into the U-Net at several of its internal resolutions:

\mathrm{Attention}(Q,K,V) = \mathrm{softmax}\!\left(\frac{QK^{\top}}{\sqrt{d}}\right)V,\qquad Q = W_Q^{(i)}\varphi_i(z_t),\ \ K = W_K^{(i)}\tau_\theta(y),\ \ V = W_V^{(i)}\tau_\theta(y)

The attention operation itself — queries, keys, values, scaled dot-product, softmax — is the exact mechanism taught in The Transformer; nothing about the operation changes here. What changes is where each piece comes from. $\varphi_i(z_t)$ is the U-Net’s own intermediate feature map at layer $i$ , projected into queries $Q$ by a learned matrix $W_Q^{(i)}$ . $y$ is the conditioning input — a text prompt, a semantic layout, anything the task calls for — and $\tau_\theta(y)$ is a separate, domain-specific encoder’s embedding of it (a frozen CLIP text encoder, in Stable Diffusion’s case), projected into keys $K$ and values $V$ by their own learned matrices. Because the keys and values come from a different sequence than the queries, this is cross-attention rather than self-attention: every spatial location in the noisy latent gets to look up and pull in whatever part of the conditioning signal is relevant to it. What this buys the architecture: swapping $\tau_\theta$ for a different domain-specific encoder is enough to repoint the same U-Net at a new conditioning modality, with no other architectural change required.

Symbol	Meaning
$Q, K, V$	query, key, and value matrices for one cross-attention layer
$\varphi_i(z_t)$	the U-Net’s intermediate feature map at layer $i$
$\tau_\theta(y)$	a domain-specific encoder’s embedding of the conditioning input $y$
$W_Q^{(i)}, W_K^{(i)}, W_V^{(i)}$	learned projection matrices at layer $i$
$d$	the dimensionality of the key vectors, used to scale the dot product

Perceptual compression is, underneath the diffusion framing, a direct answer to the memory wall (The Memory Wall): every convolution and attention layer in the U-Net has to read and write a feature map proportional in size to its input, and running that at full pixel resolution for up to a thousand sequential steps is precisely the kind of memory-bandwidth-bound workload that stalls a chip’s compute units waiting on data movement. Shrinking the spatial grid by $f^2$ before the diffusion process ever starts is what let Stable Diffusion’s 860M-parameter U-Net and its text encoder fit comfortably inside roughly 6.9 GB of VRAM on a single consumer GPU, where pixel-space guided diffusion of comparable fidelity needed institutional, multi-accelerator hardware to train and run at all.

What It Enabled

Latent diffusion’s immediate effect was to collapse the gap between “state-of-the-art image fidelity” and “a model an individual can train and run,” a gap the field had been widening since ADM.

	Pixel-space guided diffusion, ADM, 2021	Latent diffusion, LDM, 2021	Stable Diffusion, public release, 2022
Where diffusion runs	full-resolution pixel grid, every step	$f{=}4$ / $f{=}8$ compressed latent, every step	$f{=}8$ latent, $64\times64\times4$ for a $512\times512$ image
ImageNet 256 class-conditional FID	4.59, with classifier guidance	3.60, LDM-4-G, with classifier-free guidance (Guidance)	not the target task; general text-to-image model
Denoising network size	large pixel-space U-Net, institutional-scale training	400M-parameter latent U-Net	860M-parameter latent U-Net, plus a 123M-parameter frozen text encoder
Weights publicly downloadable	no	research code, not the trained weights	yes, Creative ML OpenRAIL-M license
Typical hardware to run inference	multi-accelerator, datacenter-class	a single high-end GPU	a single consumer GPU, about 6.9 GB VRAM

The CelebA-HQ result told the same story from the unconditional side: an LDM-4 model reported a new state-of-the-art FID of 5.11 on that benchmark at the time, ahead of prior likelihood-based models and GANs alike, using a fraction of the training compute pixel-space diffusion required. And the framework’s generality showed up immediately in its own paper: the same architecture, retrained on LAION-400M with a 1.45B-parameter text-conditioning branch, reached an FID of 12.63 on MS-COCO — evidence that one perceptual-compression-plus-cross-attention recipe, not a bespoke architecture per task, was enough to cover class-conditional generation, unconditional faces, and free-form text-to-image together.

Stable Diffusion’s release changed who could act on that recipe. Where OpenAI’s DALL-E 2 API, opened to developers that November, billed $0.02 per 1024×1024 image generated through a hosted endpoint, Stable Diffusion’s open weights let any team with a single GPU generate, fine-tune, or embed the model into a product at the marginal cost of their own electricity — a difference in kind, not just price, for anyone building a product around image generation rather than calling one.

Nearly every later architecture in this book's second half reuses this frozen checkpoint rather than training a new one from scratch. PNG ↓

Every one of those downstream lines runs through the same open checkpoint. ControlNet’s spatial conditioning and the broader adapter ecosystem (Control) inject new conditioning signals into Stable Diffusion’s frozen U-Net through the exact cross-attention pathway this chapter derived. Textual Inversion and DreamBooth teach the same frozen model new subjects and styles from a handful of images (Personalization and LoRA). Consistency models, LCMs, and adversarial distillation cut Stable Diffusion’s sampling steps from the dozens down toward one (Few-Step Generation). SDXL’s larger backbone is itself the launch point for the rectified-flow, transformer-backboned models that follow it (Flow Matching and Transformer Backbones). And bolting a temporal dimension onto this same latent U-Net, rather than building a video model from scratch, is the foundation under Stable Video Diffusion and the rest of the video-generation lineage that opens with it (Latent Video Diffusion). A comparison of the two Stable Diffusion generations released under this architecture shows how much of that later growth came from scaling the same recipe rather than replacing it:

	Stable Diffusion 1.4/1.5, Aug 2022	Stable Diffusion 2.0/2.1, late 2022	SDXL, Jul 2023
U-Net parameters	860M	865M	2.6B
Text encoder(s)	CLIP ViT-L/14	OpenCLIP ViT-H/14, trained from scratch	CLIP ViT-L/14 + OpenCLIP ViT-bigG
Cross-attention context dimension	768	1024	2048
Native resolution	$512 \times 512$	$768 \times 768$	multi-aspect, around $1024 \times 1024$
Human-preference win rate vs. SD 1.5	—	—	48.44%, with the refiner stage

Key Papers

Paper	Authors	Year	Link	Why it matters
Taming Transformers for High-Resolution Image Synthesis	Esser, Rombach, Ommer	2020	arXiv:2012.09841	VQGAN’s perceptual-plus-adversarial autoencoder and transformer prior supplied the exact compression recipe latent diffusion reuses
High-Resolution Image Synthesis with Latent Diffusion Models	Rombach, Blattmann, Lorenz, Esser, Ommer	2021	arXiv:2112.10752	Moved the diffusion process into a frozen perceptual-compression latent space and introduced cross-attention conditioning; the direct architecture behind Stable Diffusion
SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis	Podell et al.	2023	arXiv:2307.01952	Scaled the same latent-diffusion recipe to a 2.6B-parameter U-Net with dual text encoders and a refinement stage

PM Lens

The number I carry out of this chapter isn’t a parameter count, it’s a license. Stable Diffusion’s August 2022 release under Creative ML OpenRAIL-M meant a product team could fine-tune and self-host a state-of-the-art text-to-image model at the cost of one GPU, while a comparable call to DALL-E 2’s API that same season billed $0.02 per image with no weights to touch. I’ve made both calls: pay per call for a low-volume feature where you don’t want to own an inference stack, or self-host when volume or customization makes owning the weights cheaper and more controllable. What I watch for now, whenever a vendor pitches a new latent-diffusion-based product, is which of the three levers this chapter names they actually pulled — a smaller downsampling factor, a bigger U-Net, or a better conditioning encoder — because those three levers, not marketing language, are still what separates SD 1.5 from SDXL from whatever ships next.

Search