First Words to Pictures

It is December 2016. Text-to-image generation already exists in a narrow sense — encode a sentence, concatenate the resulting vector with noise, feed it to a conditional GAN, and a small, blurry photograph comes out the other end. But the images are 64×64 pixels, the beaks are the wrong shape, and there is no way to ask the model which word it was looking at when it drew a given patch of feathers. Over the next eleven months, two papers close that gap from opposite directions: one by stacking generators to reach higher resolution, the other by teaching the network to look at individual words.

Where We Were

The conditional GAN (see image-generation/03) had already shown, in November 2014, that a generator and discriminator could be conditioned on a side variable — originally a class label — rather than sampling from an unconditioned prior. Extending that variable from a discrete class to a text embedding was a natural next step, and by 2016 several groups had done exactly that: encode a caption into a single global vector, concatenate it with noise, and train the usual adversarial minimax game against that combined input.

The results were recognizable but limited in a specific way. A sentence like “this bird is red and brown in color, with a stubby beak” would produce a red-and-brown blob roughly bird-shaped, but the fine structure — the eye ring, the exact ratio of beak length to head size, the pattern on the wing — was usually wrong or missing. Two separate problems were tangled together. First, GANs of the period were unstable at resolutions above roughly 64×64: adding upsampling layers to reach 256×256 tended to produce nonsensical output rather than a sharper image, because the training distribution and the model’s distribution have less and less overlap as pixel-space dimensionality grows. Second, compressing an entire sentence into one fixed-length vector threw away exactly the information needed to place details in the right sub-region of the image — a global vector says “red and brown, stubby beak” but not where the beak is relative to the eye.

Part of the difficulty was architectural inertia. The text-conditioning approach these early models inherited was built for class labels, where a single one-hot vector is genuinely all the information there is. Swapping that one-hot vector for a pooled sentence embedding preserved the architecture but not the assumption underneath it — a caption is not one atomic fact, it is a small structured description with several facts bound to several places in the image, and pooling collapses that structure before the generator ever sees it. A generator conditioned on a single vector has no mechanism to ask “which part of this sentence is relevant to the patch I am drawing right now”; it can only ask “given everything the sentence says, on average, what should this whole image look like.”

Both problems needed solving before text-to-image synthesis could produce anything a person would call photorealistic, and the two 2016–2017 papers this chapter covers solved them one at a time.

A single pooled sentence vector could not fix resolution and word-level alignment at once, so the two fixes arrived a year apart. PNG ↓

The Breakthrough

StackGAN (Zhang et al., 2016) attacked the resolution problem by refusing to generate a high-resolution image in one shot. Instead it decomposes the task the way a sketch artist works: a Stage-I generator reads the text embedding and a noise vector and draws a rough 64×64 image — approximate shape, approximate color, roughly placed background. A Stage-II generator then reads that 64×64 result and the text embedding again, and produces a 256×256 image that corrects Stage-I’s defects and adds detail the first pass omitted. Because the Stage-II generator’s input distribution (a rough but roughly-aligned image) overlaps far better with the real high-resolution data distribution than raw noise would, the notoriously unstable jump straight to 256×256 became tractable.

The paper’s second contribution addressed a narrower but real problem: a training set of a few thousand image–caption pairs leaves the text-conditioning space badly under-sampled, so nearby points in that space, corresponding to slightly different phrasings of the same idea, may map to visually distinct outputs with no smooth interpolation between them. StackGAN’s fix, Conditioning Augmentation, samples the actual conditioning vector from a Gaussian distribution whose mean and variance are themselves learned functions of the text embedding, rather than using a fixed deterministic embedding. This is the same reparameterization idea that the variational autoencoder chapter introduces for a different purpose; here it is applied specifically to smooth the text-conditioning manifold and encourage diversity in outputs from a limited number of captions per image.

The training regimen behind these numbers was modest by later standards: both stages used the Adam optimizer with an initial learning rate of 0.0002, decayed by half every 100 epochs, batch size 64, with Stage-I trained for 600 epochs before Stage-II trained for another 600 on top of it. Nothing about the recipe demanded unusual scale — the improvement came from decomposing the problem, not from more compute.

The rough 64px draft is what makes the jump to 256px trainable. Conditioning Augmentation's sampled vector is reused, not recomputed, at both stages. PNG ↓

AttnGAN (Xu et al., 2017), published eleven months later, attacked the second problem — the information bottleneck of a single global sentence vector — directly. Instead of conditioning every stage on one fixed vector, AttnGAN keeps the individual word embeddings around and lets each stage’s generator query them with an attention mechanism (the general softmax-alignment idea is introduced in language-models/04): for every sub-region of the image currently being drawn, the model computes a weighted combination of word vectors, weighted by how relevant each word is to that specific sub-region. A generator drawing the region around a bird’s throat can therefore attend most strongly to the words “yellow” and “throat” in the caption, while a generator drawing the wing attends to “brown” and “wings” — something a single pooled sentence vector cannot represent.

The paper’s second component, the Deep Attentional Multimodal Similarity Model (DAMSM), supplies an additional training signal by measuring how well a generated image matches its caption at the word level, not just the sentence level. A pretrained text encoder and image encoder map words and image sub-regions into a shared space; an attention-weighted similarity score between them becomes an extra loss term that pushes the generator toward images whose fine details are verifiably consistent with individual words in the caption, not merely images that look plausible on average.

The generator asks which words matter per pixel while painting; the DAMSM loss then checks, after the fact, whether the result actually matches those words. PNG ↓

The Mathematics

StackGAN’s Conditioning Augmentation replaces a fixed text-conditioning vector with a sampled one:

\hat{c}_0 = \mu_0(\varphi_t) + \sigma_0(\varphi_t) \odot \epsilon, \qquad \epsilon \sim \mathcal{N}(0, I)

To keep that Gaussian close to a standard normal distribution — and stop the network from cheating by making $\sigma_0$ collapse to zero and turning the sampling back into a fixed vector — StackGAN adds a regularization term to the generator’s loss:

\mathcal{L}_{CA} = D_{KL}\Big(\mathcal{N}\big(\mu_0(\varphi_t), \Sigma_0(\varphi_t)\big) \,\|\, \mathcal{N}(0, I)\Big)

AttnGAN’s attention model computes, for every sub-region $j$ of the image currently being generated, a word-context vector built from a weighted sum over all word features:

\beta_{j,i} = \frac{\exp\!\left(h_j^{\top} e_i'\right)}{\displaystyle\sum_{k=0}^{T-1} \exp\!\left(h_j^{\top} e_k'\right)}, \qquad c_j = \sum_{i=0}^{T-1} \beta_{j,i}\, e_i'

Symbol	Meaning
$h_j$	Generator’s hidden feature (query) for image sub-region $j$
$e_i'$	Word feature $i$ , projected into the image feature space
$T$	Number of words in the caption
$\beta_{j,i}$	Attention weight: how much sub-region $j$ attends to word $i$
$c_j$	Word-context vector for sub-region $j$

The DAMSM turns that same kind of attention-weighted similarity into a training loss by asking, for a batch of $M$ image–caption pairs, how confidently the correct caption can be picked out for each image:

P(D_i \mid Q_i) = \frac{\exp\!\big(\gamma_3\, R(Q_i, D_i)\big)}{\displaystyle\sum_{j=1}^{M} \exp\!\big(\gamma_3\, R(Q_i, D_j)\big)}, \qquad \mathcal{L}_1^{w} = -\sum_{i=1}^{M} \log P(D_i \mid Q_i)

$R(Q_i, D_i)$ is an attention-weighted similarity score between image $Q_i$ and caption $D_i$ , built from the same kind of word-region attention as above. $P(D_i \mid Q_i)$ treats this as a classification problem: out of $M$ captions in the batch, how much posterior probability lands on the one that actually matches image $Q_i$ , versus the $M-1$ mismatched ones. $\gamma_3$ is a smoothing factor (set to 10 in the paper) that sharpens or softens that distribution. $\mathcal{L}_1^{w}$ is simply the negative log of that probability, minimized when the correct caption dominates the posterior. The paper mirrors this loss in the other direction — retrieving the matching image given a caption — and at the sentence level as well as the word level, summing all four terms into the full DAMSM loss added to the generator’s adversarial loss with weight $\lambda$ .

Symbol	Meaning
$Q_i, D_i$	The $i$ -th image and its matching caption in a training batch
$R(Q_i, D_j)$	Attention-weighted image–text similarity score
$\gamma_3$	Smoothing factor sharpening the posterior distribution
$M$	Number of image–caption pairs in the batch
$P(D_i \mid Q_i)$	Posterior probability that caption $D_i$ matches image $Q_i$

What It Enabled

Both papers reported their gains on the same benchmarks — the CUB birds dataset (200 species, 11,788 images) and the Oxford-102 flowers dataset (8,189 images across 102 categories), each with roughly ten captions per image, plus the larger and more cluttered MS COCO dataset — using the Inception Score as the primary automatic metric.

Model	Conditioning	Max resolution	Inception score, CUB
Global-sentence conditional GAN (GAN-INT-CLS baseline, as reported in the StackGAN paper)	single pooled sentence vector	64×64	2.88 ± 0.04
StackGAN	sentence vector, Gaussian conditioning augmentation, two-stage refinement	256×256	3.70 ± 0.04
AttnGAN (λ = 5, two attention stages)	per-word, per-region attention plus DAMSM loss	256×256	4.36 ± 0.03

StackGAN’s own comparison reported a 28.47% inception-score improvement over the GAN-INT-CLS baseline on CUB and a 20.30% improvement on Oxford-102; average human ranking of generated images improved on every dataset tested. AttnGAN’s abstract reported boosting the best previously published inception score by 14.14% on CUB and by 170.25% on the more cluttered COCO dataset — a far larger jump on the harder, more compositional dataset, consistent with the idea that word-level attention matters most when a caption describes many distinct objects and relationships rather than one bird.

AttnGAN’s own ablation study isolated exactly how much of that gain came from the DAMSM loss specifically, using R-precision (the fraction of generated images correctly matched back to their caption via retrieval) as the diagnostic:

Variant	Inception score (CUB)	R-precision (CUB)
Single attention stage, no DAMSM loss	3.98 ± 0.04	10.37% ± 5.88
Single attention stage, DAMSM weight λ = 1	4.35 ± 0.05	34.96% ± 4.02
Single attention stage, DAMSM weight λ = 5	4.35 ± 0.04	58.65% ± 5.41
Two attention stages, DAMSM weight λ = 5	4.36 ± 0.03	67.82% ± 4.43

R-precision jumping from roughly 10% to nearly 68% on the same architecture, holding everything but the presence and weight of the DAMSM loss constant, is direct evidence that a fine-grained, word-level matching signal — not just a sharper generator — is what makes generated images verifiably consistent with what a caption actually says. R-precision itself is a retrieval metric borrowed from information retrieval, not a bespoke image-quality score: generate an image, use it to query a ranked list of candidate captions (one true match plus ninety-nine mismatched decoys), and check whether the true caption lands in the top result. That a generator’s own images can be used to retrieve their captions back out is a more direct test of semantic fidelity than the Inception Score, which only measures whether generated images look like plausible members of their class — a distinction the field’s evaluation practices, covered in evaluation, provenance, and safety, would keep refining well after both these papers.

The word-level attention idea did not stay confined to GANs for long. When CLIP arrived in 2021, it replaced AttnGAN’s small, jointly-trained DAMSM encoders with a single contrastive image-text model trained on hundreds of millions of pairs — the same instinct (align words and image regions in a shared space) at a scale two orders of magnitude larger. And when the field moved to diffusion, the cross-attention layers that let a U-Net condition on text in latent diffusion and Stable Diffusion are a direct architectural descendant of AttnGAN’s per-region word attention: a generator, at every spatial location, still asks “which words matter here?” — it just asks the question inside a denoising network instead of a GAN generator. GAN-based text-to-image itself was largely superseded within a few years by the autoregressive and diffusion approaches covered in DALL-E and GLIDE, once those methods proved easier to scale and to train stably at high resolution.

Key Papers

Paper	Authors	Year	Link	Why it matters
StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networks	Zhang et al.	2016	arXiv:1612.03242	Two-stage generator plus Conditioning Augmentation; first GAN text-to-image pipeline to reach 256×256 with recognizable detail
AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks	Xu et al.	2017	arXiv:1711.10485	Word-region attention plus the DAMSM matching loss; first demonstration that a GAN can be shown attending to individual words per image region

PM Lens

When a designer complains that a Midjourney or a similar text-to-image tool “ignored half the prompt,” the diagnostic question underneath that complaint — which word was the model actually looking at when it drew this part of the image — is exactly what AttnGAN made visualizable for the first time, back when its authors published attention maps showing the top-attended words for each generated sub-region. That same instinct now shows up in product decisions far removed from birds and flowers: when I scope an image-generation feature, “does it attend to the whole prompt or just the dominant noun” is a real acceptance-test question, and it traces directly back to the DAMSM ablation table above, where adding a word-level matching loss took retrieval accuracy from about 10% to nearly 70% on the same generator. Resolution scaling and semantic fidelity are separable failure modes, and this pair of 2016–2017 papers is the cleanest place to see why teams need two different fixes for them, not one.

Search