The GAN Golden Age

By late 2017 the adversarial objective from 2014 and the stabilization tricks that followed it had made GANs trainable without the coin-flip instability of the earlier years. What they had not yet produced was a face that survived a close look, or a resolution above a few hundred pixels on a side. Between October 2017 and December 2019, four papers out of two labs — NVIDIA and Google DeepMind — closed that gap and then overshot it, producing synthetic faces and ImageNet classes sharp enough that a website built to display them needed no caption to make its point.

Where We Were

The adversarial minimax game — a generator and a discriminator trained against each other until the generator’s output is indistinguishable from real data — was defined in 2014; its objective, its interpretation as a divergence minimization, and its characteristic failure mode of mode collapse are covered in Chapter 3. That objective trained, in principle, but not reliably: early GANs oscillated, collapsed onto a handful of outputs, or diverged outright. Chapter 4 covers the recipes that made training itself tractable — the convolutional architecture guidelines of DCGAN, the switch to the Wasserstein distance, and the gradient-penalty fix that made that distance enforceable in practice.

Those fixes solved stability, not scale. Through 2017, a GAN that trained reliably still tended to top out around 128×128 pixels; push the resolution higher and the same instabilities that gradient penalties had tamed at low resolution came back, because a single discriminator judging a high-resolution image all at once has an enormously harder job than one judging a thumbnail. Text-conditioned GANs of the same period, covered in Chapter 7, faced the identical ceiling from the opposite direction — StackGAN and AttnGAN generated recognizable scenes from captions, but at resolutions and fidelities well short of photographic.

The open question entering the second half of 2017 was not whether GANs could be made to train — that had been answered — but whether the same adversarial objective could be scaled to resolutions and fidelities that would hold up under scrutiny, and whether the smooth, disentangled latent space that generative models are supposed to offer could be delivered in practice rather than only in theory.

The Breakthrough

Progressive growing (ProGAN, October 2017). The first fix was procedural rather than architectural: instead of asking a generator and discriminator to learn a high-resolution mapping from scratch, start both networks at 4×4 pixels, let them converge, then add a new pair of layers that doubles the resolution, fading the new layers in gradually so the network never has to absorb a sudden jump in task difficulty (Karras et al., 2017). Repeating that doubling from 4×4 up to 1024×1024 let both networks master coarse structure — pose, overall shape — before ever being asked to render fine detail, and it made training an order of magnitude more stable at high resolution than training the full-resolution network directly. The paper shipped alongside a new dataset, CelebA-HQ, thirty thousand celebrity portraits cleaned and upscaled to 1024×1024, built specifically because no existing face dataset was clean enough at that resolution to train on.

Scale as a lever (BigGAN, September 2018). Where ProGAN scaled resolution through a training schedule, BigGAN scaled everything else: parameter count, batch size, and dataset breadth, applied to class-conditional generation across all 1,000 ImageNet categories rather than a single narrow domain like faces (Brock et al., 2018). The paper trained with a batch size of 2,048, distributed across 128 to 512 cores of a Google TPUv3 Pod with batch-normalization statistics computed across all devices rather than per-device — roughly an order of magnitude larger than typical GAN training at the time, and found that simply scaling batch size and channel width, without any other change, improved both the Inception Score and Fréchet Inception Distance substantially, at the cost of new training instabilities that had to be controlled with orthogonal regularization on the generator’s weights. That regularization had a second benefit: it made the generator well-behaved enough to support a “truncation trick,” in which sampling latent vectors from a narrower slice of the input distribution at inference time trades some output diversity for higher per-sample fidelity, a dial the paper’s users could turn continuously rather than a fixed choice baked into training.

Style as the interface (StyleGAN, December 2018). The same month CelebA-HQ was superseded by an even larger NVIDIA-built face dataset, Flickr-Faces-HQ (FFHQ), the field’s third structural idea arrived: stop feeding the latent code directly into the generator’s convolutional stack, and instead route it through an eight-layer mapping network into an intermediate latent space, then inject that intermediate code into every resolution level of the synthesis network via adaptive instance normalization (Karras et al., 2018). The generator also received an independent stream of per-pixel random noise, injected after each convolution, to explain the fine stochastic detail — individual strands of hair, skin pore placement — that has no business being tied to a person’s identity. The practical payoff was a latent space substantially more disentangled than a GAN’s raw input noise: style mixing, feeding different intermediate codes to different resolution levels, could swap a generated face’s coarse pose independently of its fine texture, because the two were no longer entangled in a single input vector.

Fixing the artifacts (StyleGAN2, December 2019). StyleGAN’s images carried tell-tale defects once anyone looked for them: water-droplet-like blob artifacts that tracked back to the instance normalization step in AdaIN, which could amplify a signal wherever the network needed to sneak information past it, and a separate “phase” artifact — details like teeth or eyes sticking to a fixed position on screen instead of following the face’s pose — that traced back to the progressive-growing schedule itself. StyleGAN2 replaced AdaIN’s explicit normalization with weight demodulation — folding the same effect into the convolution’s weights instead of the activations — and replaced progressive growing with a fixed-resolution architecture using skip connections and residual connections between resolutions, trained end to end from the start (Karras et al., 2019). A second regularizer, applied to the path length of the mapping from latent code to image, made that mapping more uniformly scaled and, as a side effect, made the generator dramatically easier to invert — to find the latent code that would have produced a given real photograph, a capability the original architecture had made unreasonably hard.

The Mathematics

Progressive growing’s fade-in is the simplest of the four ideas to state precisely: during a transition between resolutions, the network’s output is a weighted blend of the old, lower-resolution path and the new, higher-resolution path, with the weight moved linearly from one to the other over training.

$$x_{\text{out}} = (1 - \alpha) \cdot \mathrm{Upsample}(x_{\text{low}}) + \alpha \cdot x_{\text{high}}$$

Symbol	Meaning
$x_{\text{out}}$	blended output shown to the discriminator during a resolution transition
$x_{\text{low}}$	output of the already-trained, lower-resolution path, upsampled
$x_{\text{high}}$	output of the newly added, higher-resolution layer
$\alpha$	fade-in weight, linearly annealed from 0 to 1

StyleGAN’s core mechanism is adaptive instance normalization, which takes a feature map produced by a convolution and rescales it using statistics derived from the style code rather than from the feature map’s own mean and variance:

$$\mathrm{AdaIN}(x_i, y) = y_{s,i} \, \frac{x_i - \mu(x_i)}{\sigma(x_i)} + y_{b,i}$$

Symbol	Meaning
$x_i$	the $i$-th feature map after a convolution
$\mu(x_i)$, $\sigma(x_i)$	that feature map’s own per-channel mean and standard deviation
$y_{s,i}$	learned scale for channel $i$, derived from the style code
$y_{b,i}$	learned bias for channel $i$, derived from the style code

StyleGAN2 removes that explicit normalization step and folds an equivalent effect directly into the convolution’s weights, which is what “demodulation” means:

$$w'_{ijk} = \frac{s_i \, w_{ijk}}{\sqrt{\sum_{i,k} (s_i \, w_{ijk})^2 + \epsilon}}$$

Symbol	Meaning
$w_{ijk}$	original convolution weight, indexed by input channel, output channel, spatial position
$s_i$	style-derived scale for input channel $i$
$w'_{ijk}$	demodulated weight actually used in the convolution
$\epsilon$	small constant preventing division by zero

StyleGAN2’s second fix targets a different problem: making equal steps in latent space correspond to roughly equal-sized changes in the image, which the path length regularizer enforces by penalizing deviation from a running-average step size.

$$L_{\mathrm{pl}} = \mathbb{E}_{\mathbf{w}, \, \mathbf{y}} \Big[ \big( \lVert \mathbf{J}_{\mathbf{w}}^{\top} \mathbf{y} \rVert_2 - a \big)^2 \Big]$$

Symbol	Meaning
$\mathbf{w}$	sampled intermediate latent code
$\mathbf{y}$	random probe image drawn from a fixed noise distribution
$\mathbf{J}_{\mathbf{w}}$	Jacobian of the generator’s output with respect to $\mathbf{w}$
$a$	running average of the measured step size, updated during training

What It Enabled

The four papers moved unconditional face generation and class-conditional ImageNet generation from “recognizable but flawed” to “quantitatively state-of-the-art for the GAN family,” measured by the Fréchet Inception Distance whose construction and failure modes are covered in Chapter 21.

Model	Year	Dataset	Resolution	Quality metric
ProGAN architecture, FFHQ baseline	2018	FFHQ	1024×1024	FID 8.04
StyleGAN	2018	FFHQ	1024×1024	FID 4.40
StyleGAN2 (config F)	2019	FFHQ	1024×1024	FID 2.84
BigGAN	2018	ImageNet 128px	128×128	IS 166.5, FID 7.4

FFHQ postdates the original 2017 ProGAN paper, which trained on CelebA-HQ; the FID 8.04 baseline above is the StyleGAN authors’ own reimplementation of the ProGAN architecture, retrained on FFHQ for a like-for-like comparison (Karras et al., 2018).

Two axes of progress ran in parallel and stayed separate for years: resolution and disentanglement, pushed by the NVIDIA line on faces, and scale and class-conditional breadth, pushed by BigGAN on ImageNet. Neither paper borrowed the other’s central idea — BigGAN’s self-attention layers, adapted from the alignment-era attention mechanism covered in language-models Chapter 4, addressed long-range spatial coherence, not style control; StyleGAN’s mapping network addressed disentanglement, not scale.

Capability	Before this chapter (2017)	After (2019)
Practical face resolution	128–256px before instability returns	1024×1024, trained end to end
Latent space structure	Raw input noise, entangled	Intermediate $w$ space, style-mixable
Class-conditional ImageNet fidelity	Blurry at scale, low diversity control	IS 166.5, continuous fidelity–diversity dial
Characteristic artifacts	Mode collapse, checkerboarding	Diagnosed to source (AdaIN, progressive growing) and removed
Latent-to-image inversion	Effectively impractical	Tractable, enabled by path length regularization

The latent-space control this generation of GANs delivered — style mixing, a smoothly interpolable $w$ space, truncation as a fidelity dial — is the direct ancestor of the fine-tuning and personalization techniques covered in Chapter 16, even though the underlying generator family changed. Within two years, diffusion-based image models matched and then exceeded GAN sample quality on the same benchmarks, a transition covered in Chapter 10; GANs did not disappear, but they stopped being the default architecture for new state-of-the-art image models after roughly 2021.

Key Papers

Paper	Authors	Year	Link	Why it matters
Progressive Growing of GANs for Improved Quality, Stability, and Variation	Karras et al.	2017	arXiv:1710.10196	Introduced the low-to-high resolution training schedule that first made 1024×1024 GAN training stable, plus the CelebA-HQ dataset
Large Scale GAN Training for High Fidelity Natural Image Synthesis	Brock et al.	2018	arXiv:1809.11096	Showed that scaling batch size and parameters, with orthogonal regularization for stability, set new class-conditional ImageNet quality records and introduced the truncation trick
A Style-Based Generator Architecture for Generative Adversarial Networks	Karras et al.	2018	arXiv:1812.04948	Replaced direct latent injection with a mapping network and AdaIN-based style modulation, producing a disentangled, style-mixable latent space and the FFHQ dataset
Analyzing and Improving the Image Quality of StyleGAN	Karras et al.	2019	arXiv:1912.04958	Diagnosed StyleGAN’s blob artifacts to AdaIN and replaced it with weight demodulation; added path length regularization, improving both quality and latent-space invertibility

PM Lens