Evaluation, Provenance, and Safety

It is June 2017. Generative adversarial networks have multiplied faster than anyone can agree on which ones are actually good, and the field’s one shared number — a classifier’s confidence score — cannot tell a sharp, diverse fake from a blurry, mode-collapsed one. Fixing that measurement problem turns out to be the easy part of a longer story: over the following seven years, three harder questions surface behind it — whether an image matches what a person actually wanted, whether a model has quietly memorized a training photograph it was never supposed to reproduce, and whether a synthetic image can be marked at the moment it is made so a viewer downstream can tell it apart from a real one.

Where We Were

Early GAN papers were judged largely by eye. A results section showed a grid of the model’s best-looking samples, sometimes alongside a small “visual Turing test” in which human raters tried to spot the fakes, and readers were left to trust that the grid was representative rather than cherry-picked. That worked poorly as a way to compare architectures, because two labs’ grids could not be ranked against each other without either lab lending the other its trained model.

The first fix was the Inception Score, introduced by Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen alongside a set of GAN training tricks in June 2016 (Salimans et al., 2016). It ran every generated image through a classifier pretrained on ImageNet’s one thousand categories and combined two properties of the resulting label predictions into a single number: whether each image looked confidently like one class, and whether the whole batch of images covered many different classes rather than collapsing onto a few. That gave the field its first widely reported automatic metric, and the Inception Score appears throughout the GAN-era results tables in this book, including the comparison in Chapter 6. It had an odd property, though: it never once looked at a real photograph. A generator could, in principle, score well by producing confident, varied fakes with no resemblance at all to the training distribution.

Three other gaps sat mostly unaddressed through the rest of the GAN era and into diffusion’s early years. There was no widely used way to check whether an image actually matched what a text prompt asked for, beyond ad hoc human spot-checks or a retrieval-style proxy like the R-precision metric used to evaluate AttnGAN and StackGAN (Chapter 7). There was no public evidence, at the scale of a billion-image training set, of whether a generative model actually reproduced specific training photographs rather than merely learning their statistics. And there was no cross-industry standard for marking a piece of media with where it came from — the Coalition for Content Provenance and Authenticity, founded in February 2021 by Adobe, Arm, the BBC, Intel, Microsoft, and Truepic to build exactly that standard (C2PA, 2021), predated by a year and a half the August 2022 public release of Stable Diffusion (Chapter 13) — the model that put high-volume, personal-GPU image generation, and everything that came with it, into general circulation with no default provenance signal attached at all.

One measurement fix in 2017 did not close the other three gaps; matching a prompt, catching memorization and marking provenance each waited for their own later paper. PNG ↓

The Breakthrough

FID and TTUR: comparing to a moving target, correctly. Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter’s paper actually bundled two separate fixes under one submission (Heusel et al., 2017). The first, the two time-scale update rule (TTUR), gave the generator and discriminator separate learning rates and proved, using stochastic approximation theory, that training under this scheme converges to a stationary local Nash equilibrium — the first such convergence guarantee for GAN training, addressed more fully alongside GAN training stability in Chapter 4. The second, the Fréchet Inception Distance (FID), fixed the Inception Score’s blind spot directly: instead of scoring generated images against a classifier’s opinion of what an object should look like, it passes both real and generated images through the same pretrained Inception-v3 network, fits a multivariate Gaussian to each set of resulting features, and measures the distance between those two Gaussians. A generator that drifts from the real distribution — through blur, noise, or the repetitive, low-diversity output of mode collapse — moves its Gaussian away from the real one, and FID increases. The paper’s own comparison across six kinds of synthetic distortion showed FID responding to each one, monotonically, in a way the Inception Score did not.

Rethinking the fit: FID reaches its limits. FID’s Gaussian assumption and its reliance on Inception-v3 — a 2015-era classifier trained on a thousand ImageNet categories — held up reasonably well for photorealistic, ImageNet-adjacent GAN output. It held up worse once text-to-image models started generating the far wider, stranger range of content real prompts ask for: illustrations, logos, mixed media, impossible scenes. Sadeep Jayasumana, Srikumar Ramalingam, Andreas Veit, Daniel Glasner, Ayan Chakrabarti, and Sanjiv Kumar, submitting in November 2023, argued that FID’s normality assumption is often wrong for this richer content, that Inception-v3’s features represent it poorly, and that FID needs a large sample — the original paper’s own protocol generates 50,000 images per model — to estimate reliably at all (Jayasumana et al., 2023). Their proposed replacement, CLIP-based Maximum Mean Discrepancy (CMMD), swaps Inception-v3 features for CLIP embeddings (Chapter 11) and computes an unbiased distance between the two sets of embeddings directly, without ever assuming they form a Gaussian. FID did not disappear from results tables afterward, but it stopped being treated as the last word on image quality for modern text-to-image models.

Pick-a-Pic: asking people, at scale, which image they actually prefer. Even a perfect distributional-distance metric only asks whether generated images look like real ones; it says nothing about whether a specific image matches a specific prompt the way a person wanted. Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy closed that gap with data rather than a cleverer formula (Kirstain et al., 2023). They built a web app that showed visitors two images generated from the same prompt — drawn from Stable Diffusion 2.1, Dreamlike Photoreal 2.0, and Stable Diffusion XL variants, each sampled at a different classifier-free guidance scale (Chapter 10) — and asked which one they preferred, or whether it was a tie. Over time this produced Pick-a-Pic: close to a million preference votes across nearly 67,000 distinct prompts, contributed by 6,394 real users of the app, and released as an open dataset. The authors then trained PickScore, a CLIP-style model, directly on those preference labels.

Close to a million real votes from 6,394 users turn into a scoring function that later fine-tunes generators, not just ranks them after the fact. PNG ↓

Extracting training data: memorization made visible and quantified. Language models had already been shown, in earlier work, to occasionally reproduce verbatim passages from their training text; Nicholas Carlini, Jamie Hayes, Milad Nasr, Matthew Jagielski, Vikash Sehwag, Florian Tramèr, Borja Balle, Daphne Ippolito, and Eric Wallace asked whether image diffusion models had the same weakness, and built a pipeline to find out (Carlini et al., 2023). Rather than searching a model’s entire training set, they exploited duplication: they picked the 350,000 most-duplicated captions in Stable Diffusion’s training data, generated 500 candidate images per caption — 175 million generations in total — and searched that output for clusters of near-identical images, on the logic that a caption a model was shown many times during training is the caption most likely to have been memorized rather than merely learned from statistically. Comparing the resulting clusters against the real training images they matched, the strict version of their detector confirmed 94 extracted images, rising to 109 near-copies once the team added manual inspection; they demonstrated the same weakness, to a lesser degree, on Imagen as well, and reported that diffusion models overall were measurably less private, by this test, than the GANs of the previous era.

The search only works because it exploits duplication: a caption seen many times in training is the one most likely to have been memorized. PNG ↓

Marking the output: watermarks and provenance metadata. Two different fixes emerged for a downstream question the previous three items don’t address at all: once an image is generated, how does anyone looking at it later know that it was? Pierre Fernandez, Guillaume Couairon, Hervé Jégou, Matthijs Douze, and Teddy Furon’s Stable Signature exploited a structural fact about latent diffusion models: every image the model ever produces passes through the same decoder (Fernandez et al., 2023). Fine-tune that one decoder, conditioned on a fixed binary signature, and every subsequent generation carries the signature baked into its pixels, recoverable later by a separately trained extractor network. C2PA took the opposite approach: rather than hiding a signal in pixels, its Content Credentials standard attaches a cryptographically signed manifest — what produced or edited a file, and when — to the file itself, as metadata rather than as an image alteration. OpenAI began attaching C2PA metadata to DALL-E 3 images generated through ChatGPT and its API in February 2024, the same week Meta announced AI-content labeling across its own platforms, and OpenAI joined the C2PA steering committee that May. Google took a third route with SynthID, a pixel-level watermark launched for Imagen images on Vertex AI in August 2023 — mechanically closer to Stable Signature’s approach than to C2PA’s, but a shipped product feature rather than a published, peer-reviewed method.

The Mathematics

The Inception Score combines a classifier’s per-image confidence with its across-batch diversity into one number:

\mathrm{IS} = \exp\Big( \mathbb{E}_{x \sim p_g}\big[\, D_{\mathrm{KL}}\big(p(y \mid x) \,\|\, p(y)\big) \,\big] \Big)

Symbol	Meaning
$x, p_g$	a generated image, and the generator’s output distribution
$p(y \mid x)$	the classifier’s label distribution for one image
$p(y)$	the average label distribution across the batch
$D_{\mathrm{KL}}$	Kullback-Leibler divergence between the two distributions
$\mathrm{IS}$	the resulting Inception Score

The Fréchet Inception Distance closes that gap by comparing two Gaussians fit to Inception-v3 features — one from real images, one from generated ones:

\mathrm{FID} = \lVert m - m_w \rVert_2^2 + \mathrm{Tr}\Big( C + C_w - 2\left(C\, C_w\right)^{1/2} \Big)

Symbol	Meaning
$m, C$	mean and covariance of generated-image Inception features
$m_w, C_w$	mean and covariance of real-image Inception features
$\mathrm{Tr}(\cdot)$	matrix trace, sum of diagonal entries
$\mathrm{FID}$	the resulting distance, lower means closer to the real distribution

Carlini and coauthors flagged a generated image as a candidate extraction using a distance normalized against how tightly clustered its own near-neighbors already are:

\ell(\hat x, x; S_{\hat x}) = \frac{\ell_2(\hat x, x)}{\alpha \cdot \mathbb{E}_{y \in S_{\hat x}}\big[\ell_2(\hat x, y)\big]}

$\hat x$ is a generated image and $x$ is a real training image it is being compared against; $S_{\hat x}$ is the set of the $n = 50$ training images closest to $\hat x$ , and the denominator averages the ordinary Euclidean distance $\ell_2$ between $\hat x$ and each of those neighbors, scaled by a constant $\alpha = 0.5$ . Dividing by that local average, rather than using raw pixel distance alone, keeps the test fair in regions of the training set that are naturally dense or sparse. The paper called an image extracted when this ratio fell below 0.15 for some real $x$ — meaning $\hat x$ sat far closer to that one training image than to its other neighbors, which is what a memorized near-copy looks like. What this buys you: a concrete, thresholded test for memorization, rather than an impression that two images “look similar.”

Symbol	Meaning
$\hat x, x$	generated image and a candidate matching real training image
$S_{\hat x}$	the $n=50$ real training images nearest to $\hat x$
$\ell_2(\cdot,\cdot)$	Euclidean distance between two images in feature space
$\alpha$	scaling constant, set to 0.5
$\ell(\hat x, x; S_{\hat x})$	the normalized distance; below 0.15 counted as extracted

PickScore turns Pick-a-Pic’s logged votes into a training signal with a scoring function and a softmax over the two candidate images:

\begin{aligned} s(x,y) &= E_{\mathrm{txt}}(x) \cdot E_{\mathrm{img}}(y) \cdot T \\ \hat p_i &= \frac{\exp\big(s(x,y_i)\big)}{\sum_{j=1}^{2} \exp\big(s(x,y_j)\big)} \\ \mathcal{L}_{\mathrm{pref}} &= \sum_{i=1}^{2} p_i \big(\log p_i - \log \hat p_i\big) \end{aligned}

$E_{\mathrm{txt}}$ and $E_{\mathrm{img}}$ are text and image encoders in a shared CLIP-style embedding space; $s(x,y)$ scores how well image $y$ matches prompt $x$ , scaled by a learned temperature $T$ . $\hat p_i$ turns the two candidate images’ scores into a predicted probability that image $i$ is the preferred one, and $\mathcal{L}_{\mathrm{pref}}$ is a cross-entropy loss against $p_i$ , the actual human label (1 and 0 for a clear pick, 0.5 and 0.5 for a tie). What this buys you: a scoring function trained directly on what real users picked, rather than on a proxy like class confidence or caption retrieval. This is the two-option case of the same preference-learning setup that trains reward models for language-model alignment, where the general Bradley-Terry math and its use inside RLHF are covered in Language Models, Chapter 13; PickScore applies the identical logic to CLIP embeddings and image pairs instead of language-model responses.

Symbol	Meaning
$E_{\mathrm{txt}}, E_{\mathrm{img}}$	text and image encoders in a shared embedding space
$s(x,y), T$	prompt-image score, and a learned temperature scaling it
$\hat p_i, p_i$	predicted and actual preference probability for image $i$
$\mathcal{L}_{\mathrm{pref}}$	cross-entropy preference loss

Every method in this chapter spends compute to look at a model rather than to run it for a user. Carlini and coauthors’ extraction pipeline alone required 175 million Stable Diffusion generations to surface 94 confirmed copies — inference compute spent purely to audit the model, with no product-facing output attached to any of it. Pick-a-Pic’s data collection ran a live generation service for every visitor to its web app, and membership-inference attacks of the kind used alongside the extraction study typically require training many shadow copies of a model rather than one. As base models grow larger and a single inference pass gets more expensive, safety and evaluation work increasingly needs its own line in a compute budget rather than a rounding error against training — the same accelerator cost curves that shape training runs, covered in Engine Room, Chapter 20, also gate how much a lab can afford to spend looking for its own model’s failures.

What It Enabled

None of the five threads in this chapter replaced the others; a product team now typically needs all of them, applied to the same model at different stages of shipping it.

Metric	Introduced	Compares to real images	Feature space	Known weakness
Inception Score	2016	No	Inception-v3, 1,000 ImageNet classes	Rewards confident, diverse fakes with no real-data reference; insensitive to several distortion types
FID via TTUR	2017	Yes	Inception-v3 pooling features	Assumes Gaussian feature distributions; needs roughly 50,000 samples to estimate reliably
CMMD	2023	Yes	CLIP embeddings	No distributional assumption and more sample-efficient, but far less widely reported than FID as of this writing

Provenance mechanism	Kind	What it carries	Example deployment	Robustness limit
Stable Signature	Pixel watermark via decoder fine-tuning	A fixed binary signature	Applied per latent-diffusion release	Survives heavy cropping in testing; not yet a cross-vendor standard
SynthID	Pixel watermark, product feature	An imperceptible pixel pattern	Google Imagen on Vertex AI, August 2023	Single-vendor, proprietary at launch
C2PA Content Credentials	File-level metadata standard	A cryptographically signed edit history	Adobe, OpenAI, Google, Meta, and camera makers	Stripped by re-encoding or by platforms that discard metadata

FID and its lineage answered “does this look like the real distribution,” and CMMD’s arrival shows that lineage is still being revised rather than settled; the analogous evaluation problem for text — measuring whether a language model’s outputs are actually good rather than merely fluent — is its own extended argument, covered in Language Models, Chapter 22, and video generation inherited the same distributional-distance idea directly, as the Fréchet Video Distance, covered in Video Generation, Chapter 18. Pick-a-Pic answered “does this match what a person wanted,” and PickScore-style preference models are now used both to rank existing outputs and, increasingly, to fine-tune generators directly against human preference rather than just to score them after the fact. Carlini and coauthors’ extraction result reframed a training-data question that personalization techniques like DreamBooth and textual inversion (Chapter 16) had already made commercially urgent from the other direction — deliberately teaching a model one person’s likeness from a handful of photos raises the same consent and reproduction questions that unintentional memorization does, just on purpose rather than by accident. And the provenance mechanisms answered a question none of the other four touch: once an image exists, how does anyone downstream know where it came from.

Three chains, not one: quality, preference and provenance never merge, because each answers a question the other two do not touch. PNG ↓

Key Papers

Paper	Authors	Year	Link	Why it matters
Improved Techniques for Training GANs	Salimans, Goodfellow, Zaremba, Cheung, Radford, and Chen	2016	arXiv:1606.03498	Introduced the Inception Score, the field’s first widely used automatic quality metric, with a built-in blind spot: it never compares to real images
GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium	Heusel, Ramsauer, Unterthiner, Nessler, and Hochreiter	2017	arXiv:1706.08500	Introduced the Fréchet Inception Distance, comparing generated and real image statistics directly, alongside a convergence proof for two-time-scale GAN training
Rethinking FID: Towards a Better Evaluation Metric for Image Generation	Jayasumana, Ramalingam, Veit, Glasner, Chakrabarti, and Kumar	2023	arXiv:2401.09603	Identified FID’s Gaussian assumption and outdated feature space as failure modes for modern text-to-image content, and proposed CLIP-based CMMD as a replacement
Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation	Kirstain, Polyak, Singer, Matiana, Penna, and Levy	2023	arXiv:2305.01569	Collected close to a million real human preference votes and trained PickScore, which exceeded individual human-annotator agreement on the same task
Extracting Training Data from Diffusion Models	Carlini, Hayes, Nasr, Jagielski, Sehwag, Tramèr, Balle, Ippolito, and Wallace	2023	arXiv:2301.13188	Demonstrated and quantified verbatim memorization in deployed diffusion models, extracting 94 to 109 near-identical training images at scale
The Stable Signature: Rooting Watermarks in Latent Diffusion Models	Fernandez, Couairon, Jégou, Douze, and Furon	2023	arXiv:2303.15435	Fine-tuned a latent diffusion model’s shared decoder to embed a recoverable watermark in every image it ever generates
C2PA Founding Press Release	C2PA	2021	c2pa.org	Founded the cross-industry content-provenance standard that OpenAI, Adobe, Google, and Meta later adopted for AI-generated images

PM Lens

When I scope an image-generation feature now, evaluation, memorization, and provenance are three separate line items, not one quality checkbox. FID and its successors are fine for catching whether a fine-tuned checkpoint has regressed, but they say nothing about legal exposure — Getty Images’ lawsuits against Stability AI over training data are a reminder that the training-data question is a legal review, not a benchmark score. Adobe’s answer has been commercial as much as technical: Firefly’s Premium plan, a few dollars a month for individual creators, bundles IP indemnification that most competing image generators still don’t offer, which is why some brand-safety-conscious clients default to it regardless of raw output quality. On provenance, I’ve stopped assuming a C2PA credential survives a customer’s actual pipeline — it disappears the moment someone screenshots an image or re-uploads it through a platform that strips metadata — so a pixel-level watermark, in the spirit of Stable Signature, is the fallback I ask engineering about before shipping anything meant to stay traceable downstream.

Search