#evaluation

Metrics, benchmarks, human preference measurement.

11 chapters, oldest era first: one idea traced across the books.

The Language Machines 1948–2003 Ch. 1: From Shannon to n-grams Language modeling as next-symbol prediction, perplexity, and Bengio's first neural language model.
From Noise to Image 2006–2013 Ch. 1: The Generative Problem What modeling p(x) means, density versus sampling, and why image generation stalled until deep learning's revival.
From Noise to Image 2015–2017 Ch. 4: Taming GAN Training DCGAN recipes, the Wasserstein distance, gradient penalties, and the birth of quantitative image-quality metrics.
From Noise to Image Cross-era Ch. 21: Evaluation, Provenance, and Safety FID's math and failure modes, human preference data, memorization and extraction, watermarking and C2PA.
From Frames to Worlds Cross-era Ch. 18: Evaluating and Monetizing Motion FVD's blind spots, VBench, physics tests, and the economics of the most expensive pixel in AI.
The Language Machines 2020–2025 Ch. 11: The Data Question Corpora, dedup, filtering, synthetic data, and contamination — data as the real moat.
The Language Machines 2020–2024 Ch. 17: Retrieval and Grounding RAG, RETRO, and hallucination: knowledge in weights versus knowledge in indexes.
The Language Machines Cross-era Ch. 22: Measuring Minds and the Frontier MMLU to HELM to Arena, contamination, benchmark saturation, and the open problems.
From Noise to Image 2021–2022 Ch. 10: Guidance Classifier guidance, the classifier-free guidance formula, and the dial between fidelity and diversity.
From Noise to Image 2021 Ch. 11: CLIP: Bridging Text and Image The InfoNCE contrastive loss, a shared embedding space for words and pictures, and why it unlocked text-to-image.
The Language Machines 2022–2026 Ch. 21: Agents The ReAct loop, tool use, computer use, SWE-bench, and MCP — from chat to work.