skip to content

Search
← All concepts

#optimization

Optimizers, schedules, and training dynamics.

5 chapters, oldest era first: one idea traced across the books.

  1. The Engine Room 2014–2025 Ch. 5: Optimizers SGD to momentum to Adam, AdamW, LAMB, and beyond — the update equations that train everything.
  2. From Noise to Image 2015–2017 Ch. 4: Taming GAN Training DCGAN recipes, the Wasserstein distance, gradient penalties, and the birth of quantitative image-quality metrics.
  3. The Engine Room 2015–2022 Ch. 6: Keeping Training Alive BatchNorm, LayerNorm, RMSNorm, initialization, residuals, and muP transfer.
  4. The Engine Room 2017–2019 Ch. 8: Data Parallelism and Large Batches Ring all-reduce, the linear scaling rule, and the gradient noise scale.
  5. The Language Machines 2023–2024 Ch. 15: Preference Optimization Beyond PPO DPO's closed-form trick, Constitutional AI and RLAIF, and GRPO.