skip to content

Search
← All concepts

#training-stability

Normalization, initialization, loss-spike engineering.

4 chapters, oldest era first: one idea traced across the books.

  1. From Noise to Image 2015–2017 Ch. 4: Taming GAN Training DCGAN recipes, the Wasserstein distance, gradient penalties, and the birth of quantitative image-quality metrics.
  2. The Engine Room 2015–2022 Ch. 6: Keeping Training Alive BatchNorm, LayerNorm, RMSNorm, initialization, residuals, and muP transfer.
  3. The Engine Room 2017–2024 Ch. 7: Mixed Precision FP32, FP16, BF16, FP8 and loss scaling: halving the bytes doubles the model.
  4. The Engine Room 2022–2024 Ch. 11: Training Runs in Practice Loss spikes, node failures, checkpointing, and MFU — what the OPT, BLOOM, and Llama 3 logbooks reveal.