skip to content

Search
← All concepts

#scaling-laws

Compute/data/parameter ↔ performance relationships.

15 chapters, oldest era first: one idea traced across the books.

  1. The Engine Room 2012–2026 Ch. 20: The Co-evolution Thesis The master mapping: GTX 580 to AlexNet, V100 to the transformer, A100 to GPT-3, H100 to GPT-4-class, GB200 to reasoning — and what the next silicon implies.
  2. The Engine Room 2015–2022 Ch. 6: Keeping Training Alive BatchNorm, LayerNorm, RMSNorm, initialization, residuals, and muP transfer.
  3. From Noise to Image 2017–2020 Ch. 6: The GAN Golden Age ProGAN to StyleGAN2: style modulation, latent arithmetic, and photorealistic faces of people who do not exist.
  4. The Engine Room 2017–2019 Ch. 8: Data Parallelism and Large Batches Ring all-reduce, the linear scaling rule, and the gradient noise scale.
  5. The Engine Room 2017–2025 Ch. 13: Mixture of Experts Router softmax, top-k dispatch, load-balancing losses — paying for a fraction of the model.
  6. The Language Machines 2018–2020 Ch. 7: The GPT Lineage Decoder-only scaling from GPT-1 to GPT-3, and in-context learning as an emergent phenomenon.
  7. The Language Machines 2020–2022 Ch. 10: Scaling Laws Power laws, compute-optimal training, the Chinchilla re-balance, and the emergence debate.
  8. From Noise to Image 2022 Ch. 14: The Year Images Went Mainstream DALL-E 2, Imagen, Parti, and Midjourney: unCLIP, giant text encoders, cascades, and autoregression at scale.
  9. From Noise to Image 2022–2024 Ch. 19: Flow Matching and Transformer Backbones Velocity-field regression, rectified flow, and DiT replacing the U-Net — the SD3/FLUX lineage.
  10. The Language Machines 2022–2023 Ch. 14: ChatGPT and the Open Explosion The product moment, then the LLaMA leak, Alpaca, Llama 2, and Mistral.
  11. The Language Machines 2022–2024 Ch. 18: Multimodal Frontier Models Vision encoders meet LLMs: Flamingo, GPT-4, LLaVA, Gemini.
  12. The Language Machines 2022–2026 Ch. 20: Reasoning Models STaR, process rewards, o1, RLVR, and DeepSeek-R1 — test-time compute as the new scaling axis.
  13. From Noise to Image 2024–2026 Ch. 20: The Current Frontier Autoregression strikes back: VAR, native multimodal generation inside LLMs, and instruction-following images.
  14. From Frames to Worlds 2024 Ch. 9: Sora and Spacetime Patches Diffusion transformers for video, spacetime patches, and the video-models-as-world-simulators claim.
  15. From Frames to Worlds 2024–2025 Ch. 10: The Commercial Race Gen-3, Kling, Veo, Pika, Luma: capability versus cost per second of footage.