skip to content

Search
← All concepts

#distributed-training

Data/tensor/pipeline parallelism.

5 chapters, oldest era first: one idea traced across the books.

  1. The Engine Room 2016–2023 Ch. 10: ZeRO and the Memory Tricks Bytes-per-parameter accounting, gradient checkpointing, ZeRO stages, and FSDP.
  2. The Engine Room 2017–2019 Ch. 8: Data Parallelism and Large Batches Ring all-reduce, the linear scaling rule, and the gradient noise scale.
  3. The Engine Room 2017–2025 Ch. 13: Mixture of Experts Router softmax, top-k dispatch, load-balancing losses — paying for a fraction of the model.
  4. The Engine Room 2018–2022 Ch. 9: Splitting the Model Tensor-parallel matrix math, pipeline bubbles, and 3D parallelism.
  5. The Engine Room 2022–2024 Ch. 11: Training Runs in Practice Loss spikes, node failures, checkpointing, and MFU — what the OPT, BLOOM, and Llama 3 logbooks reveal.