skip to content

Search
← All concepts

#memory-efficiency

ZeRO, checkpointing, offloading.

7 chapters, oldest era first: one idea traced across the books.

  1. The Engine Room Cross-era Ch. 4: The Memory Wall The memory hierarchy, HBM, the roofline model, and why LLM inference is bandwidth-bound.
  2. The Engine Room 2016–2023 Ch. 10: ZeRO and the Memory Tricks Bytes-per-parameter accounting, gradient checkpointing, ZeRO stages, and FSDP.
  3. The Engine Room 2018–2022 Ch. 9: Splitting the Model Tensor-parallel matrix math, pipeline bubbles, and 3D parallelism.
  4. The Engine Room 2019–2023 Ch. 14: PEFT and LoRA Adapters, prompt and prefix tuning, the low-rank update, and QLoRA's NF4.
  5. The Engine Room 2022–2024 Ch. 12: FlashAttention and the Kernel Era Online softmax, tiling, and IO-complexity: software catching up to silicon.
  6. The Engine Room 2022–2024 Ch. 15: Quantization Int8 outliers, GPTQ's Hessian trick, and AWQ — the math of throwing bits away safely.
  7. The Engine Room 2022–2025 Ch. 17: Inference Engines The KV-cache memory formula, multi-query and grouped-query attention, PagedAttention's block-based memory management, speculative decoding, and continuous batching.