skip to content

Search
← All concepts

#inference-optimization

Fast, cheap generation at serving time.

8 chapters, oldest era first: one idea traced across the books.

  1. The Engine Room 2015–2025 Ch. 16: Distillation Soft targets and temperature, DistilBERT, and distilling reasoning traces.
  2. From Noise to Image 2020–2021 Ch. 9: Scores, SDEs, and Faster Sampling The SDE/ODE unification, probability flow, and DDIM's deterministic shortcut.
  3. The Engine Room 2020–2025 Ch. 18: Beyond Attention Linear attention, S4 and Mamba's selective state spaces, RWKV, hybrids — and why attention persists.
  4. From Noise to Image 2022–2023 Ch. 18: Few-Step Generation Progressive distillation, consistency models, LCMs, and adversarial distillation: 50 steps to 1.
  5. The Engine Room 2022–2024 Ch. 15: Quantization Int8 outliers, GPTQ's Hessian trick, and AWQ — the math of throwing bits away safely.
  6. The Engine Room 2022–2025 Ch. 17: Inference Engines The KV-cache memory formula, multi-query and grouped-query attention, PagedAttention's block-based memory management, speculative decoding, and continuous batching.
  7. From Frames to Worlds 2024–2026 Ch. 14: Making Video Cheap Step distillation for video, causal decoders, and the road to real-time generation.
  8. The Engine Room 2024–2026 Ch. 19: The Economics of Test-Time Compute Inference scaling as the new capex: train once versus think longer, and the batch economics of reasoning.