skip to content

Search
← All concepts

#hardware

GPUs, TPUs, accelerators, memory and interconnect.

9 chapters, oldest era first: one idea traced across the books.

  1. The Engine Room 1958–2009 Ch. 1: The Hardware Lottery Ideas win when hardware fits: why neural networks waited for GPUs — the book's thesis.
  2. The Engine Room 1958–2026 Appendix: Timeline and Reading List A one-page dated timeline of hardware and training methods and every paper cited in this book.
  3. The Engine Room 2007–2014 Ch. 2: GPUs Learn to Think CUDA, AlexNet on two gaming GPUs, and cuDNN — graphics chips become math engines.
  4. The Engine Room Cross-era Ch. 4: The Memory Wall The memory hierarchy, HBM, the roofline model, and why LLM inference is bandwidth-bound.
  5. The Engine Room 2012–2026 Ch. 20: The Co-evolution Thesis The master mapping: GTX 580 to AlexNet, V100 to the transformer, A100 to GPT-3, H100 to GPT-4-class, GB200 to reasoning — and what the next silicon implies.
  6. The Engine Room 2015–2024 Ch. 3: TPUs and the Accelerator Cambrian Systolic arrays, TPU v1 through v5, Cerebras and Groq — specialize versus generalize.
  7. The Engine Room 2017–2024 Ch. 7: Mixed Precision FP32, FP16, BF16, FP8 and loss scaling: halving the bytes doubles the model.
  8. The Engine Room 2018–2022 Ch. 9: Splitting the Model Tensor-parallel matrix math, pipeline bubbles, and 3D parallelism.
  9. The Engine Room 2022–2024 Ch. 12: FlashAttention and the Kernel Era Online softmax, tiling, and IO-complexity: software catching up to silicon.