Timeline
Every dated event this book covers, in one place. Each row links to the chapter that teaches it.
| Chapter | Title | Era |
|---|---|---|
| 1 | The Hardware Lottery | 1958–2009 |
| 2 | GPUs Learn to Think | 2007–2014 |
| 3 | TPUs and the Accelerator Cambrian | 2015–2024 |
| 4 | The Memory Wall | Cross-era |
| 5 | Optimizers | 2014–2025 |
| 6 | Keeping Training Alive | 2015–2022 |
| 7 | Mixed Precision | 2017–2024 |
| 8 | Data Parallelism and Large Batches | 2017–2019 |
| 9 | Splitting the Model | 2018–2022 |
| 10 | ZeRO and the Memory Tricks | 2016–2023 |
| 11 | Training Runs in Practice | 2022–2024 |
| 12 | FlashAttention and the Kernel Era | 2022–2024 |
| 13 | Mixture of Experts | 2017–2025 |
| 14 | PEFT and LoRA | 2019–2023 |
| 15 | Quantization | 2022–2024 |
| 16 | Distillation | 2015–2025 |
| 17 | Inference Engines | 2022–2025 |
| 18 | Beyond Attention | 2020–2025 |
| 19 | The Economics of Test-Time Compute | 2024–2026 |
| 20 | The Co-evolution Thesis | 2012–2026 |
Reading List
Every paper cited across The Engine Room, 70 in total, in chronological order.
| Year | Paper | Authors | Link |
|---|---|---|---|
| 1958 | The perceptron: a probabilistic model for information storage and organization in the brain | Rosenblatt | link |
| 1969 | Perceptrons: An Introduction to Computational Geometry | Minsky and Papert | link |
| 1986 | Learning representations by back-propagating errors | Rumelhart, Hinton, and Williams | link |
| 1995 | Hitting the Memory Wall: Implications of the Obvious | Wulf & McKee | link |
| 1998 | Gradient-Based Learning Applied to Document Recognition | LeCun, Bottou, Bengio, and Haffner | link |
| 2006 | A Fast Learning Algorithm for Deep Belief Nets | Hinton, Osindero, and Teh | link |
| 2009 | ImageNet: A Large-Scale Hierarchical Image Database | Deng, Dong, Socher, Li, Li, and Fei-Fei | link |
| 2009 | Large-scale Deep Unsupervised Learning Using Graphics Processors | Raina, Madhavan, and Ng | link |
| 2009 | Roofline: An Insightful Visual Performance Model for Multicore Architectures | Williams et al. | link |
| 2012 | ImageNet Classification with Deep Convolutional Neural Networks | Krizhevsky, Sutskever, and Hinton | link |
| 2014 | Adam: A Method for Stochastic Optimization | Kingma & Ba | 1412.6980 |
| 2014 | cuDNN: Efficient Primitives for Deep Learning | Chetlur et al. | 1410.0759 |
| 2015 | Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift | Ioffe and Szegedy | 1502.03167 |
| 2015 | Distilling the Knowledge in a Neural Network | Hinton et al. | 1503.02531 |
| 2016 | Layer Normalization | Ba et al. | 1607.06450 |
| 2016 | Training Deep Nets with Sublinear Memory Cost | Chen, Xu, Zhang, and Guestrin | 1604.06174 |
| 2017 | Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour | Goyal et al. | 1706.02677 |
| 2017 | Decoupled Weight Decay Regularization | Loshchilov & Hutter | 1711.05101 |
| 2017 | In-Datacenter Performance Analysis of a Tensor Processing Unit | Jouppi et al. | 1704.04760 |
| 2017 | Mixed Precision Training | Micikevicius et al. | 1710.03740 |
| 2017 | Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer | Shazeer et al. | 1701.06538 |
| 2018 | An Empirical Model of Large-Batch Training | McCandlish et al. | 1812.06162 |
| 2018 | GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism | Huang et al. | 1811.06965 |
| 2018 | Horovod: fast and easy distributed deep learning in TensorFlow | Sergeev and Del Balso | 1802.05799 |
| 2018 | Shampoo: Preconditioned Stochastic Tensor Optimization | Gupta et al. | 1802.09568 |
| 2019 | A Study of BFLOAT16 for Deep Learning Training | Kalamkar et al. | 1905.12322 |
| 2019 | DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter | Sanh et al. | 1910.01108 |
| 2019 | Fast Transformer Decoding: One Write-Head is All You Need | Shazeer | 1911.02150 |
| 2019 | Large Batch Optimization for Deep Learning: Training BERT in 76 Minutes | You et al. | 1904.00962 |
| 2019 | Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism | Shoeybi et al. | 1909.08053 |
| 2019 | Parameter-Efficient Transfer Learning for NLP | Houlsby et al. | 1902.00751 |
| 2019 | Root Mean Square Layer Normalization | Zhang and Sennrich | 1910.07467 |
| 2019 | The Bitter Lesson | Sutton | link |
| 2019 | ZeRO: Memory Optimizations Toward Training Trillion Parameter Models | Rajbhandari, Rasley, Ruwase, and He | 1910.02054 |
| 2020 | The Hardware Lottery | Hooker | 2009.06489 |
| 2020 | Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention | Katharopoulos, Vyas, Pappas, and Fleuret | 2006.16236 |
| 2021 | Efficiently Modeling Long Sequences with Structured State Spaces | Gu, Goel, and Ré | 2111.00396 |
| 2021 | LoRA: Low-Rank Adaptation of Large Language Models | Hu et al. | 2106.09685 |
| 2021 | Prefix-Tuning: Optimizing Continuous Prompts for Generation | Li & Liang | 2101.00190 |
| 2021 | Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity | Fedus et al. | 2101.03961 |
| 2021 | ZeRO-Offload: Democratizing Billion-Scale Model Training | Ren et al. | 2101.06840 |
| 2022 | BLOOM: A 176B-Parameter Open-Access Multilingual Language Model | Le Scao et al. | 2211.05100 |
| 2022 | FP8 Formats for Deep Learning | Micikevicius et al. | 2209.05433 |
| 2022 | Fast Inference from Transformers via Speculative Decoding | Leviathan, Kalman, and Matias | 2211.17192 |
| 2022 | FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness | Dao et al. | 2205.14135 |
| 2022 | GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers | Frantar et al. | 2210.17323 |
| 2022 | LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale | Dettmers et al. | 2208.07339 |
| 2022 | OPT: Open Pre-trained Transformer Language Models | Zhang et al. | 2205.01068 |
| 2022 | SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models | Xiao et al. | 2211.10438 |
| 2022 | Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer | Yang et al. | 2203.03466 |
| 2022 | Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model | Smith et al. | 2201.11990 |
| 2023 | AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration | Lin et al. | 2306.00978 |
| 2023 | Efficient Memory Management for Large Language Model Serving with PagedAttention | Kwon et al. | 2309.06180 |
| 2023 | FP8-LM: Training FP8 Large Language Models | Peng et al. | 2310.18313 |
| 2023 | FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning | Dao | 2307.08691 |
| 2023 | GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints | Ainslie et al. | 2305.13245 |
| 2023 | Mamba: Linear-Time Sequence Modeling with Selective State Spaces | Gu and Dao | 2312.00752 |
| 2023 | Orca: Progressive Learning from Complex Explanation Traces of GPT-4 | Mukherjee et al. | 2306.02707 |
| 2023 | PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel | Zhao et al. | 2304.11277 |
| 2023 | QLoRA: Efficient Finetuning of Quantized LLMs | Dettmers et al. | 2305.14314 |
| 2023 | RWKV: Reinventing RNNs for the Transformer Era | Peng et al. | 2305.13048 |
| 2023 | TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddings | Jouppi et al. | 2304.01433 |
| 2024 | DeepSeek-V3 Technical Report | DeepSeek-AI | 2412.19437 |
| 2024 | FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision | Shah et al. | 2407.08608 |
| 2024 | Jamba: A Hybrid Transformer-Mamba Language Model | Lieber et al. | 2403.19887 |
| 2024 | Mixtral of Experts | Jiang et al. | 2401.04088 |
| 2024 | Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters | Snell et al. | 2408.03314 |
| 2024 | The Llama 3 Herd of Models | Grattafiori et al. | 2407.21783 |
| 2025 | DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning | DeepSeek-AI | 2501.12948 |
| 2025 | Energy and AI | International Energy Agency | link |