Timeline
Every dated event this book covers, in one place. Each row links to the chapter that teaches it.
| Chapter | Title | Era |
|---|---|---|
| 1 | From Shannon to n-grams | 1948–2003 |
| 2 | word2vec and the Geometry of Meaning | 2013–2014 |
| 3 | RNNs, LSTMs, and seq2seq | 1997–2015 |
| 4 | Attention | 2014–2015 |
| 5 | The Transformer | 2017 |
| 6 | Pretrain, Then Fine-tune | 2018 |
| 7 | The GPT Lineage | 2018–2020 |
| 8 | Tokenization | 2015–2024 |
| 9 | One Model, Many Tasks | 2019 |
| 10 | Scaling Laws | 2020–2022 |
| 11 | The Data Question | 2020–2025 |
| 12 | Instruction Tuning | 2021–2022 |
| 13 | RLHF | 2017–2022 |
| 14 | ChatGPT and the Open Explosion | 2022–2023 |
| 15 | Preference Optimization Beyond PPO | 2023–2024 |
| 16 | Long Context | 2021–2024 |
| 17 | Retrieval and Grounding | 2020–2024 |
| 18 | Multimodal Frontier Models | 2022–2024 |
| 19 | Prompting Becomes Reasoning | 2022–2023 |
| 20 | Reasoning Models | 2022–2026 |
| 21 | Agents | 2022–2026 |
| 22 | Measuring Minds and the Frontier | Cross-era |
Reading List
Every paper cited across The Language Machines, 112 in total, in chronological order.
| Year | Paper | Authors | Link |
|---|---|---|---|
| 1948 | A Mathematical Theory of Communication | Shannon | link |
| 1951 | Prediction and Entropy of Printed English | Shannon | link |
| 1952 | Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons | Bradley and Terry | link |
| 1987 | Estimation of Probabilities from Sparse Data for the Language Model Component of a Speech Recognizer | Katz | link |
| 1990 | Finding Structure in Time | Elman | link |
| 1994 | Learning Long-Term Dependencies with Gradient Descent is Difficult | Bengio, Simard & Frasconi | link |
| 1995 | Improved Backing-Off for M-gram Language Modeling | Kneser and Ney | link |
| 1996 | An Empirical Study of Smoothing Techniques for Language Modeling | Chen and Goodman | 9606011 |
| 1997 | Long Short-Term Memory | Hochreiter & Schmidhuber | link |
| 2000 | Learning to Forget: Continual Prediction with LSTM | Gers, Schmidhuber & Cummins | link |
| 2003 | A Neural Probabilistic Language Model | Bengio et al. | link |
| 2010 | Recurrent Neural Network Based Language Model | Mikolov et al. | link |
| 2012 | On the difficulty of training Recurrent Neural Networks | Pascanu, Mikolov & Bengio | 1211.5063 |
| 2013 | Distributed Representations of Words and Phrases and their Compositionality | Mikolov et al. | 1310.4546 |
| 2013 | Efficient Estimation of Word Representations in Vector Space | Mikolov et al. | 1301.3781 |
| 2014 | GloVe: Global Vectors for Word Representation | Pennington et al. | link |
| 2014 | Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation | Cho et al. | 1406.1078 |
| 2014 | Neural Machine Translation by Jointly Learning to Align and Translate | Bahdanau et al. | 1409.0473 |
| 2014 | On the Properties of Neural Machine Translation: Encoder-Decoder Approaches | Cho et al. | 1409.1259 |
| 2014 | Sequence to Sequence Learning with Neural Networks | Sutskever, Vinyals & Le | 1409.3215 |
| 2015 | Effective Approaches to Attention-based Neural Machine Translation | Luong et al. | 1508.04025 |
| 2015 | Neural Machine Translation of Rare Words with Subword Units | Sennrich et al. | 1508.07909 |
| 2015 | Show, Attend and Tell: Neural Image Caption Generation with Visual Attention | Xu et al. | 1502.03044 |
| 2016 | Enriching Word Vectors with Subword Information | Bojanowski et al. | 1607.04606 |
| 2016 | Google’s Neural Machine Translation System: Bridging the Gap between Human and Machine Translation | Wu et al. | 1609.08144 |
| 2016 | Layer Normalization | Ba et al. | 1607.06450 |
| 2017 | Attention Is All You Need | Vaswani et al. | 1706.03762 |
| 2017 | Deep Reinforcement Learning from Human Preferences | Christiano et al. | 1706.03741 |
| 2017 | Proximal Policy Optimization Algorithms | Schulman et al. | 1707.06347 |
| 2018 | BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding | Devlin et al. | 1810.04805 |
| 2018 | Deep contextualized word representations | Peters et al. | 1802.05365 |
| 2018 | GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding | Wang et al. | 1804.07461 |
| 2018 | Improving Language Understanding by Generative Pre-Training | Radford et al. | link |
| 2018 | SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing | Kudo & Richardson | 1808.06226 |
| 2018 | Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates | Kudo | 1804.10959 |
| 2018 | Universal Language Model Fine-tuning for Text Classification | Howard & Ruder | 1801.06146 |
| 2019 | BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension | Lewis et al. | 1910.13461 |
| 2019 | Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer | Raffel et al. | 1910.10683 |
| 2019 | Language Models are Unsupervised Multitask Learners | Radford et al. | link |
| 2019 | MASS: Masked Sequence to Sequence Pre-training for Language Generation | Song et al. | 1905.02450 |
| 2019 | RoBERTa: A Robustly Optimized BERT Pretraining Approach | Liu et al. | 1907.11692 |
| 2019 | Unified Language Model Pre-training for Natural Language Understanding and Generation | Dong et al. | 1905.03197 |
| 2020 | Dense Passage Retrieval for Open-Domain Question Answering | Karpukhin et al. | 2004.04906 |
| 2020 | Language Models are Few-Shot Learners | Brown et al. | 2005.14165 |
| 2020 | Learning to Summarize from Human Feedback | Stiennon et al. | 2009.01325 |
| 2020 | Measuring Massive Multitask Language Understanding | Hendrycks et al. | 2009.03300 |
| 2020 | REALM: Retrieval-Augmented Language Model Pre-Training | Guu et al. | 2002.08909 |
| 2020 | Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks | Lewis et al. | 2005.11401 |
| 2020 | Scaling Laws for Neural Language Models | Kaplan et al. | 2001.08361 |
| 2020 | mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer | Xue et al. | 2010.11934 |
| 2021 | ByT5: Towards a token-free future with pre-trained byte-to-byte models | Xue et al. | 2105.13626 |
| 2021 | Cross-Task Generalization via Natural Language Crowdsourcing Instructions | Mishra et al. | 2104.08773 |
| 2021 | Deduplicating Training Data Makes Language Models Better | Lee et al. | 2107.06499 |
| 2021 | Evaluating Large Language Models Trained on Code | Chen et al. | 2107.03374 |
| 2021 | Finetuned Language Models Are Zero-Shot Learners | Wei et al. | 2109.01652 |
| 2021 | Improving Language Models by Retrieving from Trillions of Tokens | Borgeaud et al. | 2112.04426 |
| 2021 | Multitask Prompted Training Enables Zero-Shot Task Generalization | Sanh et al. | 2110.08207 |
| 2021 | RoFormer: Enhanced Transformer with Rotary Position Embedding | Su et al. | 2104.09864 |
| 2021 | Scaling Language Models: Methods, Analysis & Insights from Training Gopher | Rae et al. | 2112.11446 |
| 2021 | The Pile: An 800GB Dataset of Diverse Text for Language Modeling | Gao et al. | 2101.00027 |
| 2021 | Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation | Press et al. | 2108.12409 |
| 2021 | TruthfulQA: Measuring How Models Mimic Human Falsehoods | Lin et al. | 2109.07958 |
| 2022 | Chain-of-Thought Prompting Elicits Reasoning in Large Language Models | Wei et al. | 2201.11903 |
| 2022 | Constitutional AI: Harmlessness from AI Feedback | Bai et al. | 2212.08073 |
| 2022 | Emergent Abilities of Large Language Models | Wei et al. | 2206.07682 |
| 2022 | Flamingo: a Visual Language Model for Few-Shot Learning | Alayrac et al. | 2204.14198 |
| 2022 | Holistic Evaluation of Language Models | Liang et al. | 2211.09110 |
| 2022 | Introducing ChatGPT | OpenAI | link |
| 2022 | PaLM: Scaling Language Modeling with Pathways | Chowdhery et al. | 2204.02311 |
| 2022 | ReAct: Synergizing Reasoning and Acting in Language Models | Yao et al. | 2210.03629 |
| 2022 | STaR: Bootstrapping Reasoning With Reasoning | Zelikman et al. | 2203.14465 |
| 2022 | Scaling Instruction-Finetuned Language Models | Chung et al. | 2210.11416 |
| 2022 | Self-Consistency Improves Chain of Thought Reasoning in Language Models | Wang et al. | 2203.11171 |
| 2022 | Self-Instruct: Aligning Language Models with Self-Generated Instructions | Wang et al. | 2212.10560 |
| 2022 | Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks | Wang et al. | 2204.07705 |
| 2022 | Training Compute-Optimal Large Language Models | Hoffmann et al. | 2203.15556 |
| 2022 | Training Language Models to Follow Instructions with Human Feedback | Ouyang et al. | 2203.02155 |
| 2022 | What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization? | Wang et al. | 2204.05832 |
| 2023 | Alpaca: A Strong, Replicable Instruction-Following Model | Taori et al. | link |
| 2023 | Are Emergent Abilities of Large Language Models a Mirage? | Schaeffer et al. | 2304.15004 |
| 2023 | Direct Preference Optimization: Your Language Model is Secretly a Reward Model | Rafailov et al. | 2305.18290 |
| 2023 | Extending Context Window of Large Language Models via Positional Interpolation | Chen et al. | 2306.15595 |
| 2023 | GPQA: A Graduate-Level Google-Proof Q&A Benchmark | Rein et al. | 2311.12022 |
| 2023 | GPT-4 Technical Report | OpenAI | 2303.08774 |
| 2023 | Gemini: A Family of Highly Capable Multimodal Models | Gemini Team, Google | 2312.11805 |
| 2023 | LLaMA: Open and Efficient Foundation Language Models | Touvron et al. | 2302.13971 |
| 2023 | Language Model Tokenizers Introduce Unfairness Between Languages | Petrov et al. | 2305.15425 |
| 2023 | Let’s Verify Step by Step | Lightman et al. | 2305.20050 |
| 2023 | Llama 2: Open Foundation and Fine-Tuned Chat Models | Touvron et al. | 2307.09288 |
| 2023 | Mistral 7B | Jiang et al. | 2310.06825 |
| 2023 | SWE-bench: Can Language Models Resolve Real-World GitHub Issues? | Jimenez et al. | 2310.06770 |
| 2023 | Scaling Data-Constrained Language Models | Muennighoff et al. | 2305.16264 |
| 2023 | Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection | Asai et al. | 2310.11511 |
| 2023 | Textbooks Are All You Need | Gunasekar et al. | 2306.11644 |
| 2023 | The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only | Penedo et al. | 2306.01116 |
| 2023 | Time Travel in LLMs: Tracing Data Contamination in Large Language Models | Golchin & Surdeanu | 2308.08493 |
| 2023 | Toolformer: Language Models Can Teach Themselves to Use Tools | Schick et al. | 2302.04761 |
| 2023 | Tree of Thoughts: Deliberate Problem Solving with Large Language Models | Yao et al. | 2305.10601 |
| 2023 | Visual Instruction Tuning | Liu et al. | 2304.08485 |
| 2023 | Voyager: An Open-Ended Embodied Agent with Large Language Models | Wang et al. | 2305.16291 |
| 2023 | YaRN: Efficient Context Window Extension of Large Language Models | Peng et al. | 2309.00071 |
| 2024 | Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference | Chiang et al. | 2403.04132 |
| 2024 | DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models | Shao et al. | 2402.03300 |
| 2024 | Fishing for Magikarp: Automatically Detecting Under-trained Tokens in Large Language Models | Land & Bartolo | 2405.05417 |
| 2024 | Introducing computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku | Anthropic | link |
| 2024 | Introducing the Model Context Protocol | Anthropic | link |
| 2024 | Learning to Reason with LLMs | OpenAI | link |
| 2024 | MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark | Wang et al. | 2406.01574 |
| 2024 | Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters | Snell et al. | 2408.03314 |
| 2024 | The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale | Penedo et al. | 2406.17557 |
| 2024 | The Llama 3 Herd of Models | Grattafiori et al. | 2407.21783 |
| 2025 | DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning | DeepSeek-AI | 2501.12948 |