skip to content

Search
← All concepts

#language-modeling

Predicting text sequences.

5 chapters, oldest era first: one idea traced across the books.

  1. The Language Machines 1948–2003 Ch. 1: From Shannon to n-grams Language modeling as next-symbol prediction, perplexity, and Bengio's first neural language model.
  2. The Language Machines 1948–2026 Appendix: Timeline and Reading List A one-page dated timeline of language models and every paper cited in this book.
  3. The Language Machines 1997–2015 Ch. 3: RNNs, LSTMs, and seq2seq Gating equations, the bottleneck vector, and machine translation as generation.
  4. The Language Machines 2013–2014 Ch. 2: word2vec and the Geometry of Meaning Skip-gram, negative sampling, and king minus man plus woman.
  5. The Language Machines 2015–2024 Ch. 8: Tokenization The BPE merge algorithm, SentencePiece, glitch tokens, and why tokenizers cause weird failures.