skip to content

Search
← All concepts

#attention

Attention mechanisms and their variants.

7 chapters, oldest era first: one idea traced across the books.

  1. The Language Machines 2014–2015 Ch. 4: Attention Alignment scores and softmax weighting — the fix for the bottleneck, and the seed of everything after.
  2. From Noise to Image 2015–2018 Ch. 7: First Words to Pictures Text conditioning before diffusion: StackGAN, AttnGAN, and attention over words.
  3. The Language Machines 2017 Ch. 5: The Transformer The full QKV mathematics, multi-head attention, and positional encoding — the chapter every other book links to.
  4. The Engine Room 2020–2025 Ch. 18: Beyond Attention Linear attention, S4 and Mamba's selective state spaces, RWKV, hybrids — and why attention persists.
  5. The Language Machines 2021–2024 Ch. 16: Long Context RoPE's rotation math, ALiBi, interpolation and YaRN — the context-window arms race.
  6. The Engine Room 2022–2024 Ch. 12: FlashAttention and the Kernel Era Online softmax, tiling, and IO-complexity: software catching up to silicon.
  7. The Engine Room 2022–2025 Ch. 17: Inference Engines The KV-cache memory formula, multi-query and grouped-query attention, PagedAttention's block-based memory management, speculative decoding, and continuous batching.