skip to content

Search
← All concepts

#transformers

The transformer family of architectures.

6 chapters, oldest era first: one idea traced across the books.

  1. The Language Machines 2017 Ch. 5: The Transformer The full QKV mathematics, multi-head attention, and positional encoding — the chapter every other book links to.
  2. The Language Machines 2018 Ch. 6: Pretrain, Then Fine-tune ELMo, ULMFiT, GPT-1, and BERT: masked versus causal objectives.
  3. The Language Machines 2019 Ch. 9: One Model, Many Tasks T5's text-to-text framing and BART — and why decoder-only eventually won.
  4. The Language Machines 2021–2024 Ch. 16: Long Context RoPE's rotation math, ALiBi, interpolation and YaRN — the context-window arms race.
  5. From Noise to Image 2022–2024 Ch. 19: Flow Matching and Transformer Backbones Velocity-field regression, rectified flow, and DiT replacing the U-Net — the SD3/FLUX lineage.
  6. From Frames to Worlds 2024 Ch. 9: Sora and Spacetime Patches Diffusion transformers for video, spacetime patches, and the video-models-as-world-simulators claim.