skip to content

Search
← All concepts

#pretraining

Large-scale self-supervised training.

7 chapters, oldest era first: one idea traced across the books.

  1. The Language Machines 2018 Ch. 6: Pretrain, Then Fine-tune ELMo, ULMFiT, GPT-1, and BERT: masked versus causal objectives.
  2. The Language Machines 2018–2020 Ch. 7: The GPT Lineage Decoder-only scaling from GPT-1 to GPT-3, and in-context learning as an emergent phenomenon.
  3. The Language Machines 2019 Ch. 9: One Model, Many Tasks T5's text-to-text framing and BART — and why decoder-only eventually won.
  4. The Language Machines 2020–2022 Ch. 10: Scaling Laws Power laws, compute-optimal training, the Chinchilla re-balance, and the emergence debate.
  5. The Language Machines 2020–2025 Ch. 11: The Data Question Corpora, dedup, filtering, synthetic data, and contamination — data as the real moat.
  6. From Noise to Image 2021 Ch. 11: CLIP: Bridging Text and Image The InfoNCE contrastive loss, a shared embedding space for words and pictures, and why it unlocked text-to-image.
  7. The Language Machines 2022–2024 Ch. 18: Multimodal Frontier Models Vision encoders meet LLMs: Flamingo, GPT-4, LLaVA, Gemini.