#distillation
Compressing teachers into students (knowledge or step).
3 chapters, oldest era first: one idea traced across the books.
- The Engine Room 2015–2025 Ch. 16: Distillation Soft targets and temperature, DistilBERT, and distilling reasoning traces.
- From Noise to Image 2022–2023 Ch. 18: Few-Step Generation Progressive distillation, consistency models, LCMs, and adversarial distillation: 50 steps to 1.
- From Frames to Worlds 2024–2026 Ch. 14: Making Video Cheap Step distillation for video, causal decoders, and the road to real-time generation.