skip to content

Search
← All concepts

#multimodal

Models spanning text/image/video/audio.

6 chapters, oldest era first: one idea traced across the books.

  1. From Frames to Worlds 2020–2025 Ch. 13: Sound and Speech Lip sync, video-to-audio, and natively audiovisual generation.
  2. From Noise to Image 2021 Ch. 11: CLIP: Bridging Text and Image The InfoNCE contrastive loss, a shared embedding space for words and pictures, and why it unlocked text-to-image.
  3. From Noise to Image 2021–2023 Ch. 17: Editing SDEdit, attention-map surgery, instruction-based editing, and inpainting.
  4. The Language Machines 2022–2024 Ch. 18: Multimodal Frontier Models Vision encoders meet LLMs: Flamingo, GPT-4, LLaVA, Gemini.
  5. From Noise to Image 2024–2026 Ch. 20: The Current Frontier Autoregression strikes back: VAR, native multimodal generation inside LLMs, and instruction-following images.
  6. From Frames to Worlds 2025–2026 Ch. 16: World Models as Foundation Models Genie 2 and 3, Cosmos, V-JEPA 2 — robotics and embodied simulation as the endgame.