Timeline
Every dated event this book covers, in one place. Each row links to the chapter that teaches it.
| Chapter | Title | Era |
|---|---|---|
| 1 | Why Video Is Hard | 2014–2015 |
| 2 | GAN Video | 2016–2019 |
| 3 | Predicting the Next Frame | 2018–2019 |
| 4 | Tokens for Video | 2021–2022 |
| 5 | Video Diffusion Models | 2022 |
| 6 | Latent Video Diffusion | 2023 |
| 7 | Video-to-Video and Editing | 2023 |
| 8 | Image-to-Video and Motion Control | 2023 |
| 9 | Sora and Spacetime Patches | 2024 |
| 10 | The Commercial Race | 2024–2025 |
| 11 | Open Video Models | 2024–2025 |
| 12 | Long Video and Consistency | 2024 |
| 13 | Sound and Speech | 2020–2025 |
| 14 | Making Video Cheap | 2024–2026 |
| 15 | Video Models Become Game Engines | 2024 |
| 16 | World Models as Foundation Models | 2025–2026 |
| 17 | The Third Dimension | 2020–2023 |
| 18 | Evaluating and Monetizing Motion | Cross-era |
Reading List
Every paper cited across From Frames to Worlds, 52 in total, in chronological order.
| Year | Paper | Authors | Link |
|---|---|---|---|
| 2015 | Deep Multi-Scale Video Prediction Beyond Mean Square Error | Mathieu, Couprie & LeCun | 1511.05440 |
| 2015 | Unsupervised Learning of Video Representations using LSTMs | Srivastava, Mansimov & Salakhutdinov | 1502.04681 |
| 2016 | Generating Videos with Scene Dynamics | Vondrick, Pirsiavash, and Torralba | 1609.02612 |
| 2017 | MoCoGAN: Decomposing Motion and Content for Video Generation | Tulyakov, Liu, Yang, and Kautz | 1707.04993 |
| 2018 | Learning Latent Dynamics for Planning from Pixels | Hafner et al. | 1811.04551 |
| 2018 | Towards Accurate Generative Models of Video: A New Metric & Challenges | Unterthiner, van Steenkiste, Kurach, Marinier, Michalski, and Gelly | 1812.01717 |
| 2018 | World Models | Ha & Schmidhuber | 1803.10122 |
| 2019 | Adversarial Video Generation on Complex Datasets | Clark, Donahue, and Simonyan | 1907.06571 |
| 2019 | Dream to Control: Learning Behaviors by Latent Imagination | Hafner et al. | 1912.01603 |
| 2020 | A Lip Sync Expert Is All You Need for Speech to Lip Generation In the Wild | Prajwal et al. | 2008.10010 |
| 2020 | NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis | Mildenhall et al. | 2003.08934 |
| 2021 | VideoGPT: Video Generation using VQ-VAE and Transformers | Yan et al. | 2104.10157 |
| 2022 | CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformers | Hong et al. | 2205.15868 |
| 2022 | DreamFusion: Text-to-3D using 2D Diffusion | Poole et al. | 2209.14988 |
| 2022 | Imagen Video: High Definition Video Generation with Diffusion Models | Ho, Chan, Saharia, Whang, Gao, Gritsenko, Kingma, Poole, Norouzi, Fleet & Salimans | 2210.02303 |
| 2022 | Make-A-Video: Text-to-Video Generation without Text-Video Data | Singer et al. | 2209.14792 |
| 2022 | Phenaki: Variable Length Video Generation From Open Domain Textual Description | Villegas et al. | 2210.02399 |
| 2022 | Scalable Diffusion Models with Transformers | Peebles & Xie | 2212.09748 |
| 2022 | Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation | Wu et al. | 2212.11565 |
| 2022 | Video Diffusion Models | Ho, Salimans, Gritsenko, Chan, Norouzi & Fleet | 2204.03458 |
| 2023 | 3D Gaussian Splatting for Real-Time Radiance Field Rendering | Kerbl et al. | 2308.04079 |
| 2023 | Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models | Blattmann, Rombach, Ling, Dockhorn, Kim, Fidler & Kreis | 2304.08818 |
| 2023 | AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning | Guo et al. | 2307.04725 |
| 2023 | DragNUWA: Fine-grained Control in Video Generation by Integrating Text, Image, and Trajectory | Yin et al. | 2308.08089 |
| 2023 | EvalCrafter: Benchmarking and Evaluating Large Video Generation Models | Liu, Cun, Liu, Wang, Zhang, Chen, Liu, Zeng, Chan, and Shan | 2310.11440 |
| 2023 | Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inference | Luo, Tan, Huang, Li, and Zhao | 2310.04378 |
| 2023 | MotionCtrl: A Unified and Flexible Motion Controller for Video Generation | Wang et al. | 2312.03641 |
| 2023 | Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets | Blattmann et al. | 2311.15127 |
| 2023 | Structure and Content-Guided Video Synthesis with Diffusion Models | Esser et al. | 2302.03011 |
| 2023 | Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators | Khachatryan et al. | 2303.13439 |
| 2023 | VBench: Comprehensive Benchmark Suite for Video Generative Models | Huang, He, Yu, Zhang, Si, Jiang, Zhang, Wu, Jin, Chanpaisit, Wang, Chen, Wang, Lin, Qiao, and Liu | 2311.17982 |
| 2024 | CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer | Yang et al. | 2408.06072 |
| 2024 | Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusion | Chen et al. | 2407.01392 |
| 2024 | Diffusion Models Are Real-Time Game Engines | Valevski, Leviathan, Arar, and Fruchter | 2408.14837 |
| 2024 | Diffusion for World Modeling: Visual Details Matter in Atari | Alonso et al. | 2405.12399 |
| 2024 | From Slow Bidirectional to Fast Autoregressive Video Diffusion Models | Yin et al. | 2412.07772 |
| 2024 | Genie 2: A large-scale foundation world model | Google DeepMind | link |
| 2024 | Genie: Generative Interactive Environments | Bruce et al. | 2402.15391 |
| 2024 | HunyuanVideo: A Systematic Framework For Large Video Generative Models | Kong et al. | 2412.03603 |
| 2024 | LTX-Video: Realtime Video Latent Diffusion | HaCohen et al. | 2501.00103 |
| 2024 | Movie Gen: A Cast of Media Foundation Models | Polyak et al. | 2410.13720 |
| 2024 | Oasis: A Universe in a Transformer | Decart and Etched | link |
| 2024 | StreamingT2V: Consistent, Dynamic, and Extendable Long Video Generation from Text | Henschel et al. | 2403.14773 |
| 2024 | Video generation models as world simulators | OpenAI | link |
| 2024 | VideoPhy: Evaluating Physical Commonsense for Video Generation | Bansal, Lin, Xie, Zong, Yarom, Bitton, Jiang, Sun, Chang, and Grover | 2406.03520 |
| 2025 | Autoregressive Adversarial Post-Training for Real-Time Interactive Video Generation | Lin, Yang, He, Jiang, Ren, Xia, Zhao, Xiao, and Jiang | 2506.09350 |
| 2025 | Cosmos World Foundation Model Platform for Physical AI | NVIDIA | 2501.03575 |
| 2025 | Diffusion Adversarial Post-Training for One-Step Video Generation | Lin, Xia, Ren, Yang, Xiao, and Jiang | 2501.08316 |
| 2025 | Do generative video models understand physical principles? | Motamed, Culp, Swersky, Jaini, and Geirhos | 2501.09038 |
| 2025 | Genie 3: A new frontier for world models | Google DeepMind | link |
| 2025 | V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning | Assran et al. | 2506.09985 |
| 2025 | Veo 3 Technical Report | Google DeepMind | link |