skip to content

Search
← All concepts

#tokenization

Discretizing text/images/video into tokens.

4 chapters, oldest era first: one idea traced across the books.

  1. The Language Machines 2015–2024 Ch. 8: Tokenization The BPE merge algorithm, SentencePiece, glitch tokens, and why tokenizers cause weird failures.
  2. From Noise to Image 2016–2017 Ch. 5: Pixels One at a Time Autoregressive factorization, PixelRNN/PixelCNN, and VQ-VAE's discrete codebooks.
  3. From Noise to Image 2021 Ch. 12: DALL-E and GLIDE Discrete-token text-to-image versus text-conditioned diffusion — the fork in the road.
  4. From Frames to Worlds 2021–2022 Ch. 4: Tokens for Video VQ tokenization plus autoregressive transformers: VideoGPT, CogVideo, and Phenaki's variable-length generation.