The Transformer

It is June 2017. Machine translation is the proving ground of neural sequence modeling, and every competitive system is a recurrent network reading one token at a time. Eight researchers at Google propose deleting the recurrence and keeping only the attention mechanism — and the resulting architecture, the Transformer (Vaswani et al., 2017), becomes the substrate for essentially every large model that follows, in language and far beyond it.

Where We Were

By early 2017 the dominant recipe for translation was the encoder–decoder recurrent network: an LSTM read the source sentence into hidden states, and a second LSTM generated the target sentence, with an attention mechanism letting the decoder consult the encoder’s states at every step. The pieces had arrived in sequence — LSTMs in November 1997, seq2seq in September 2014, and learned soft alignment the same month with Bahdanau attention (Bahdanau et al., 2014). Chapter 3 covers the recurrent lineage; Chapter 4 covers alignment-era attention in detail.

The recipe worked, but it had a structural flaw that no amount of tuning could remove: recurrence is sequential. To compute the hidden state at position $t$, the network must first compute the hidden state at position $t-1$. A sentence of $n$ tokens requires $n$ dependent steps, and no parallel processor can collapse a dependency chain. This mattered doubly. First, for training throughput: GPUs are massively parallel devices, and a computation that serializes along the sequence leaves most of the chip idle. Second, for learning itself: information from token 1 reaching token 100 had to survive 99 state updates, and even gated architectures degraded over such paths.

Attention had already shown the escape hatch. When the decoder attended over encoder states, it connected any output position to any input position in a single step — no chain of updates. But in 2014–2016 attention was an accessory bolted onto a recurrent trunk. The question the Transformer answered was whether the accessory could be the entire vehicle.

The Breakthrough

“Attention Is All You Need” (Vaswani et al., 2017), submitted to arXiv in June 2017, proposed a sequence-to-sequence architecture built entirely from attention layers and position-wise feed-forward layers, with no recurrence and no convolution. The central move is self-attention: instead of attending from a decoder to an encoder, every token in a sequence attends to every other token in the same sequence. Each layer lets each position gather information from the whole sequence at once, so the path between any two tokens is one hop, and every position is computed simultaneously.

The architecture keeps the familiar encoder–decoder shape. The encoder is a stack of 6 identical layers; each layer contains a multi-head self-attention sublayer and a position-wise feed-forward sublayer. The decoder is also a stack of 6 layers, with three sublayers each: masked self-attention over the tokens generated so far, cross-attention over the encoder’s output, and the same feed-forward network. The mask in the decoder’s self-attention zeroes out connections to future positions, so the model can be trained on whole sequences in parallel while still generating left-to-right — a property that later becomes the foundation of causal language-model pretraining, covered in Chapter 6.

Each sublayer is wrapped in a residual connection followed by layer normalization — the residual idea comes from ResNet-style deep-network training (engine room, Chapter 1), and LayerNorm (Ba et al., 2016) supplies per-example normalization that, unlike batch normalization, does not depend on batch statistics; the normalization mathematics itself is taught in engine room, Chapter 6. Because attention is a weighted average with no notion of order, the model injects position information by adding fixed sinusoidal positional encodings to the token embeddings at the input.

Two design choices did the quiet work. Multi-head attention runs several attention functions in parallel over lower-dimensional projections, so different heads can specialize — one tracking syntax-like adjacency, another long-distance agreement — where a single attention distribution would have to average them together. And the entire layer is built from dense matrix multiplications, exactly the operation contemporary accelerators are best at.

The empirical result was decisive on the standard WMT 2014 benchmarks. The big Transformer reached 28.4 BLEU on English-to-German — more than 2 BLEU above the previous best results, including ensembles — and 41.8 BLEU on English-to-French, a new single-model state of the art. The base model (65M parameters) trained in about 12 hours on 8 NVIDIA P100 GPUs; the big model (213M parameters) trained in 3.5 days on the same 8 GPUs, a small fraction of the training cost of the recurrent and convolutional systems it displaced.

The Mathematics

The Transformer’s core primitive is scaled dot-product attention. Every token’s embedding is linearly projected into three roles: a query (what am I looking for?), a key (what do I contain, as advertised to others?), and a value (what do I actually pass along if selected?). Attention scores every query against every key, normalizes the scores into weights, and returns a weighted average of the values.

$$\mathrm{Attention}(Q, K, V) = \mathrm{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right)V$$

Symbol	Meaning
$Q$	query matrix, one row per token, shape $n \times d_k$
$K$	key matrix, one row per token, shape $n \times d_k$
$V$	value matrix, one row per token, shape $n \times d_v$
$d_k$	dimension of queries and keys (64 in the paper)
$\mathrm{softmax}$	row-wise normalization of scores into weights summing to 1
$QK^\top$	the $n \times n$ matrix of all pairwise relevance scores

The $\sqrt{d_k}$ scaling is not cosmetic. If query and key components behave like independent random variables with unit variance, their dot product has variance $d_k$ — so for large $d_k$ the raw scores grow large, the softmax saturates toward a one-hot distribution, and gradients through it vanish. Dividing by $\sqrt{d_k}$ restores unit variance and keeps the softmax in its trainable regime.

A single attention distribution must average all the relationships a token participates in. Multi-head attention instead projects the inputs $h$ times with different learned matrices, runs attention in each projected subspace, and concatenates the results:

$$\begin{aligned} \mathrm{MultiHead}(Q, K, V) &= \mathrm{Concat}(\mathrm{head}_1, \ldots, \mathrm{head}_h)\,W^O \\ \mathrm{head}_i &= \mathrm{Attention}(Q W_i^Q,\; K W_i^K,\; V W_i^V) \end{aligned}$$

Symbol	Meaning
$h$	number of heads (8 in the base model)
$\mathrm{head}_i$	output of the $i$-th attention head, dimension $d_v = 64$
$W_i^Q, W_i^K, W_i^V$	per-head projection matrices, $d_{\mathrm{model}} \times d_k$
$W^O$	output projection mixing the concatenated heads back to $d_{\mathrm{model}}$
$d_{\mathrm{model}}$	the model’s working width (512 in the base model)

In the base configuration $d_{\mathrm{model}} = 512$, $h = 8$, and $d_k = d_v = 64$, so the eight 64-dimensional heads together match the model width. The big model doubles the width to 1024 with 16 heads.

Between attention sublayers sits a position-wise feed-forward network, applied identically and independently to every token:

$$\mathrm{FFN}(x) = \max(0,\; xW_1 + b_1)\,W_2 + b_2$$

Symbol	Meaning
$x$	one token’s representation, dimension $d_{\mathrm{model}} = 512$
$W_1, b_1$	expansion weights and bias, mapping 512 → 2048
$W_2, b_2$	contraction weights and bias, mapping 2048 → 512
$\max(0, \cdot)$	ReLU nonlinearity, zeroing negative components

Every sublayer — attention or feed-forward — is wrapped the same way:

$$\mathrm{output} = \mathrm{LayerNorm}\big(x + \mathrm{Sublayer}(x)\big)$$

Finally, because nothing so far distinguishes “the dog bit the man” from “the man bit the dog” — attention is permutation-invariant — the model adds sinusoidal positional encodings to the input embeddings:

$$\begin{aligned} PE_{(pos,\,2i)} &= \sin\!\left(pos / 10000^{2i/d_{\mathrm{model}}}\right) \\ PE_{(pos,\,2i+1)} &= \cos\!\left(pos / 10000^{2i/d_{\mathrm{model}}}\right) \end{aligned}$$

Symbol	Meaning
$pos$	the token’s position in the sequence, 0, 1, 2, …
$i$	index of the dimension pair within the encoding vector
$d_{\mathrm{model}}$	embedding width, 512, so 256 sine–cosine pairs
$10000$	base of the geometric wavelength progression

Positional encoding turned out to be the least settled part of the design: learned positions, rotary embeddings, and attention biases all followed, and Chapter 16 traces that lineage. The model was trained with the Adam optimizer under a warmup-then-decay learning-rate schedule (optimizer math in engine room, Chapter 5), with dropout of 0.1 (0.3 for the big model on English-to-German) and label smoothing of 0.1, on BPE-style subword vocabularies — subword tokenization is taught in Chapter 8.

What It Enabled

The immediate win was translation quality per unit of compute. The structural win was bigger: the Transformer decoupled sequence length from sequential computation.

	Recurrent seq2seq with attention	Transformer
Sequential operations per layer	$O(n)$ — one step per token	$O(1)$ — all positions at once
Maximum path between two tokens	$O(n)$ state updates	1 attention hop
Per-layer complexity	$O(n \cdot d^2)$	$O(n^2 \cdot d)$
Training parallelism	poor — serialized along sequence	full — batched matrix multiplies
WMT14 EN-DE BLEU (2017)	over 2 BLEU worse, including ensembles	28.4 (big model)
Training cost (best model)	much higher — the Transformer trained at a small fraction of it	3.5 days on 8 P100 GPUs

Note the complexity trade in row three: the Transformer is worse than recurrence when the sequence length $n$ exceeds the representation width $d$, because every token attends to every token. In 2017, with sentences of a few dozen tokens and $d_{\mathrm{model}} = 512$, the trade was overwhelmingly favorable. At the 100K-token contexts of later years, the quadratic term became the industry’s central systems problem — driving IO-aware kernels (engine room, Chapter 12), KV-cache serving economics (engine room, Chapter 17), and the long-context methods of Chapter 16.

Because the architecture parallelized cleanly, it scaled — and the field promptly took it apart and scaled each piece. Within eighteen months of the paper, two of the three canonical body plans had emerged: keep only the decoder and pretrain causally (GPT-1, June 2018); keep only the encoder and pretrain bidirectionally (BERT, October 2018). The third followed a year later: keep both and cast every task as text-to-text (T5, October 2019). Chapter 6 covers the pretraining split, Chapter 7 follows the decoder-only line through GPT-2 (February 2019) and GPT-3 (May 2020), and Chapter 9 covers the text-to-text consolidation. The architecture’s uniformity — the same layer repeated $N$ times, with a handful of width and depth knobs — is also what made the scaling-law studies of Chapter 10 possible: you cannot fit clean power laws to a menagerie, but you can to one architecture family.

The export beyond language was just as consequential. Attention over token sequences does not care what the tokens denote, so images, audio, video, protein residues, and game states all became “sequences” once tokenized. Within image generation, the transformer backbone eventually displaced the U-Net in diffusion models — the DiT line of December 2022, taught in Image Generation, Chapter 19. This chapter is the reference point: when any other chapter in these four books says “a transformer,” it means the machinery defined above.

Key Papers

Paper	Authors	Year	Link	Why it matters
Attention Is All You Need	Vaswani et al.	2017	arXiv:1706.03762	Defines the Transformer — scaled dot-product attention, multi-head attention, positional encoding — the architecture behind essentially all subsequent large models
Neural Machine Translation by Jointly Learning to Align and Translate	Bahdanau et al.	2014	arXiv:1409.0473	The alignment-era attention mechanism the Transformer generalized and made self-sufficient
Layer Normalization	Ba et al.	2016	arXiv:1607.06450	Per-example normalization that made deep attention stacks trainable; used in every Transformer sublayer

PM Lens