Attention

It is September 2014. Neural machine translation has just become plausible — encoder-decoder networks can turn an English sentence into a French one — but every input, whether five words or fifty, must be squeezed through a single fixed-size vector, and translation quality collapses as sentences grow. A three-author paper posted to arXiv on September 1 proposes letting the decoder look back at the entire input and learn where to look. The mechanism is called attention, and within three years it will not merely assist the recurrent network — it will replace it.

Where We Were

The previous chapter (Recurrent Networks and seq2seq) described the encoder-decoder architecture that made neural machine translation viable. Two papers defined the pattern in 2014. In June, the RNN Encoder-Decoder of Cho et al. (Cho et al., 2014) encoded a source sentence into a fixed-length vector and decoded a target sentence from it, initially as a feature inside a statistical translation pipeline. In September, seq2seq (Sutskever et al., 2014) showed that a pure neural system — deep LSTM encoder, deep LSTM decoder — could stand on its own, reaching a BLEU score of 34.8 on WMT’14 English-to-French against 33.3 for a phrase-based statistical baseline (Sutskever et al., 2014). BLEU, the standard machine-translation metric of the era, scores overlap between system output and human reference translations; higher is better.

Both systems shared a structural commitment: the entire meaning of the source sentence had to pass through one fixed-size vector — the encoder’s final hidden state. Researchers began calling this the bottleneck. Its cost was measurable. A concurrent analysis by Cho et al. (Cho et al., 2014) found that encoder-decoder translation “performs relatively well on short sentences without unknown words, but its performance degrades rapidly as the length of the sentence and the number of unknown words increase.” Sutskever et al. had partially masked the problem with a trick — reversing the order of source words so that the beginning of the source sat close to the beginning of the target, creating short-term dependencies the optimizer could exploit — but a trick that helps the first words of a sentence does nothing for a clause forty tokens deep.

The bottleneck was also conceptually unsatisfying. Statistical machine translation, the incumbent technology, had spent two decades building explicit alignment models: probabilistic tables of which source words produce which target words. The neural encoder-decoder threw that structure away and asked one vector to carry everything. The question hanging over the field in mid-2014 was whether alignment could be brought back — not as a hand-built table, but as something learned end-to-end.

The Breakthrough

Neural Machine Translation by Jointly Learning to Align and Translate — commonly shortened to Bahdanau attention (Bahdanau, Cho & Bengio, 2014) — was submitted to arXiv on September 1, 2014, nine days before the seq2seq paper (Bahdanau et al., 2014). Its central move was to stop asking the encoder to compress the whole sentence.

Instead, the encoder — a bidirectional RNN — produces one annotation vector per source word: a hidden state that summarizes that word in the context of the words around it, reading both left-to-right and right-to-left. The full sentence is now represented as a variable-length sequence of vectors, one per position, rather than a single vector.

The decoder then does something new at every output step. Before emitting the next target word, it scores every source annotation against its own current state — “how relevant is source word $j$ to what I am about to say?” — converts those scores into a probability distribution with a softmax, and takes a weighted average of all the annotations. That weighted average, the context vector, is recomputed fresh for every target word. Translating the verb of a French sentence, the decoder can place most of its weight on the English verb, wherever it sits. The fixed bottleneck is gone: information flows from every source position directly to every decoding step.

Two properties made the mechanism land. First, it is soft and therefore differentiable: rather than making a hard choice of one source word, the model blends all of them with learned weights, so ordinary backpropagation trains the alignment jointly with translation — no separate alignment step, no reinforcement learning. Second, the learned weights are interpretable. The paper’s qualitative analysis showed that the soft alignments “agree well with our intuition”: plotted as a heat map, the attention weights of an English-French model recover word correspondences, including the reorderings between English and French noun phrases, without ever being supervised on alignment.

The results were decisive on exactly the axis where the bottleneck hurt. On WMT’14 English-to-French, trained on 348 million words with a 30,000-word vocabulary, the attention model (called RNNsearch) beat the plain encoder-decoder (RNNencdec) at every sentence length: 21.50 vs 13.93 BLEU for models trained on sentences up to 30 words, and 26.75 vs 17.82 for models trained up to 50 words. More telling than the averages was the shape of the curve: the baseline’s quality “dramatically drops” as sentences lengthen, while RNNsearch “shows no performance deterioration even with sentences of length 50 or more.” The best variant, RNNsearch-50★, reached 36.15 BLEU on sentences without rare words — edging past the phrase-based Moses system’s 35.63 on the same subset, the paper’s basis for claiming performance “comparable to the existing state-of-the-art phrase-based system.”

Eleven months later, Effective Approaches to Attention-based Neural Machine Translation (Luong, Pham & Manning, August 2015) reorganized the design space (Luong et al., 2015). Where Bahdanau et al. had proposed one architecture, Luong et al. treated attention as a family with interchangeable parts: a choice of scoring function (how decoder state and source annotation are compared) and a choice of scope — global attention, which attends over every source position, versus local attention, which first predicts where in the source to look and then attends only within a small window around that point, cutting the per-step cost on long inputs. Their simplifications mattered as much as their taxonomy: a plain dot product between hidden states, with no extra parameters, turned out to be a competitive scoring function — the design the Transformer would later adopt. On WMT’15 English-to-German, local attention delivered a gain of 5.0 BLEU over their non-attentional baseline, and an ensemble of attention variants set a new state of the art at 25.9 BLEU, 1.0 point above the previous best system.

The Mathematics

Notation follows Bahdanau et al. (2014): the encoder reads source words $x_1, \dots, x_{T_x}$ and produces annotations $h_1, \dots, h_{T_x}$; the decoder generates target words $y_i$ with hidden states $s_i$.

The attentional decoder defines each output word’s probability as

$$p(y_i \mid y_1, \dots, y_{i-1}, \mathbf{x}) = g(y_{i-1},\, s_i,\, c_i)$$

Symbol	Meaning
$y_i$	target word being generated at step $i$
$\mathbf{x}$	the full source sentence $x_1, \dots, x_{T_x}$
$g$	output network mapping its inputs to a distribution over the vocabulary
$s_i$	decoder hidden state at step $i$
$c_i$	context vector for step $i$ — recomputed every step

The context vector is a weighted average of all source annotations:

$$c_i = \sum_{j=1}^{T_x} \alpha_{ij}\, h_j$$

Symbol	Meaning
$c_i$	context vector for target step $i$
$\alpha_{ij}$	attention weight on source position $j$ at step $i$
$h_j$	encoder annotation at source position $j$
$T_x$	length of the source sentence

The weights come from a softmax over alignment scores:

$$\alpha_{ij} = \frac{\exp(e_{ij})}{\sum_{k=1}^{T_x} \exp(e_{ik})}$$

Symbol	Meaning
$\alpha_{ij}$	normalized attention weight on source position $j$ at step $i$
$e_{ij}$	raw alignment score for the same pair
$e_{ik}$	raw score for every other source position $k$, summed in the normalizer
$T_x$	source sentence length — number of terms in the sum

Bahdanau et al. compute the score with a small feedforward network, a form now called additive (or concat) attention:

$$e_{ij} = v_a^{\top} \tanh\!\left(W_a\, s_{i-1} + U_a\, h_j\right)$$

Symbol	Meaning
$e_{ij}$	raw alignment score between target step $i$ and source position $j$
$s_{i-1}$	decoder state before emitting word $i$
$h_j$	bidirectional-encoder annotation of source position $j$
$W_a, U_a$	learned projection matrices for decoder state and annotation
$v_a$	learned vector reducing the hidden layer to a scalar score

Luong et al. (2015) compared three content-based scoring functions, using $h_t$ for the current decoder state and $\bar{h}_s$ for a source annotation:

$$\mathrm{score}(h_t, \bar{h}_s) = \begin{cases} h_t^{\top} \bar{h}_s & \text{dot} \\ h_t^{\top} W_a\, \bar{h}_s & \text{general} \\ v_a^{\top} \tanh\!\left(W_a [h_t; \bar{h}_s]\right) & \text{concat} \end{cases}$$

Symbol	Meaning
$h_t$	decoder hidden state at target step $t$ (Luong notation)
$\bar{h}_s$	encoder annotation at source position $s$
$W_a$	learned matrix — bilinear map in general, input projection in concat
$v_a$	learned scoring vector in the concat form
$[\cdot\,;\cdot]$	vector concatenation

Their local-attention variant first predicts a source position and then applies a Gaussian window around it:

$$\begin{aligned} p_t &= S \cdot \sigma\!\left(v_p^{\top} \tanh(W_p\, h_t)\right) \\ a_t(s) &= \mathrm{align}(h_t, \bar{h}_s)\, \exp\!\left(-\frac{(s - p_t)^2}{2\sigma_w^2}\right) \end{aligned}$$

Symbol	Meaning
$p_t$	predicted source position to center attention on, in $[0, S]$
$S$	source sentence length
$v_p, W_p$	learned parameters of the position-prediction network
$a_t(s)$	final attention weight on source position $s$
$\sigma_w$	Gaussian window spread, set to $D/2$ for window radius $D$

One accounting note that matters later: computing $e_{ij}$ for every pair of target step and source position makes the cost of attention proportional to the product of the two sequence lengths. For 2015-scale translation this was negligible next to the recurrent computation; once attention became the whole model, that quadratic term became the field’s central systems problem.

What It Enabled

The most direct payoff was translation quality that survived sentence length, which turned neural machine translation from a research demonstration into deployable infrastructure. GNMT (Wu et al., September 2016), Google’s production neural translation system, used a deep LSTM network with 8 encoder and 8 decoder layers, residual connections, and an attention connection from the bottom decoder layer to the top encoder layer; by human evaluation it reduced translation errors by an average of 60% compared to Google’s phrase-based production system (Wu et al., 2016). Two years separated the arXiv preprint from attention running inside one of the most-used software products on earth.

The mechanism also escaped text almost immediately. Show, Attend and Tell (Xu et al., February 2015) applied the same score-softmax-blend loop to image captioning: the decoder attends over a grid of convolutional image features instead of word annotations, and the attention maps show the model fixating on the object it is currently naming (Xu et al., 2015). The pattern — attention as a general interface between any encoder and any decoder — carried into text-to-image generation (First Words to Pictures) and beyond.

	Fixed-vector encoder-decoder, 2014	Attentional encoder-decoder, 2014–2015
Source representation	one fixed-size vector	one annotation per source position
Information path to decoder	through the bottleneck only	direct weighted access to every position
Long sentences	quality drops rapidly with length	no deterioration observed at length 50+
WMT’14 En-Fr BLEU, 50-word training	17.82 — RNNencdec-50	26.75 — RNNsearch-50
Alignment	implicit, unrecoverable	explicit, inspectable heat maps
Cost per decoder step	constant in source length	grows with source length

The deepest consequence was conceptual. Bahdanau attention demonstrated that a network could learn where to route information as a differentiable part of the model itself, rather than having connectivity fixed by architecture. Luong et al. showed the scoring function could be as simple as a dot product. The step taken by the Transformer (Vaswani et al., June 2017) was to ask what remains if the recurrence is deleted and attention — including attention from the sequence to itself — does all the work (The Transformer). Every chapter after this one is downstream of that question, and this chapter’s machinery — scores, softmax, weighted sum — passes into it unchanged.

Key Papers

Paper	Authors	Year	Link	Why it matters
Neural Machine Translation by Jointly Learning to Align and Translate	Bahdanau, Cho, Bengio	2014	arXiv:1409.0473	Introduces soft attention: learned, differentiable alignment that removes the fixed-vector bottleneck
Effective Approaches to Attention-based Neural Machine Translation	Luong, Pham, Manning	2015	arXiv:1508.04025	Systematizes attention — dot, general, concat scores; global vs local — and shows dot-product scoring suffices
Sequence to Sequence Learning with Neural Networks	Sutskever, Vinyals, Le	2014	arXiv:1409.3215	The pure-neural encoder-decoder whose fixed-vector design attention was built to fix
Learning Phrase Representations using RNN Encoder-Decoder	Cho et al.	2014	arXiv:1406.1078	Establishes the RNN encoder-decoder pattern and the fixed-length sentence vector
On the Properties of Neural Machine Translation	Cho et al.	2014	arXiv:1409.1259	Documents the failure mode — quality degrades rapidly with sentence length — that motivated attention
Show, Attend and Tell	Xu et al.	2015	arXiv:1502.03044	Ports attention to vision, proving it is a general interface between modalities, not a translation trick
Google’s Neural Machine Translation System	Wu et al.	2016	arXiv:1609.08144	Attention at production scale: 8+8-layer attentional LSTMs cut translation errors by an average of 60%