Long Context

It is April 2021. Every deployed Transformer still leans on the fix Chapter 5 introduced in 2017: a table of position vectors, one slot per index, with a hard ceiling at whatever length training used. A paper out of Zhuiyi Technology proposes rotating queries and keys instead of adding anything to them, and by late 2023 that single change — plus a companion idea from Ofir Press four months later, and two rescaling tricks the following summer — turns context length from a fixed architectural constant into a dial vendors keep turning further.

Where We Were

The original Transformer added fixed sinusoidal vectors to token embeddings at the input, one function of position per dimension pair. BERT, GPT-1, and GPT-2 replaced that fixed scheme with learned absolute position embeddings — a trainable row for each index from 0 up to a chosen maximum, n_ctx. GPT-2, released February 2019, set that maximum at 1024 tokens; GPT-3, released May 2020, doubled it to 2048. Both numbers were architectural constants, not measurements: they described how many rows existed in an embedding table, nothing more.

Learned absolute embeddings made the length ceiling literal rather than statistical. A sinusoidal function is defined at every position, so a model trained on short sequences can in principle be fed a longer one — the numbers are just unfamiliar. A learned embedding table has no row for position 2048 if it was only ever trained up to 2047; there is nothing to look up. Even where a model could technically accept a longer input, attention patterns tuned to a narrow range of position indices did not generalize past it: perplexity on held-out text degraded sharply the moment a sequence crossed the training length, a failure mode the field came to call poor extrapolation.

This was not a cosmetic limitation. Document summarization, multi-turn dialogue, retrieval-augmented prompts, and codebase-scale reasoning all wanted more than a couple thousand tokens of working context, and the naive fix — retrain from scratch at a longer length — meant redoing the entire, increasingly expensive pretraining run. The field needed a position scheme that was cheap to extend after the fact, which meant rethinking what “position” should encode in the first place.

The Breakthrough

Two independent proposals in 2021 attacked the position problem at its root by dropping the idea of an added position vector entirely. Two more, in the summer of 2023, showed how to stretch an already-trained model past its original length without retraining it from zero.

RoFormer, introducing Rotary Position Embedding, or RoPE (Su et al., 2021), encodes position as a rotation applied to the query and key vectors inside attention, rather than as a vector added to the token embedding beforehand. Each query and key is split into pairs of dimensions, and each pair is rotated by an angle proportional to the token’s position. Rotation is a multiplicative, not additive, operation, and it has a property additive position vectors lack: the dot product between a rotated query and a rotated key depends only on the difference between their positions, not on either position individually. Attention becomes sensitive to relative offset by construction, and the resulting model — RoFormer — was reported to consistently outperform its alternatives on long-text classification benchmarks. RoPE’s inter-token dependency also decays smoothly with increasing relative distance, giving attention a mild built-in locality bias without discarding long-range connections outright.

ALiBi, Attention with Linear Biases (Press et al., 2021), goes further and removes position embeddings from the model altogether. Instead, it adds a static, non-learned penalty directly to the attention scores before the softmax: a query at position $i$ attending to a key at position $j$ has its raw dot product reduced by an amount proportional to the distance $i - j$, with the proportionality constant — the slope — fixed per head. Because the bias is a simple arithmetic function of distance rather than a lookup into a bounded table, it is defined at any distance, including ones the model never saw in training. Press et al. trained a 1.3-billion-parameter model on sequences of 1024 tokens and showed it extrapolated to 2048-token sequences at inference, matching the perplexity of a sinusoidal-position model trained directly on 2048-token sequences — while training 11% faster and using 11% less memory, since no position vectors need to be learned or stored at all.

By early 2023, RoPE had become the default position scheme in a new generation of open models, including LLaMA (Chapter 14) at a 2048-token training length. This created a new, narrower problem: how do you take a model already pretrained with RoPE at some fixed length and push it further, without paying for pretraining again?

Position Interpolation (Chen et al., 2023) answered by working against the same instinct that motivated ALiBi’s extrapolation: rather than feeding RoPE unfamiliar large position indices, it linearly compresses the new, longer range of indices down to fit inside the numeric range the model was originally trained on. Extrapolating past the trained range, the authors showed, can push attention scores to values far outside anything the softmax saw during training; interpolating keeps every index inside familiar territory, at the cost of finer, unfamiliar spacing between adjacent positions — a gap a short round of fine-tuning closes quickly. The method extended RoPE-based LLaMA models, from 7B to 65B parameters, to context windows of up to 32,768 tokens with under 1,000 fine-tuning steps.

YaRN, Yet another RoPE extensioN method (Peng et al., 2023), refined the same idea two months later. Instead of compressing every rotation frequency by the same factor, YaRN treats each frequency differently: high-frequency dimensions, which complete many rotation cycles within the original training length, are left to extrapolate, since the model already saw a full range of their angles; low-frequency dimensions, which barely complete a cycle, are interpolated, since stretching them further would push the model into genuinely unfamiliar territory. A second adjustment rescales the attention softmax’s effective temperature to compensate for the redistribution this blend causes. The combination let YaRN match or beat Position Interpolation’s results using roughly ten times fewer training tokens and 2.5 times fewer training steps, and open follow-on releases used it to push LLaMA 2 to context windows as large as 128,000 tokens using on the order of 0.1% of the model’s original pretraining-corpus size in fine-tuning data.

The Mathematics

RoPE rotates each 2D slice of a query or key vector by an angle proportional to its position. For one dimension pair with base frequency $\theta$:

$$R(m\theta) = \begin{pmatrix} \cos m\theta & -\sin m\theta \\ \sin m\theta & \cos m\theta \end{pmatrix}$$

A full query or key vector of dimension $d$ is split into $d/2$ such pairs, each rotated at its own frequency $\theta_i = 10000^{-2i/d}$ for $i = 0, \ldots, d/2 - 1$ — the same geometric progression of frequencies the original Transformer used for its sinusoidal encoding, repurposed here as rotation angles rather than added values. Applying the full block-diagonal rotation $R_{\Theta,m}$ to a projected query or key gives:

$$q_m = R_{\Theta,m}\, W_Q x_m, \qquad k_n = R_{\Theta,n}\, W_K x_n$$

Symbol	Meaning
$x_m, x_n$	token embeddings at positions $m$ and $n$
$W_Q, W_K$	query and key projection matrices
$R_{\Theta,m}$	block-diagonal rotation matrix at position $m$
$\theta_i$	rotation frequency for dimension pair $i$
$m, n$	absolute position indices

Because rotations compose — rotating by $m\theta$ then by $n\theta$ in the opposite sense leaves a rotation by $(n-m)\theta$ — the dot product between a rotated query and a rotated key reduces to a function of their offset alone:

$$q_m^\top k_n = x_m^\top W_Q^\top R_{\Theta,\,n-m}\, W_K\, x_n$$

ALiBi skips rotation and instead subtracts a distance penalty directly from the attention logits, before the softmax:

$$\mathrm{score}(q_i, k_j) = q_i^\top k_j \;-\; m_h\,(i - j), \qquad j \le i$$

Symbol	Meaning
$q_i, k_j$	query at position $i$, key at position $j$
$i - j$	distance between query and key positions
$m_h$	fixed, head-specific slope
$h$	attention head index

Position Interpolation leaves RoPE’s mechanics untouched and instead rescales the position index fed into it:

$$m' = \frac{L}{L'}\, m$$

Symbol	Meaning
$m$	true position index in the extended sequence
$m'$	rescaled index passed to RoPE
$L$	original trained maximum length
$L'$	target extended length, $L' > L$

YaRN applies that same compression selectively, per rotation frequency, instead of uniformly across all of them:

$$\theta_i' = (1 - \gamma_i)\,\frac{\theta_i}{s} + \gamma_i\,\theta_i, \qquad s = \frac{L'}{L}$$

Symbol	Meaning
$\theta_i, \theta_i'$	original and adjusted rotation frequency for pair $i$
$s$	extension ratio, $L'/L$
$\gamma_i$	per-dimension blend weight between extrapolation and interpolation
$L, L'$	original and extended maximum length

YaRN pairs this per-dimension blend with a temperature adjustment on the attention softmax, scaling logits by a factor found empirically as a function of $s$, to keep the sharpness of attention comparable to the model’s pretrained regime even as the effective density of positions changes. Su et al.’s decay property noted above is what makes both interpolation methods viable at all: because RoPE’s attention weight naturally tapers with distance, compressing distant positions closer together perturbs a part of the score the model already discounted.

What It Enabled

The four methods did not converge on a single winner; they split into a scheme built into pretraining (RoPE, and ALiBi as its no-embedding alternative) and a pair of post-hoc extension tricks (Position Interpolation, YaRN) applied to models already trained with RoPE. Context windows across the industry grew accordingly, in steps tied to whichever technique a given lab adopted.

Model or method	Date	Position scheme	Max context
Original Transformer	2017-06	Sinusoidal absolute	trained-length only, no extrapolation
GPT-2	2019-02	Learned absolute	1,024 tokens
GPT-3	2020-05	Learned absolute	2,048 tokens
LLaMA	2023-02	RoPE	2,048 tokens
GPT-4	2023-03	undisclosed	8,192 tokens standard, 32,768 tokens in a limited variant
LLaMA + Position Interpolation	2023-06	RoPE, linearly interpolated	32,768 tokens
Claude (Anthropic)	2023-05	undisclosed	100,000 tokens
Llama 2	2023-07	RoPE	4,096 tokens
Llama 2 + YaRN (community extension)	2023-08	RoPE, NTK-aware interpolation	up to 128,000 tokens
GPT-4 Turbo	2023-11	undisclosed	128,000 tokens

Two things stand out. First, the jump from thousands to tens of thousands of tokens took only a few months once interpolation-based extension existed, because it decoupled context length from the cost of pretraining — a lab could ship a longer-context variant of a model it had already paid to train once. Second, several of the largest context windows in the table belong to models whose position scheme was never published; RoPE, ALiBi, and their extensions are the methods this chapter can verify, not necessarily the only ones in production.

Longer context did not remove the case for retrieval. Feeding an entire corpus into a prompt is bounded by the quadratic cost of attention over that many tokens — the same $O(n^2)$ term Chapter 5 flags as the Transformer’s structural weakness at scale — while retrieving only the relevant passages keeps the prompt short regardless of corpus size. Chapter 17 covers that alternative, and in practice the two composed rather than competed: longer context windows made it more forgiving to retrieve a few extra, possibly irrelevant, passages, since the model had room to ignore them.

Key Papers

Paper	Authors	Year	Link	Why it matters
RoFormer: Enhanced Transformer with Rotary Position Embedding	Su et al.	2021	arXiv:2104.09864	Introduces RoPE — rotating queries and keys so attention scores depend only on relative position; became the default scheme in LLaMA and most open models that followed
Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation	Press et al.	2021	arXiv:2108.12409	Introduces ALiBi — a distance-proportional bias on attention scores, removing position embeddings entirely and enabling extrapolation past the trained length
Extending Context Window of Large Language Models via Positional Interpolation	Chen et al.	2023	arXiv:2306.15595	Shows that compressing new position indices into the originally trained range, plus brief fine-tuning, extends RoPE-based LLaMA models to 32,768 tokens
YaRN: Efficient Context Window Extension of Large Language Models	Peng et al.	2023	arXiv:2309.00071	Refines interpolation per rotation frequency plus an attention-temperature correction, matching prior extension methods with 10x fewer tokens and 2.5x fewer training steps

PM Lens