Retrieval and Grounding

It is February 2020. GPT-2 has shown that a large enough transformer can answer trivia it never saw a question form for, simply because the fact was baked into its weights during pretraining. But baking facts into weights has a cost: every new fact requires retraining, every wrong fact requires retraining, and nobody can point to which weight holds which fact. A pair of papers, months apart, propose the alternative — give the model a library card instead of asking it to memorize the library.

Where We Were

By early 2020 the field’s dominant recipe for making a language model “know” things was to make it bigger. Scaling laws were about to formalize the intuition already driving GPT-2 (February 2019) and its successors: more parameters, trained on more text, produced a model that answered more questions correctly. Knowledge was implicit — distributed across billions of weights, extracted only by a forward pass, with no record of which training document contributed which fact.

This had three consequences that were becoming hard to ignore. First, freshness: a model’s knowledge froze at its training cutoff, and updating a single fact meant retraining or fine-tuning the whole network, an absurd price for a one-line correction. Second, provenance: when a model stated something, there was no way to show a source, because there was no source object anywhere in the computation — only weights. Third, scale-inefficiency: fact recall on rare entities behaved like the tail of a power law, so covering more of the long tail of world knowledge meant paying the full cost of scaling laws just to memorize more trivia, rather than paying a much smaller cost to look it up.

Search engines had solved a version of this problem for decades with an inverted index and a ranking function. Question-answering systems built on top of search (retrieve a paragraph, extract a span) worked but were brittle and could not generate fluent, synthesized answers. The unresolved question was architectural: could retrieval be fused into a neural network as a differentiable component, trained end to end with gradient descent, rather than bolted on as a separate preprocessing step?

The Breakthrough

Act one: retrieval as a pretraining objective. REALM (Guu et al., 2020), submitted in February 2020, argued that standard pretraining “store[s] knowledge implicitly in the parameters of a neural network, requiring ever-larger networks to cover more facts.” Its fix was to add a knowledge retriever inside the pretraining loop itself: for each masked-language-modeling example, the model first retrieves a small set of documents from a large corpus (Wikipedia), then attends over them while predicting the masked tokens. The retrieval step uses Maximum Inner Product Search (MIPS) over document embeddings, and — the paper’s central engineering claim — this step is differentiable, so gradients from the language-modeling loss flow back through the retrieval scores and reshape which documents get retrieved for which queries. REALM reported outperforming prior open-domain question-answering methods “by a significant margin (4-16% absolute accuracy),” while also, in the authors’ words, offering “qualitative benefits such as interpretability and modularity” — you could inspect which document the model had pulled up before it answered.

The retriever gets its own paper. Making the retrieval side of these systems work well turned out to be a problem in its own right. Dense Passage Retrieval, or DPR (Karpukhin et al., 2020), submitted two months after REALM, trained a bi-encoder — one encoder for questions, one for passages, both producing vectors in the same space via the embedding primitive introduced for words in chapter 2 — using only question-answer pairs as supervision, with no separate pretraining objective. The result “outperform[ed] a strong Lucene-BM25 system largely by 9%-19% absolute in terms of top-20 passage retrieval accuracy,” and better retrieval accuracy translated directly into better end-task QA accuracy, which is the load-bearing assumption underneath every retrieval-augmented system built afterward: if the retriever finds the wrong passage, the generator cannot recover.

Act two: retrieval as a generation-time lookup. RAG (Lewis et al., 2020), submitted in May 2020 — three months after REALM — moved the fusion point from pretraining to generation. Instead of retraining a masked-language model with retrieval baked in, RAG combined a pretrained DPR-style retriever with a pretrained seq2seq generator (BART), fine-tuning the whole pipeline jointly. The paper defined two ways to combine retrieved passages with generation: RAG-Sequence, which uses the same retrieved passage for every token of an output, and RAG-Token, which can draw on a different passage for each token — useful when an answer synthesizes information from more than one document. RAG models “set the state-of-the-art on three open domain QA tasks, outperforming parametric seq2seq models,” and, notably, generated “more specific, diverse and factual language than a state-of-the-art parametric-only seq2seq baseline” — the first controlled evidence that grounding in retrieved text changed not just correctness but the character of the generated prose.

Act three: retrieval at trillion-token scale, frozen. RETRO (Borgeaud et al., 2021), submitted in December 2021, took the opposite bet from REALM and RAG. Rather than making the retriever trainable and expensive, RETRO froze a BERT-style retriever entirely and pointed it at a database of two trillion tokens — several orders of magnitude larger than a Wikipedia index — then trained only a lightweight, differentiable mechanism for fusing retrieved chunks into the transformer’s internal representations, called chunked cross-attention. The headline result: a RETRO model achieved “comparable performance to GPT-3 and Jurassic-1 on the Pile, despite using 25× fewer parameters.” Retrieval, at sufficient database scale, substituted for parameters — the clearest empirical statement yet that knowledge in an index and knowledge in weights could be economically interchangeable.

The three systems sit on a spectrum of one design choice: how tightly retriever and generator are coupled. REALM trains the retriever as part of pretraining itself. RAG trains a pretrained retriever and generator jointly at fine-tuning time. RETRO freezes the retriever and trains only the fusion mechanism, trading retrieval quality for the ability to scale the index almost without limit. All three share the same underlying claim: that an external, searchable store of text is a legitimate second memory system for a language model, distinct from and complementary to the model’s parameters.

The Mathematics

REALM and RAG share a common probabilistic skeleton: the retrieved document is a latent variable that gets marginalized out.

$$p(y \mid x) = \sum_{z \in \mathcal{Z}} p(y \mid x, z)\, p(z \mid x)$$

Symbol	Meaning
$y$	The generated output — an answer or continuation
$x$	The input query or masked context
$z$	A single retrieved document, treated as a latent variable
$\mathcal{Z}$	The full retrieval corpus, e.g. all of Wikipedia
$p(y \mid x, z)$	The generator’s probability of the output given the query and one document
$p(z \mid x)$	The retriever’s probability of choosing that document

The retriever’s distribution $p(z \mid x)$ is not learned as an arbitrary function — both REALM and DPR compute it as a softmax over inner products between dense vectors, which is what makes MIPS the right search primitive:

$$p(z \mid x) = \frac{\exp\left(f(x)^{\top} g(z)\right)}{\displaystyle\sum_{z' \in \mathcal{Z}} \exp\left(f(x)^{\top} g(z')\right)}$$

Symbol	Meaning
$f(x)$	Query encoder, maps the input to a vector
$g(z)$	Document encoder, maps a candidate document to a vector
$f(x)^{\top} g(z)$	Inner product, the similarity score between query and document
$\mathcal{Z}$	The full retrieval corpus
$z, z'$	A candidate document; $z'$ ranges over all candidates in the normalization

RAG makes the truncation to a top-k list explicit and offers two ways to combine the retrieved set with a sequence of generated tokens rather than a single output:

$$\begin{aligned} p_{\text{RAG-Seq}}(y \mid x) &\approx \sum_{z \in \text{top-}k} p_\eta(z \mid x) \prod_{i=1}^{N} p_\theta\!\left(y_i \mid x, z, y_{1:i-1}\right) \\ p_{\text{RAG-Tok}}(y \mid x) &\approx \prod_{i=1}^{N} \sum_{z \in \text{top-}k} p_\eta(z \mid x)\, p_\theta\!\left(y_i \mid x, z, y_{1:i-1}\right) \end{aligned}$$

Symbol	Meaning
$x$	Input query
$y$, $y_i$	Generated output, and its $i$-th token
$N$	Length of the generated output in tokens
$z$	A retrieved document from the top-k set
$p_\eta(z \mid x)$	Retriever’s probability of a document, with learned parameters $\eta$
$p_\theta(\cdot)$	Generator’s token probability, with learned parameters $\theta$

RETRO does not marginalize over documents at all — it freezes retrieval and fuses retrieved text with ordinary cross-attention, applied between a chunk’s own representations and the encoded text of its retrieved neighbors:

$$\mathrm{CCA}(H, E) = \operatorname{softmax}\!\left(\frac{H E^{\top}}{\sqrt{d}}\right) E$$

What It Enabled

The immediate, measurable effect was that factual accuracy and parameter count decoupled. A model no longer needed to be large to be well-informed; it needed a good retriever and a good index.

System	Retrieval coupling	Index size	Headline result
Parametric-only (GPT-2 / GPT-3 style)	None — all facts in weights	—	Fact recall scales with parameters and compute
REALM (Guu et al., 2020)	Trained jointly with pretraining	Wikipedia	+4 to 16% absolute accuracy on open-domain QA
RAG (Lewis et al., 2020)	Pretrained retriever, fine-tuned jointly	Wikipedia dense index	State-of-the-art on three open-domain QA tasks; more factual generations
RETRO (Borgeaud et al., 2021)	Frozen retriever, trained fusion layer	2 trillion tokens	GPT-3 and Jurassic-1-level performance on the Pile with 25× fewer parameters

This decoupling reframed a whole category of product problem. Knowledge freshness became an indexing problem, not a training problem — add a document to the corpus and the next query can retrieve it, with no gradient update required. Provenance became a byproduct of the architecture rather than a feature that had to be reverse-engineered: because the generator conditions on retrieved passages, a product can simply show the passage it used, which is why “cited sources” became a near-universal UI pattern for retrieval-backed assistants.

But retrieval did not solve hallucination outright, and TruthfulQA (Lin et al., 2021), submitted in September 2021, is why the field could say so with a number instead of an impression. Testing models on questions designed to elicit common misconceptions, the paper found that “the best model was truthful on 58% of questions, while human performance was 94%,” and — the more unsettling finding — “the largest models were generally the least truthful,” a direct contradiction of the scaling-laws intuition that bigger models are strictly better. Larger parametric-only models had absorbed more confidently stated misinformation from their training text, not less. TruthfulQA gave the field a concrete instrument for a claim that had previously been anecdotal, and later evaluation suites built on this instrument are covered in measuring minds and the frontier.

The lineage from these three papers runs in two directions: toward smarter retrieval policies, and toward retrieval as one tool among several for an acting model. Self-RAG (Asai et al., 2023), submitted in October 2023, replaced RAG’s fixed retrieve-every-time policy with a learned one: the model “adaptively retrieves passages on-demand, and generates and reflects on retrieved passages and its own generations using special tokens, called reflection tokens” — deciding for itself when a query needs external grounding and when parametric knowledge is enough, and critiquing its own retrieved evidence before using it. The paper reported that Self-RAG, at 7B and 13B parameters, “outperforms ChatGPT and retrieval-augmented Llama2-chat on Open-domain QA, reasoning and fact verification tasks.” That same on-demand, judged use of an external tool is the general pattern that chapter 21 covers for tool use broadly, and the deliberate, multi-step querying that precedes an answer connects to the reasoning techniques in chapter 19.

Key Papers

Paper	Authors	Year	Link	Why it matters
REALM: Retrieval-Augmented Language Model Pre-Training	Guu et al.	2020	arXiv:2002.08909	First differentiable retrieval fused directly into pretraining
Dense Passage Retrieval for Open-Domain Question Answering	Karpukhin et al.	2020	arXiv:2004.04906	The bi-encoder retriever that made dense retrieval outperform BM25
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks	Lewis et al.	2020	arXiv:2005.11401	Named the pattern; RAG-Sequence and RAG-Token fusion at generation time
TruthfulQA: Measuring How Models Mimic Human Falsehoods	Lin et al.	2021	arXiv:2109.07958	Showed scale alone worsens truthfulness, motivating grounding
Improving Language Models by Retrieving from Trillions of Tokens	Borgeaud et al.	2021	arXiv:2112.04426	Frozen retrieval at trillion-token scale; 25x fewer parameters at equal quality
Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection	Asai et al.	2023	arXiv:2310.11511	Learned, on-demand retrieval with self-critique instead of retrieve-always

PM Lens