Measuring Minds and the Frontier

It is September 2020. GPT-3 has spent four months showing that a single model can draft legal boilerplate, sketch code, and answer trivia across wildly different domains, but the field still grades it with narrow, single-task leaderboards built for much smaller models. A 57-subject exam changes that in one paper. Over the next four years the industry builds, saturates, and grows suspicious of three successive generations of benchmark — an exam, a scorecard, an arena — each engineered to patch the blind spot of the one before it.

Where We Were

Through the late 2010s, progress on language models was tracked with task-specific leaderboards: one score for sentiment classification, another for textual entailment, another for question answering, each a separate contest with its own training set. These leaderboards worked when models were narrow specialists fine-tuned per task. They stopped working the moment a single model could attempt dozens of unrelated tasks from a prompt alone.

GPT-3 (Brown et al., 2020, chapter 7) forced the issue. Its headline result was breadth: few-shot performance across translation, arithmetic, and reading comprehension without any task-specific fine-tuning. But breadth has no natural scoreboard. The field’s oldest yardstick, perplexity (chapter 1), measured how surprised a model was by held-out text — a proxy for fluency, not for whether the model actually knew that the mitochondria is the powerhouse of the cell or how a lien differs from a mortgage. There was no standard way to ask, in one number, “how much does this model know, across how many domains, compared to a human?”

The gap mattered commercially as well as scientifically. Labs were about to make the multi-million-dollar bets described in scaling laws, and a bet needs a scoreboard. Without one, “our model is better” was a claim nobody outside the lab could check, and every research paper reporting a new state of the art was implicitly grading itself.

There was also no agreed answer to a more basic question: evaluated how? Some tasks reward the model for producing the single most likely next token; others require sampling and voting; others depend on exactly how the question is phrased. Two labs reporting “our model gets 80% on task X” could be running two different procedures and comparing incomparable numbers, a problem that persisted well after MMLU arrived and one that later benchmarks tried explicitly to standardize away.

The Breakthrough

Act one: the exam. “Measuring Massive Multitask Language Understanding” (Hendrycks et al., 2020), submitted in September 2020, built a single instrument for breadth: 57 subjects spanning “elementary mathematics, US history, computer science, law, and more,” assembled into 15,908 four-way multiple-choice questions. Chance accuracy on a four-option question is 25%. GPT-3’s largest model reached 43.9% in the five-shot setting — “almost 20 percentage points” above chance, proof that scale bought broad knowledge, not just fluency. The paper anchored that score against two human baselines: unspecialized crowdworkers on Amazon Mechanical Turk scored 34.5%, while the authors estimated expert-level accuracy at roughly 89.8% by extrapolating from real professional-exam pass rates. GPT-3 landed between the two, and unevenly — “almost 70% accuracy for its best subject but near-random performance for several other subjects,” a lopsidedness no single headline number could show. The paper’s other finding was a warning rather than a scoreboard: on every one of the 57 subjects the best models remained well short of expert-level accuracy, and models were poorly calibrated — GPT-3’s confidence diverged from its actual accuracy by “up to 24%” on some subjects, meaning it did not reliably know what it did not know. MMLU gave the field its first common exam, its first human benchmarks to compare against, and its first evidence of how far there was to go.

An exam has a structural weakness: it only asks multiple-choice questions and reports one score. It says nothing about how a model behaves off the exam paper — how consistent it is under paraphrased questions, how biased its outputs are, how expensive it is to run, or how well its confidence tracks its correctness.

Act two: the scorecard. “Holistic Evaluation of Language Models” (Liang et al., 2022), submitted in November 2022, answered that weakness directly. Rather than one score, HELM evaluated 30 prominent language models across 42 scenarios — 16 “core” scenarios plus 26 scenarios targeting specific concerns such as bias, toxicity, and disinformation — and reported seven metrics per scenario wherever it was applicable to do so, which the authors note was possible 87.5% of the time: “accuracy, calibration, robustness, fairness, bias, toxicity, and efficiency.” The point was not to crown a single winner but to make every model’s trade-offs visible in the same table: a model could top accuracy and still be flagged for toxicity, or be cheap to run and still be miscalibrated. HELM turned evaluation from a single leaderboard into a standardized, multi-dimensional public record, refreshed as new models arrived.

HELM was still an offline instrument: fixed scenarios, run once, scored automatically. It could not capture what millions of people were about to start doing with language models in the open, which is judge them conversationally, in the moment, against each other.

Act three: the arena. ChatGPT’s public launch on November 30, 2022 (chapter 14) put open-ended, multi-turn conversation in front of a mass audience, and users judged answers the way people judge answers — by preferring one over another, not by grading against an answer key. “Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference” (Chiang et al., 2024), submitted in March 2024 after the platform had been “operational for several months,” formalized that instinct into an instrument: visitors chat blind with two randomly paired, anonymized models and vote for the better response. By the time of publication the platform had collected “over 240K votes.” No exam question, no fixed metric, and no expert grader in the loop — just a running, crowdsourced tally of which model ordinary users actually preferred when neither model’s identity was revealed.

The Mathematics

MMLU reports one headline number by averaging accuracy uniformly across its 57 subjects, not across all questions pooled together:

$$\text{Acc}_{\text{MMLU}} = \frac{1}{|S|}\sum_{s \in S}\left(\frac{1}{N_s}\sum_{i=1}^{N_s}\mathbb{1}\left[\hat y_{s,i}=y_{s,i}\right]\right)$$

Symbol	Meaning
$S$	Set of the 57 MMLU subjects
$N_s$	Number of questions in subject $s$
$y_{s,i}$	Correct answer to question $i$ in subject $s$
$\hat y_{s,i}$	Model’s predicted answer
$\mathbb{1}[\cdot]$	Indicator: 1 if correct, else 0

Chatbot Arena turns hundreds of thousands of pairwise human votes into a single per-model rating the same way competitive chess does: fit each model’s strength once, by maximum likelihood over the entire vote history (a Bradley-Terry model), then rescale onto a familiar ratings axis:

$$P(i \succ j) = \frac{1}{1+10^{-(R_i - R_j)/400}}$$

Symbol	Meaning
$R_i, R_j$	Fitted Elo-style rating of model $i$ and model $j$
$P(i \succ j)$	Predicted probability model $i$ beats model $j$
$i, j$	The two models in a given head-to-head battle

What It Enabled

The clearest downstream effect is a treadmill: every benchmark that becomes a target eventually gets run down. GPT-4 (OpenAI, 2023, submitted March 15, 2023 — full model discussed in chapter 18) scored 86.4% on MMLU against GPT-3.5’s 70.0%, less than three years after GPT-3’s 43.9% had been the headline result. All three figures were evaluated the same way — five-shot, the exact protocol MMLU’s authors specified in 2020 — so the jump is not an artifact of a friendlier test procedure; it is the same exam, taken by three successive generations of model. As scores from frontier models clustered near the top of a 100-point scale, the exam stopped separating good models from great ones — a problem the field calls benchmark saturation.

Model	MMLU accuracy	Source
GPT-3 175B, five-shot	43.9%	Hendrycks et al., 2020
GPT-3.5	70.0%	OpenAI, 2023
GPT-4	86.4%	OpenAI, 2023

Two responses followed directly from that saturation. “GPQA: A Graduate-Level Google-Proof Q&A Benchmark” (Rein et al., 2023), submitted November 2023, built 448 multiple-choice questions in biology, physics, and chemistry written by domain experts, deliberately hard enough that “highly skilled non-expert validators” with unrestricted web access for 30 minutes reached only 34% accuracy, while PhD-level experts in the relevant field reached 65% (74% discounting their own flagged mistakes). The strongest GPT-4-based baseline the authors tested reached 39% — well below the human-expert floor, restoring the headroom MMLU had lost. The “Google-proof” in the name is a literal design constraint, not marketing: the benchmark’s difficulty was validated against searchers with a browser open, not against a closed-book exam, which is why a model that has merely memorized web text should not be able to game it. “MMLU-Pro” (Wang et al., 2024), submitted June 2024, took the more direct route of hardening MMLU itself: harder, more reasoning-dependent questions with ten answer choices instead of four cut top-model accuracy by “16% to 33%” relative to the original test and cut prompt-format sensitivity from 4–5% down to 2%, making scores both harder to max out and more reproducible.

A second response targeted a different suspicion: that public benchmarks were no longer clean measures of ability, because their questions had leaked into training data. “Time Travel in LLMs” (Golchin & Surdeanu, 2023), submitted August 2023, proposed “guided instruction” prompting — feeding a model a dataset’s name and the opening fragment of a known example, then checking whether it completes the rest verbatim. The method detected contamination with 92–100% accuracy across seven datasets when checked against human judgment, and using it the authors found evidence that GPT-4 had been contaminated on three public datasets, including AG News and WNLI. A high benchmark score bought from a model that had already seen the test is not evidence of capability; contamination-detection tooling exists precisely because that distinction is easy to fake and hard to audit from outside a lab.

The same pressure that produced harder multiple-choice exams also pushed evaluation past multiple-choice entirely. Reasoning benchmarks that grade multi-step derivations rather than single answers are the subject of prompting becomes reasoning and reasoning models; benchmarks that grade whether an agent actually resolves a GitHub issue or completes a multi-tool task, such as SWE-bench, are covered in agents. None of those can be scored by matching a letter to an answer key — they require running code, checking a final state, or, as with Chatbot Arena, asking a human.

Put side by side, the three instruments this chapter opened with are not competitors so much as a division of labor, and every later chapter in this book that reports a number is implicitly borrowing one of them:

Instrument	Published	Format	What it measures	Known limit
MMLU	Sep 2020	57-subject multiple choice	Breadth of factual and procedural knowledge	Saturates; contamination-prone; no open-ended quality signal
HELM	Nov 2022	42 scenarios × 7 metrics	Accuracy alongside calibration, robustness, fairness, bias, toxicity, efficiency	Offline and fixed; cannot capture live conversational preference
Chatbot Arena	Mar 2024	Anonymous pairwise human votes	Human preference between two live responses	Rewards confident style over verifiable correctness

None of the three alone tells a team whether a model is safe, cheap, or correct enough to ship — that is why production evaluation, as the PM Lens below describes, still ends with a private test set the public leaderboards have never seen.

Key Papers

Paper	Authors	Year	Link	Why it matters
Measuring Massive Multitask Language Understanding	Hendrycks et al.	2020	arXiv:2009.03300	First broad, multi-domain exam; exposed the gap between fluency and knowledge
Holistic Evaluation of Language Models	Liang et al.	2022	arXiv:2211.09110	Standardized multi-metric, multi-scenario evaluation across 30 models
Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference	Chiang et al.	2024	arXiv:2403.04132	Crowdsourced pairwise human-preference ranking via a Bradley-Terry model
GPQA: A Graduate-Level Google-Proof Q&A Benchmark	Rein et al.	2023	arXiv:2311.12022	Expert-written, web-search-resistant exam built once MMLU began saturating
MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark	Wang et al.	2024	arXiv:2406.01574	Hardened successor to MMLU; cut top-model scores and prompt sensitivity
Time Travel in LLMs: Tracing Data Contamination in Large Language Models	Golchin & Surdeanu	2023	arXiv:2308.08493	Guided-instruction method for detecting benchmark leakage into training data
GPT-4 Technical Report	OpenAI	2023	arXiv:2303.08774	Documented the MMLU jump (70.0% to 86.4%) that made saturation concrete

PM Lens