Reasoning Models

It is March 2022. A language model can be prompted into writing out its reasoning step by step, but nothing in its training ever told it that some chains of reasoning are better than others — the model that writes a correct proof and the model that writes a plausible-sounding wrong one were trained identically. Over the next three years, three separate teams answer the same question — can a model be trained, not just prompted, to reason? — and by January 2025 the answer ships as an open-weights model that a hedge-fund-adjacent Chinese lab trained for a reported fraction of a frontier lab’s budget.

Where We Were

Chain-of-thought prompting, submitted in January 2022, had already shown that a large enough pretrained model produces better answers on multi-step problems if it is asked to show its work first. That result was purely an inference-time trick: no parameter of the model changed, and the technique only worked because pretraining on internet text happened to expose the model to enough step-by-step explanations that it could imitate the pattern when prompted correctly. Self-consistency and tree-of-thoughts, covered in the same chapter, spent more inference compute exploring multiple reasoning paths, but the underlying policy — what the model actually tended to write — was fixed at whatever pretraining and instruction tuning had produced.

That left a gap that prompting alone could not close. A model prompted to “think step by step” was still sampling from the same distribution it always sampled from; if that distribution rarely produced a correct multi-step derivation, no amount of clever prompt wording manufactured one. RLHF had already shown, starting in 2022, that a reward signal could reshape a model’s output distribution — but the reward models built for helpfulness and harmlessness were trained on human preference judgments between two free-form answers, not on whether a forty-step algebra derivation was actually correct. Math, code, and logic problems have something friendlier available: an answer can often be checked mechanically, by a calculator, a unit test, or a symbolic equality check, without any human in the loop. Nobody had yet built the training loop that turned mechanical checking into a learning signal at scale.

The other constraint was economic. Scaling laws had, since Kaplan (2020) and Chinchilla (2022), treated model size and pretraining compute as the dial that bought capability. That dial was getting expensive to turn — frontier pretraining runs were already costing tens of millions of dollars — and it was not obvious that a bigger model would reason any better per dollar than a same-size model that was simply allowed to think longer before answering. The alternative axis, spending more compute at inference time rather than at training time, was unexplored territory.

Three years separate a purely inference-time prompting trick from an open-weights model trained by reinforcement learning to reason on its own. PNG ↓

The Breakthrough

STaR: bootstrap rationales from the model’s own successes. “STaR: Bootstrapping Reasoning With Reasoning” (Zelikman et al., 2022), submitted in March 2022, was the first to close the loop between generating a rationale and training on it. Starting from a handful of hand-written examples, a GPT-J-based model is prompted to generate a chain-of-thought rationale followed by an answer for each question in a training set. Where the answer is correct, the rationale is kept, on the assumption that a rationale leading to a right answer is more likely to be a sound one than a rationale leading to a wrong one. Where the answer is wrong, STaR does not discard the example — it rationalizes: the model is re-prompted with the correct answer given as a hint and asked to produce a rationale that would justify it, working backward from the answer to a plausible derivation. Every rationale that survives either path, correct-by-construction or correct-with-a-hint, becomes fine-tuning data, and the whole process repeats on the newly fine-tuned model. On CommonsenseQA, STaR closed most of the gap between direct fine-tuning and a model roughly 30 times larger, reaching 72.5% accuracy against the larger model’s roughly 73%, using only its own generated rationales as supervision.

A wrong answer is not discarded, it is rationalised backward from the correct answer, which is why STaR needs no human-written explanations at all. PNG ↓

Process supervision: grade the steps, not just the answer. STaR’s filter — is the final answer right? — is a blunt instrument: a rationale can stumble through two wrong steps and still land on the correct number by luck, and STaR’s fine-tuning set cannot tell the difference. “Let’s Verify Step by Step” (Lightman et al., 2023), submitted in May 2023, tested whether grading every intermediate step — process supervision — beat grading only the final answer — outcome supervision. OpenAI’s team collected PRM800K: 800,000 step-level correctness labels from human annotators, spanning 75,000 model-generated solutions to 12,000 problems from the MATH dataset, with active learning used to prioritize labeling “convincing” wrong-answer solutions — ones a partially trained reward model rated highly despite their final answer being incorrect, on the logic that these are the solutions most likely to reveal what the reward model still gets wrong. A process reward model (PRM) trained on these labels, used to rerank 1,860 candidate solutions per problem, solved 78% of a representative MATH test subset against 72.4% for an outcome-supervised reward model evaluated the same way — though the paper is explicit that the two training sets differ in scale and are not a fully controlled comparison; a separate small-scale ablation, matched for data and compute, confirmed process supervision’s advantage held regardless.

Test-time compute as its own scaling axis. “Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters” (Snell et al., 2024), submitted in August 2024, formalized what PRM-guided search was already doing informally: treat the compute spent at inference — number of samples, depth of search, number of self-revisions — as a budget to be allocated, not a fixed afterthought. The paper’s central empirical claim was that the best allocation strategy depends heavily on how hard the prompt is, and that a strategy which adapts to prompt difficulty lets a small base model, given enough test-time compute, match or beat a model up to roughly 14 times larger evaluated at matched total FLOPs — a result engine-room’s chapter on test-time-compute economics covers from the cost side.

o1: fold search and verification into the policy itself. OpenAI’s o1 (OpenAI, 2024), previewed in ChatGPT on September 12, 2024, took the logic of the previous three papers and trained it directly into a single model rather than bolting search onto a frozen one. o1 is trained with large-scale reinforcement learning to produce an extended, model-generated chain of thought before answering; the chain of thought itself is not shown to end users, only a summary is. OpenAI reported that on the 2024 AIME competition, o1-preview solved 83% of problems against GPT-4o’s 13%, and reached the 89th percentile on competitive-programming problems from Codeforces — evidence that RL against a model’s own reasoning traces, not just a bigger pretrained backbone, was producing the gain. The training method itself was not disclosed in technical detail.

DeepSeek-R1: reasoning from reward alone, and no secret sauce. “DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning” (DeepSeek-AI, 2025), submitted January 22, 2025, published what o1 had not: a full recipe, and an open-weights model. The paper’s first result, DeepSeek-R1-Zero, applied reinforcement learning directly to the pretrained base checkpoint (DeepSeek-V3-Base, not the instruction-tuned DeepSeek-V3) with no supervised fine-tuning step at all, using GRPO — group-relative policy optimization, introduced the previous year and already the concept-owning chapter’s subject — against a purely rule-based reward. Behaviors the field associates with “thinking,” including self-verification, backtracking, and re-planning mid-solution, emerged from this loop without ever being demonstrated to the model. R1-Zero reached 71.0% on AIME 2024, comparable to OpenAI’s o1-0912 checkpoint at 74.4%, but its reasoning traces mixed languages and were often hard for a person to read. DeepSeek-R1 fixed that with a four-stage pipeline: a cold-start supervised fine-tune on a small set of curated, human-readable long chains of thought; a reasoning-focused RL stage with the same rule-based rewards plus a language-consistency penalty; a rejection-sampling round that generated roughly 600,000 reasoning and 200,000 general-purpose examples for a second supervised fine-tune; and a final RL stage blending reasoning correctness with general helpfulness and harmlessness signals. The resulting model reached 79.8% on AIME 2024 and 97.3% on MATH-500, edging out o1-1217’s 79.2% and matching its MATH-500 score, and reached the 96.3rd percentile on Codeforces. The team distilled the same reasoning behavior into dense Qwen and Llama checkpoints from 1.5B to 70B parameters — a 32B Qwen distillation reached 72.6% on AIME 2024, and a 70B Llama distillation reached 70.0%, both well above the undistilled checkpoints they started from.

R1-Zero proves the reasoning behaviour emerges from RL alone; the four-stage pipeline exists only to make that same behaviour readable and well-rounded. PNG ↓

The Mathematics

STaR’s training signal is the standard next-token loss, restricted to a filtered set of self-generated rationales. Given a dataset $D$ of question–answer pairs $(x, y^*)$ , let $M(x)$ denote a rationale-and-answer sample $(r, y)$ drawn from the current model, and $M(x, y^*)$ the same sampling process given the true answer as a hint (the rationalization step):

\mathcal{L}_{\text{STaR}}(\theta) = -\, \mathbb{E}_{(x,y^*) \sim D} \sum_{\substack{(r,y) \,\in\, \{M(x)\} \,\cup\, \{M(x,y^*)\} \\ y = y^*}} \log p_\theta(r, y \mid x)

For every question, the model first generates a rationale cold. If the resulting answer is right, that rationale-and-answer pair is kept. If it’s wrong, the model gets a second attempt on that same question with the correct answer whispered to it as a hint, and tries to work backward to a plausible-looking derivation; only hinted attempts that now land on the right answer are kept, and a question that still fails even with the hint contributes nothing to training. The model is then fine-tuned to reproduce every surviving rationale-and-answer pair, cold and hinted alike. What this buys you is a training signal for reasoning quality without ever collecting a single human-written explanation — the filter is “did the reasoning arrive somewhere true,” which is available for free wherever answers can be checked.

Symbol	Meaning
$D$	Dataset of question–answer pairs $(x, y^*)$
$M(x)$	Rationale-and-answer sample from the current model, no hint
$M(x, y^*)$	Rationale-and-answer sample generated with the true answer as a hint
$(r, y)$	A sampled rationale $r$ and its resulting answer $y$
$\theta$	Parameters of the model being fine-tuned

Process supervision replaces the outcome-only filter with a learned per-step judge. Given a solution broken into steps $s_1, \dots, s_k$ , the PRM outputs a predicted probability that each step is correct, and is trained against human step-level labels $y_i \in \{0, 1\}$ :

\mathcal{L}_{\text{PRM}}(\phi) = -\sum_{i=1}^{k} \Big[\, y_i \log p_\phi(s_i \mid s_{<i}) + (1 - y_i) \log\big(1 - p_\phi(s_i \mid s_{<i})\big) \,\Big]

This is ordinary binary cross-entropy, just applied at every reasoning step instead of once at the end. Each step gets its own correct-or-not label from a human annotator, and the reward model learns to predict that label from the steps seen so far. What this buys you is a verifier with a finer error signal: a solution that goes wrong at step three can be flagged at step three, rather than only being told “the final number was wrong” and left to guess which of forty steps caused it. At inference time, the score for a full candidate solution is the product of the model’s predicted correctness probability at every one of its steps — one bad step pulls the whole solution’s score down — and the highest-scoring candidate among several sampled solutions is kept, the search-and-verify pattern that “Let’s Verify Step by Step” and later test-time-compute work both build on.

Symbol	Meaning
$s_1, \dots, s_k$	The $k$ steps of a candidate solution
$p_\phi(s_i \mid s_{<i})$	PRM’s predicted probability that step $i$ is correct, given prior steps
$y_i$	Human-provided correctness label for step $i$
$\phi$	Parameters of the process reward model

Test-time scaling gives that search-and-verify intuition a formal target. Let $\mathrm{Target}(\theta, N, q)$ denote the distribution over final answers produced by applying a test-time strategy — best-of- $N$ sampling, a revision chain, or PRM-guided search — with hyperparameters $\theta$ and a compute budget of $N$ , to a question $q$ . The compute-optimal strategy for that question is:

\theta^*(N, q) = \arg\max_{\theta} \; \mathbb{E}_{y \sim \mathrm{Target}(\theta, N, q)} \big[\, \mathbb{1}[\, y = y^*(q) \,] \,\big]

Symbol	Meaning
$\mathrm{Target}(\theta, N, q)$	Output distribution induced by a test-time strategy $\theta$ with budget $N$ on question $q$
$\theta$	Test-time strategy hyperparameters, e.g. how compute is split between sampling and revision
$N$	Test-time compute budget, e.g. number of samples or revision steps
$y^*(q)$	The ground-truth correct answer to question $q$

DeepSeek-R1-Zero replaces both the human step labels above and any learned reward model with a reward computed by a fixed, deterministic function — the design the field converged on calling reinforcement learning from verifiable rewards. For a sampled response $o$ to a question with known answer $y^*$ :

r(o) = \mathbb{1}\big[\, \mathrm{verify}(o) = y^* \,\big] \; + \; \mathbb{1}\big[\, \mathrm{format}(o) \,\big]

The first term checks the final answer mechanically — does the boxed math answer match, does the generated code pass its test cases — and awards one point if so, zero otherwise. The second term is a bonus, weighted equally with the first, for wrapping the reasoning trace in the expected tags before the answer, which keeps the output structured enough to be read and trained on downstream. Neither term is a neural network, and that omission is deliberate: DeepSeek-AI reported that learned reward models, whether judging outcomes or steps, were vulnerable to reward hacking once RL training ran long enough, and retraining them to patch each exploit added cost without adding reliability. A reward that a compiler or an answer-checker computes cannot be gamed by the policy learning to please it, because it never learns to please anything — it only learns to be correct. R1’s policy update itself uses GRPO’s group-relative advantage estimate against $r(o)$ , which chapter 15 derives in full.

What It Enabled

The shift from prompting-time reasoning to training-time reasoning changed what a “better” model meant. Before STaR, capability gains came almost entirely from a bigger pretrained backbone; after DeepSeek-R1, a fixed-size model’s score on a hard benchmark became something you could move with reinforcement learning alone, holding parameter count constant.

	Prompted reasoning (2022)	Process-supervised search (2023)	RL-trained reasoning (2024–2025)
Where the gain comes from	Better prompts, same weights	A learned verifier reranks sampled solutions	The policy itself is optimized against a reward
Reward source	None — inference-time only	Human-labeled PRM800K, 800K step labels	Rule-based verifier, mechanically checked
Extra inference cost	One forward pass per sample	Many samples plus a reward-model pass	One pass, but a much longer chain of thought
Representative result	Chain-of-thought lifts arithmetic accuracy	78% of a MATH subset with best-of-1,860	79.8% AIME 2024 for DeepSeek-R1
Reward-hacking risk	Low, nothing is optimized against	Present — reported in DeepSeek-R1’s ablations	Reduced by using deterministic checks

The benchmark trajectory across the anchor models makes the same point numerically:

Model	Date	AIME 2024 pass@1	MATH-500	Codeforces percentile
GPT-4o (baseline)	2024-05	13%	—	—
o1-preview	2024-09	83%	—	89th
DeepSeek-R1-Zero	2025-01	71.0%	—	—
DeepSeek-R1	2025-01	79.8%	97.3%	96.3rd
DeepSeek-R1-Distill-Qwen-32B	2025-01	72.6%	94.3%	—

Three threads run forward from this chapter. First, once reasoning could be trained rather than only prompted, it became a component to compose with other capabilities rather than a standalone trick — agents that plan multi-step tool use lean directly on RL-trained reasoning to decide what to do next, not just what to say next. Second, the arrival of an open-weights model matching a closed frontier reasoner on math and code benchmarks, at API prices roughly an order of magnitude below o1’s, reopened the build-versus-buy calculus the field had settled around GPT-4-class models — a shift chapter 22 tracks through the benchmarks that increasingly need replacing as models saturate them. Third, DeepSeek’s distillation results — a 32B dense model matching much of a 671B-parameter mixture-of-experts teacher’s reasoning ability — reinforced that RL-trained reasoning traces are themselves a transferable resource, extending the toolkit chapter 16 of the engine room covers to a new kind of teacher signal: not just output logits, but entire chains of thought.

Two independent lines, bootstrapped self-training and verifier-guided search, converge on the first RL-trained reasoners before the field forks again into agents and cheap distilled models. PNG ↓

Key Papers

Paper	Authors	Year	Link	Why it matters
STaR: Bootstrapping Reasoning With Reasoning	Zelikman et al.	2022	arXiv:2203.14465	First closed loop: filter self-generated rationales by correctness, fine-tune on the survivors
Let’s Verify Step by Step	Lightman et al.	2023	arXiv:2305.20050	PRM800K and process supervision beat outcome supervision on MATH, 78% with reranking
Learning to Reason with LLMs	OpenAI	2024	OpenAI	o1: RL-trained hidden chain of thought, 83% AIME vs GPT-4o’s 13%
Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters	Snell et al.	2024	arXiv:2408.03314	Formalized inference compute as an allocatable, prompt-adaptive budget
DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning	DeepSeek-AI	2025	arXiv:2501.12948	Open recipe and weights: pure-RL reasoning, rule-based rewards, distillation to 1.5B–70B

PM Lens

The number I keep coming back to is the price gap: o1 launched at $15 per million input tokens and $60 per million output tokens, and DeepSeek-R1 shipped weeks later at roughly $0.55 and $2.19 — open weights, comparable AIME and MATH-500 scores, an order of magnitude cheaper. That is not a marginal optimization; it changes which products a reasoning model is worth putting behind. Before R1, “let it think for 30 seconds” was a premium-tier feature I’d gate behind a paid plan. After it, treating reasoning as a per-request toggle — cheap model for routing and formatting, reasoning model only for the step that actually needs multi-hop logic — became the default architecture I’d sketch for any agentic feature. The distillation results matter just as much operationally: a 32B distilled checkpoint I can self-host beats plenty of un-distilled 70B+ models on math and code, which means “do we need the frontier reasoner or a distilled one” is now a real line item in a build plan, not a hypothetical.

Search