Prompting Becomes Reasoning · The Language Machines • Hoshank Ailani

It is January 2022. GPT-3 (Brown et al., 2020) has spent nearly two years proving that a frozen language model can translate, summarize, and answer trivia if you show it a few examples in the prompt. Ask the same model a two-step arithmetic word problem, though, and it produces a fluent, confident, wrong answer at almost the same rate whether it is 1.5 billion or 175 billion parameters. Over the next sixteen months, three papers show that the fix is not a bigger model — it is a different prompt.

Where We Were

By 2020, in-context learning — the discovery, taught in The GPT Lineage, that a frozen pretrained model can adapt to a new task from a handful of demonstrations written into its input — had become the default way to use a large language model. GPT-3 showed this worked well for tasks that were essentially pattern completion: translation, sentiment classification, question answering against a passage. The prompt supplied a few input-output pairs, and the model extended the pattern.

Multi-step reasoning behaved differently. GPT-3’s own paper reported that its few-shot performance on two- and three-digit arithmetic was strong, but tasks that required chaining several inferences together — word problems, multi-hop logic — scaled much less cleanly with model size. The standard prompting format was itself part of the problem: a few-shot exemplar consisted of a question immediately followed by its final answer, so the model was being trained, in-context, to jump straight from problem to solution. Nothing in the prompt modeled the intermediate work a person would do with pencil and paper — carrying a remainder, tracking a running total, eliminating a case. Google’s contemporaneous large models, including a 137-billion-parameter dialogue model called LaMDA and an internal 540-billion-parameter system later published as PaLM (Chowdhery et al., 2022 — introduced in Scaling Laws), showed the same pattern: bigger did not mean better at multi-step arithmetic, because scale alone doesn’t create a place in the prompt for intermediate computation to happen.

The Kaplan- and Chinchilla-era scaling literature (Scaling Laws) was, at the same moment, cataloguing capabilities that appeared only past a certain parameter threshold — later formalized as emergent abilities. Multi-step reasoning would turn out to be one of them, but only once the prompt format itself changed to expose the reasoning.

Nothing in the standard prompt format modeled the intermediate work a person would do with pencil and paper, which is why bigger models kept failing the same arithmetic problems. PNG ↓

The Breakthrough

Chain-of-thought prompting (Wei et al., 2022), submitted in January 2022, made one change to the few-shot exemplar: instead of question -> answer, each demonstration became question -> reasoning steps in natural language -> answer. Nothing about the model’s weights or architecture changed. The exemplars simply modeled showing the work.

The initial results, evaluated on GPT-3 and on LaMDA-137B, were consistent but modest: on the GSM8K benchmark of grade-school math word problems, LaMDA-137B improved from 6.5% accuracy with standard few-shot prompting to 14.3% with chain-of-thought prompting; on the simpler MultiArith benchmark it jumped from 7.6% to 44.9%. The paper’s more consequential finding was about where the effect appeared: chain-of-thought prompting helped only at sufficient model scale — smaller models produced fluent but logically broken chains of reasoning that hurt accuracy relative to standard prompting — and the benefit grew with scale rather than shrinking. As the paper’s evaluation extended to PaLM’s full 540-billion-parameter model, the gap widened dramatically: 8-shot chain-of-thought prompting lifted GSM8K accuracy from 17.9% to 56.9%, a result that made the technique impossible to dismiss as a curiosity of one model family. Reasoning ability had been latent in the pretrained network the whole time; the prompt format was the switch that turned it on.

Self-consistency (Wang et al., 2022), submitted two months later in March 2022, targeted a specific weakness in that setup: a single chain-of-thought path is generated by greedy, deterministic decoding, so one arithmetic slip or one wrong case split derails the entire answer with no recovery. The fix borrowed an old idea from ensemble methods — sample, don’t just decode once. Draw several independent reasoning paths from the model at nonzero temperature, discard the reasoning text, and keep only each path’s final answer. Whichever answer shows up most often across the sampled paths wins. Applied to PaLM-540B on GSM8K, this raised accuracy from 56.5% with a single chain-of-thought path to 74.4% by sampling and voting across forty independently generated paths, and the paper reported the same direction of improvement — smaller in absolute terms, but consistent — across arithmetic benchmarks such as SVAMP and AQuA and commonsense benchmarks such as StrategyQA, indicating the technique was not specific to grade-school math.

Tree of Thoughts (Yao et al., 2023), submitted in May 2023, generalized both ideas into an explicit search. Chain-of-thought commits to one linear sequence of reasoning steps; self-consistency runs several linear sequences and votes only at the very end. Neither can look at a partial line of reasoning, judge it weak, and abandon it early — both must run every sampled chain to completion before any evaluation happens. Tree of Thoughts instead treats a partial solution as an explicit search state, generates several candidate next steps from each state, uses the same language model to score how promising each partial state looks, and keeps only the most promising states before continuing — allowing the search to backtrack away from a branch that a chain-of-thought decode would have been stuck inside until the end. The paper tested this on three tasks chosen specifically because they punish committing early: the arithmetic puzzle Game of 24, a 5x5 mini crossword, and a creative-writing task requiring a coherent passage that ends with four independently specified sentences — problems where a locally reasonable first move can make the rest of the problem unsolvable.

Each strategy asks more of the model before it commits to an answer, and on this task only explicit search closes most of the gap. PNG ↓

The Mathematics

The self-consistency paper gives the cleanest formal description of what chain-of-thought prompting is doing, and its notation is reused here for both techniques. A standard prompt asks the model for $p_\theta(a \mid x)$ directly, where $x$ is the question and $a$ is the final answer, drawn from a fixed answer set. Chain-of-thought prompting instead couples the answer to an intermediate reasoning path $r$ — a sequence of natural-language tokens generated before the answer — so that answering the question is equivalent to marginalizing over every reasoning path that could produce it:

p_\theta(a \mid x) = \sum_{r} p_\theta(r \mid x)\, p_\theta(a \mid x, r)

Symbol	Meaning
$x$	The question or problem, as written in the prompt
$r$	A reasoning path — a sequence of generated tokens describing intermediate steps
$a$	The final answer, drawn from a fixed answer set
$p_\theta$	The frozen pretrained language model, with parameters $\theta$ unchanged from ordinary next-token prediction

Greedy decoding approximates the sum in that equation with a single sampled $r$ , which throws away all the probability mass on every other reasoning path the model might have produced. Self-consistency instead samples $m$ reasoning paths independently at nonzero temperature and takes a majority vote over their final answers:

\hat{a} = \arg\max_{a} \sum_{i=1}^{m} \mathbb{1}\!\left(a_i = a\right), \qquad (r_i, a_i) \sim p_\theta(\cdot \mid x)

Symbol	Meaning
$m$	Number of independently sampled reasoning paths (40 in the paper’s main GSM8K result)
$(r_i, a_i)$	The $i$ -th sampled reasoning path and the final answer it produces
$\mathbb{1}(\cdot)$	Indicator function: 1 if the condition holds, 0 otherwise
$\hat{a}$	The answer chosen by majority vote across all $m$ samples

Tree of Thoughts replaces the flat list of independent samples with an explicit search over partial solutions. Carrying forward this chapter’s $r$ for a reasoning step in place of the paper’s own $z$ , a state $s = [x, r_{1 \cdots i}]$ is the question plus the sequence of thoughts generated so far. At each step, a thought generator $G(p_\theta, s, k)$ proposes up to $k$ candidate next thoughts from every state kept from the previous round, and a state evaluator $V(p_\theta, s)$ — the same language model, prompted to judge its own partial work — scores every candidate:

\begin{aligned} S_t' &= \bigcup_{s \,\in\, S_{t-1}} G(p_\theta, s, k) \\ S_t &= \underset{S \,\subset\, S_t',\; |S| = b}{\arg\max} \sum_{s \,\in\, S} V(p_\theta, s) \end{aligned}

Symbol	Meaning
$s = [x, r_{1\cdots i}]$	A search state: the question plus the partial sequence of thoughts generated so far
$S_t$	The set of states kept after step $t$ , sized by the breadth limit $b$
$G(p_\theta, s, k)$	The thought generator: proposes up to $k$ candidate next thoughts from state $s$
$V(p_\theta, s)$	The state evaluator: the model’s own self-assessed score for how promising $s$ looks
$b$	The breadth limit — how many states survive each round of search

What It Enabled

The immediate effect was that reasoning benchmarks stopped being a wall and became a knob. Tree of Thoughts’ own comparison on the Game of 24 — given four numbers, combine them with arithmetic to reach 24 — makes the escalation concrete with GPT-4 as the fixed backbone model:

Method	Description	Game of 24 success rate
Standard prompting	Direct answer, no intermediate steps	7.3%
Chain-of-thought	One sampled reasoning path	4.0%
Chain-of-thought + self-consistency	100 sampled paths, majority vote	9.0%
Tree of Thoughts, breadth 1	Search, one state kept per step	45%
Tree of Thoughts, breadth 5	Search, five states kept per step	74%

The GSM8K progression tells the same story with a different backbone. PaLM-540B moved from 17.9% with standard prompting, to 56.9% with chain-of-thought, to 74.4% with chain-of-thought plus self-consistency at 40 samples — three prompting strategies, the same frozen weights, a four-fold accuracy gain. Self-consistency’s gain was not confined to grade-school arithmetic; PaLM-540B improved on every benchmark the paper tested, with the size of the gain tracking how much a task benefited from exploring more than one line of reasoning:

Benchmark	Chain-of-thought	Chain-of-thought + self-consistency
AQuA (algebra word problems)	35.8%	48.3%
SVAMP (arithmetic word problems)	79.0%	86.6%
GSM8K (grade-school math)	56.5%	74.4%
StrategyQA (multi-hop commonsense)	75.3%	81.6%
ARC-challenge (science QA)	85.2%	88.7%

Tree of Thoughts showed the same escalation on tasks where the Game of 24 numbers were the largest jump but not the only one: on a 5x5 mini crossword, GPT-4 with standard prompting solved 0% of games outright and chain-of-thought solved 1%, while Tree of Thoughts solved 20%, with per-letter accuracy rising from roughly 39% to 78%.

Both branches end the same way: reasoning that was orchestrated externally through a prompt gets internalized, either into the model's own weights or into a loop that calls tools on its own. PNG ↓

Each of these techniques trades inference compute for accuracy, and the trade is explicit in the numbers above: self-consistency’s headline result costs 40 full generations per question instead of one; Tree of Thoughts at breadth 5 costs several LM calls per search step, for every step, for every branch kept alive. That trade — spend more compute per query instead of more compute per training run — is the seed of the idea taken up directly in Reasoning Models, where STaR and later RL-trained systems internalize search and self-correction into the model’s own weights rather than orchestrating it externally through prompts. The same insight — that a model can be prompted to interleave reasoning with actions, not just with a final answer — also opens directly onto tool use, the subject of Agents.

Key Papers

Paper	Authors	Year	Link	Why it matters
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models	Wei et al.	2022	arXiv	Showed that modeling intermediate steps in the prompt, not just the answer, unlocks multi-step reasoning at sufficient model scale
Self-Consistency Improves Chain of Thought Reasoning in Language Models	Wang et al.	2022	arXiv	Replaced single greedy decoding with sample-and-vote across many reasoning paths, turning one frozen model into its own ensemble
Tree of Thoughts: Deliberate Problem Solving with Large Language Models	Yao et al.	2023	arXiv	Generalized prompted reasoning into explicit search with backtracking, using the model itself as both proposer and evaluator

PM Lens

The line I watch for in a design review is someone writing “just add chain-of-thought” or “run self-consistency” as if it were a prompt-copy edit. It isn’t — it’s a cost and latency decision with a multiplier attached. Self-consistency at 40 samples is roughly 40 generation calls per user question instead of one; Tree of Thoughts adds a scoring call at every branch of every search step. That can be the right trade for a coding agent that only needs to be right once — sample several candidate solutions, run the unit tests, keep the one that passes, and the test suite does the job self-consistency’s majority vote does for math. It is the wrong trade for a chat reply where users expect an answer in a second or two. My rule of thumb: before approving a “make it reason more” feature request, I ask what the accuracy gain is worth per additional dollar and per additional second of latency — because that ratio, not the benchmark chart, is what decides whether the feature ships.