The Economics of Test-Time Compute

It is August 2024. For four years, buying a better language model has meant buying a bigger one and pretraining it longer — the scaling-law bargain struck by Chinchilla and its predecessors. A paper out of UC Berkeley and Google DeepMind asks a different question: for a fixed accuracy target, is it actually cheaper to grow the model, or to let a smaller one spend more tokens thinking before it answers? Five months later, an open-weights model built on exactly that premise undercuts the reigning reasoning model’s price by roughly 27 times, and the question stops being academic.

Where We Were

By the middle of 2024, the economics of a large language model had a settled shape. Scaling laws — Kaplan’s 2020 curves and Chinchilla’s 2022 compute-optimal refinement — treated pretraining compute as the one dial that bought capability: more parameters, more tokens, more accelerator-hours, in a ratio the Chinchilla paper had worked out precisely. That was a one-time, up-front cost. Once training finished, a model’s per-query serving cost was treated as close to fixed: an answer took roughly as many output tokens as the question warranted, inference engineering optimized for latency and throughput at that roughly fixed length, and the serving-side techniques that make a given answer length cheap and fast — batching, key-value caching, the memory-bandwidth accounting behind both — are the subject of Inference Engines and rest on the roofline model derived in The Memory Wall.

Chain-of-thought prompting, submitted in January 2022, had already shown that spending extra output tokens on visible intermediate steps, rather than jumping straight to an answer, improved accuracy on multi-step problems. But that extra spending was still a prompting trick layered on a model priced the same way regardless of how the tokens were used — a few hundred tokens of “reasoning” cost exactly what a few hundred tokens of anything else cost, because nothing in the training loop or the pricing model distinguished them. Nobody had yet asked the question in dollar terms: given a fixed compute or dollar budget, is it better spent making the model permanently bigger, or spent per-query letting it think longer? Pretraining compute and inference compute were accounted for in entirely separate ledgers — one a capital expense paid once, the other an operating expense paid on every request — and nothing forced a comparison between them.

Part of why the comparison stayed unforced was that answer length itself was not yet a variable anyone tuned for accuracy. RLHF had, since 2022, shaped what a helpful answer looked like, but the reward models behind it were trained on human preference judgments between complete, similarly-sized responses — nudging style and helpfulness, not deliberately stretching or compressing how many tokens a model spent working toward an answer. A model’s output length was an emergent property of its training data and its instruction-tuning distribution, not a budget a product team could set per request. Test-time compute made length itself the lever.

Inference compute goes from a free byproduct of prompting to a priced budget with its own line item, in under five years. PNG ↓

The Breakthrough

“Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters” (Snell et al., 2024), submitted in August 2024, put pretraining compute and test-time compute on the same axis for the first time: FLOPs. The paper ran a FLOPs-matched comparison — for a fixed total compute budget, how much better does a fixed-size model get if that budget is spent letting it search, sample, and revise at inference time, versus how much better would a model pretrained with that same extra compute have been instead? The headline result was that on problems where a smaller base model already had “somewhat non-trivial success rates,” an adaptive, prompt-difficulty-aware test-time strategy let that smaller model outperform a model roughly 14 times larger, evaluated at matched FLOPs. The paper also reported that its compute-optimal allocation strategy — choosing, per question, how to split the budget between wider sampling and deeper revision — improved the efficiency of test-time scaling by more than four times over a naive best-of-N baseline. That distinction matters economically as much as algorithmically: best-of-N sampling pays for N complete, independent generations and keeps the best one, so its cost scales linearly with N regardless of how early a bad attempt could have been abandoned, while a revision chain pays sequentially for one attempt plus however many corrections it takes, and a verifier-guided search pays for both generation and the verifier’s own scoring passes. Three strategies, three different cost curves against the same accuracy target — which is why the paper’s finding was not “test-time compute is cheap” but “which form of test-time compute is cheapest depends on the question,” a difficulty-dependent allocation problem rather than a fixed multiplier. The mechanics of that allocation — how a model is prompted to revise its own answer, how a verifier scores partial solutions — belong to Reasoning Models, which derives the compute-optimal-strategy formalism in full; this chapter takes that result as given and asks what it means in dollars.

The paper’s own framing was still a research comparison: a fixed base model, various search-and-revise strategies applied on top of it at evaluation time, no change to the model’s weights. DeepSeek-R1 (DeepSeek-AI, 2025) closed the gap between that research result and a shipped product. Released January 20, 2025, with the accompanying paper submitted two days later, R1 trained the tendency to produce a long chain of reasoning directly into the model’s weights via reinforcement learning against rule-based, mechanically checked rewards — a training recipe Reasoning Models covers step by step. What matters for this chapter is what that choice did to the unit economics. R1 is built on DeepSeek-V3, a sparse mixture-of-experts model with 671 billion total parameters but only 37 billion activated per token — the routing mechanics belong to Mixture of Experts — so a model with frontier-scale total capacity has a per-token serving cost closer to a much smaller dense model’s. Combine cheap-per-token serving with a training recipe that produces long reasoning traces natively, and DeepSeek-AI shipped R1’s API at $0.55 per million input tokens on a cache miss and $2.19 per million output tokens, against the $15 and $60 per million tokens OpenAI’s o1-preview — the first shipped reasoning model, launched September 12, 2024 — had set four months earlier. Test-time compute went, in one product cycle, from a research finding about FLOPs to a live number on an invoice.

A bigger model pays upfront in training capex; a smaller model thinking longer pays per query, every query. PNG ↓

The Mathematics

Whatever strategy produces an answer — a single forward pass, a hidden chain of thought, a sampled-and-voted set of paths — a token-priced API bills for it as input and output tokens. Reasoning tokens are typically generated but not shown to the user, yet OpenAI’s own documentation confirms they are still billed as output tokens and suggests budgeting on the order of 25,000 reasoning tokens for problems that benefit from extended thinking. The cost of one query is therefore an accounting identity, not a modeling claim:

C(q) = p_{\text{in}}\, n_{\text{in}} + p_{\text{out}} \big( n_{\text{think}} + n_{\text{ans}} \big)

$C(q)$ is the dollar cost of answering one query $q$ . $n_{\text{in}}$ is the number of input tokens, priced at $p_{\text{in}}$ per token. $n_{\text{think}}$ is the number of hidden reasoning tokens the model generates before answering, and $n_{\text{ans}}$ is the number of tokens in the visible answer; both are priced at the output rate $p_{\text{out}}$ , since a reasoning API does not distinguish them at billing time. What this buys you: a query’s cost is not set by its price sheet alone but by $n_{\text{think}}$ , a quantity the user never sees and the model alone controls — which is why comparing two reasoning models by their per-token price sheet, without knowing how verbose each one’s hidden reasoning is, compares only half the bill.

Symbol	Meaning
$C(q)$	total dollar cost of answering query $q$
$p_{\text{in}}, p_{\text{out}}$	price per input token, price per output token
$n_{\text{in}}$	number of input tokens in the prompt
$n_{\text{think}}$	number of hidden reasoning tokens generated
$n_{\text{ans}}$	number of tokens in the final visible answer

That gap between price sheet and realized bill is not hypothetical. Testing thirty matched prompts across models, the benchmarking firm Artificial Analysis found that o1-preview, priced at six times GPT-4o’s per-token rate, cost more than thirty times as much per query in practice — because o1-preview’s hidden reasoning made $n_{\text{think}}$ far larger than anything GPT-4o generates, and that extra length compounds the price-per-token multiple rather than sitting alongside it.

The Snell et al. result reframes that same trade at the level of a whole model rather than one query, in FLOPs rather than dollars. This chapter converts it into a dollar decision rule by using the fact that API pricing tracks a model’s active-parameter count and its token count fairly closely in practice. Suppose a reference model of active-parameter size $N$ serves a baseline answer of $L$ tokens at a serving cost proportional to $c \cdot N$ per token. A model $\alpha$ times larger, priced accordingly, costs $C_{\text{big}}$ per query; the same-size model, spending a factor $k$ more output tokens on reasoning to reach comparable accuracy, costs $C_{\text{small}}$ :

\begin{aligned} C_{\text{big}} &= c \cdot (\alpha N) \cdot L \\ C_{\text{small}} &= c \cdot N \cdot (kL) \\ \frac{C_{\text{small}}}{C_{\text{big}}} &= \frac{k}{\alpha} \end{aligned}

The per-token serving cost of a model scales with $c$ , its active-parameter count, and how many tokens it generates. A bigger model ( $\alpha$ times the parameters) that still answers in $L$ tokens costs $C_{\text{big}}$ ; the same-size model that instead answers in $kL$ tokens — spending a factor $k$ more on thinking — costs $C_{\text{small}}$ . Dividing the two cancels $c$ , $N$ , and $L$ , leaving the entire decision in one ratio, $k/\alpha$ . What this buys you: a build-versus-serve rule of thumb. If the token-multiplier $k$ needed to match a bigger model’s accuracy is smaller than that model’s parameter-multiplier $\alpha$ — exactly the regime Snell et al. reported, where a well-allocated test-time strategy matched a 14 times larger model — thinking longer is the cheaper path to the same accuracy. On harder problems, where a much larger $k$ is needed to close the gap at all, the ratio flips and training the bigger model wins; the paper’s own qualifier, “somewhat non-trivial success rates,” is exactly the boundary this ratio makes precise.

Symbol	Meaning
$N$	active-parameter count of the reference model
$\alpha$	parameter multiplier of a larger model reaching the same accuracy
$k$	token multiplier the reference model needs at test time to match that accuracy
$L$	baseline answer length in tokens, no added reasoning
$c$	serving cost per active parameter per generated token
$C_{\text{big}}, C_{\text{small}}$	per-query cost of the larger model and of the same-size model with test-time compute

Paying that dollar cost also takes time, and the time is not free of its own accounting. Autoregressive decoding generates one token per forward pass, so a longer reasoning trace multiplies the number of sequential steps a query occupies a GPU for:

T(q) \approx \big(n_{\text{think}} + n_{\text{ans}}\big) \cdot t_{\text{step}}

$T(q)$ is the GPU-time one query occupies during decoding; $t_{\text{step}}$ is the latency of generating a single token, which The Memory Wall shows is set mostly by memory bandwidth rather than arithmetic once a model is decoding one token at a time, and which grows as the key-value cache it must read grows — the cache-size accounting owned by Inference Engines. What this buys you: the same $k$ -times increase in reasoning tokens that multiplies the dollar bill in the equations above also multiplies GPU-seconds occupied per query roughly linearly, which is why a provider serving reasoning models at low margin depends on batching many queries’ decode steps together — the technique that chapter covers — rather than on any single query becoming individually cheaper.

Symbol	Meaning
$T(q)$	GPU decode-time occupied by query $q$
$n_{\text{think}}, n_{\text{ans}}$	reasoning tokens and answer tokens, as above
$t_{\text{step}}$	latency per decoded token, set by memory bandwidth and KV-cache size

What It Enabled

The immediate, visible effect was a pricing ladder that put a number on the reasoning premium — and then collapsed it within months:

Model	Launch	What it does	Input $ per million tokens	Output $ per million tokens
GPT-4o	2024-05	Direct answer, no trained-in reasoning trace	$5.00	$15.00
o1-preview	2024-09	RL-trained to emit a long hidden reasoning trace before answering	$15.00	$60.00
DeepSeek-R1	2025-01	Same trained-reasoning approach, open weights, sparse MoE serving	$0.55 (cache miss)	$2.19

The comparison that matters more than any single row is structural — what a “better” answer costs, and where that cost is paid, changed shape entirely:

	Pre test-time-compute era	Test-time-compute era
What buys more accuracy	A bigger pretrained model	The same model, more inference tokens
Where the spend happens	Training capex, paid once	Serving opex, paid on every query
Typical output length	Roughly fixed, a few hundred tokens	A variable thinking budget, hundreds to tens of thousands of tokens
What inference engineering optimizes for	Latency and throughput at a fixed output length	Cost and latency at a chosen, variable reasoning depth
Main serving cost driver	Active parameters times tokens generated	Active parameters times thinking tokens plus answer tokens

Once reasoning depth became a dial rather than a fixed property of a model, it became something a product could set per request rather than something a vendor set once at training time — route easy queries to a short answer and hard ones to a long reasoning trace, spending the budget the equations above describe only where it earns its keep. DeepSeek-AI pushed the same economics one step further by distilling R1’s reasoning behavior into dense Qwen and Llama checkpoints between 1.5 billion and 70 billion parameters — an application of the general teacher-student compression Distillation covers, aimed here at a specific asset: not the teacher’s output distribution, but its tendency to produce a long, self-correcting chain of thought. A team that cannot serve a 671-billion-parameter mixture-of-experts model at all can still buy a meaningful share of its reasoning behavior in a checkpoint sized to fit on hardware it already owns, which turns the build-versus-buy decision this chapter’s cost ratio formalizes into a build-versus-buy-versus-distill decision instead. That per-request dial is what the reasoning-driven planning inside agentic systems draws on: an agent deciding how many tool calls or how much deliberation a step deserves is making exactly the $k$ -versus- $\alpha$ trade this chapter formalizes, one step at a time. It is also what keeps the frontier-evaluation problem moving: a benchmark score that ignores the dollar and latency cost paid to reach it describes only one axis of a model’s usefulness. And because the cheapest path to a long reasoning trace turned out to run through sparse serving rather than dense scaling, the interaction between mixture-of-experts routing, memory-bound decoding, and the accelerator roadmap is precisely the co-evolution thread Hardware, Model Co-evolution closes this book with.

Once thinking is priced as output tokens, undercutting the first mover's price by an order of magnitude becomes the next move. PNG ↓

Key Papers

Paper	Authors	Year	Link	Why it matters
Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters	Snell et al.	2024	arXiv:2408.03314	First FLOPs-matched comparison of pretraining compute against inference compute; found a well-allocated test-time strategy can match a 14 times larger model on problems the smaller model already handles non-trivially
DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning	DeepSeek-AI	2025	arXiv:2501.12948	Trained long reasoning traces directly into an open-weights, sparse mixture-of-experts model and shipped it roughly 27 times cheaper per token than the reasoning model that had launched four months earlier

PM Lens

The number that changed how I scope a “make it reason more” feature is the gap between price-sheet ratio and realized bill: o1-preview cost six times GPT-4o’s rate per token but, on Artificial Analysis’s own testing, more than thirty times as much per query, because nobody bills for verbosity separately from price. That taught me to ask for a token-count estimate, not just a price quote, before approving any feature that turns on extended reasoning. The second number is DeepSeek-R1’s launch price against o1’s: $0.55 and $2.19 per million tokens against $15 and $60, four months apart, for comparable benchmark scores. That is not a rounding-error optimization — it is the difference between “reasoning is a paid premium tier” and “reasoning is a default you leave on.” My working rule now is to treat the reasoning-token budget as a product parameter with its own line item, sized per use case, the way I’d size a rate-limit or a cache TTL — because the vendor has already made spending more tokens to think a purchasable dial, and the cost of leaving that dial at the SDK default is now large enough to show up in a monthly bill.

Search