Agents

It is October 2022. Chain-of-thought prompting has shown that a language model coaxed to think out loud reasons better, but a model that only reasons in text still cannot check a fact, run code, or click a button — it can describe an action, it cannot take one. ReAct (Yao et al., 2022) closes that gap by interleaving reasoning with real actions inside a single loop, opening a stretch in which language models move from answering questions to finishing jobs: writing code that runs against a test suite, browsing the open web, and eventually operating a computer the way a person does.

Where We Were

By mid-2022 the transformer-based language model had learned two separate tricks, and neither one alone could finish a job. In-context learning let a model take on an unfamiliar task from a prompt and a handful of examples, no gradient step required. Chain-of-thought prompting, submitted in January 2022, showed that asking a model to work through a problem step by step before answering raised accuracy on arithmetic and commonsense benchmarks — but every one of those steps was still just more text, generated from the model’s own prior beliefs, with no mechanism to check whether any step was true. A model reasoning through a multi-hop trivia question could talk itself into a confident, fluent, wrong answer, and nothing in the loop would notice.

Two other threads from 2021–2022 supplied pieces of the fix without assembling them. OpenAI’s Codex (Chen et al., 2021), submitted in July 2021, fine-tuned GPT-3 on public GitHub code and showed a language model could write — and, in the paper’s evaluation harness, actually execute — working programs: it solved 28.8% of the hand-written Python problems in the new HumanEval benchmark on a single attempt, and 70.2% when allowed 100 samples per problem. A distinct production version of Codex went on to power GitHub Copilot. Codex proved a model could act inside a formal environment where correctness is mechanically checkable, but it answered one self-contained prompt at a time; it never looked at the result of running its own code and decided what to do next. Separately, retrieval-augmented generation let a model pull outside text into its context window, but retrieval was one-shot and read-only — fetch a passage, answer, done. Neither Codex’s execution nor RAG’s retrieval closed the loop of observing an outcome and adjusting the next step, and InstructGPT’s RLHF, submitted that same March, made models more compliant conversationalists without touching the acting question at all. By the autumn of 2022 the field had reasoning, execution, and grounding as three separately demonstrated capabilities, running in three different papers.

The Breakthrough

ReAct: put reasoning and acting in the same loop. “ReAct: Synergizing Reasoning and Acting in Language Models” (Yao et al., 2022), submitted in October 2022, made the fix structural rather than additive. Instead of prompting a model to either think (chain-of-thought) or act (call a tool), ReAct prompts it to alternate: emit a thought — a free-text reasoning step that plans or diagnoses — then an action drawn from a small fixed set (in the paper’s Wikipedia-based tasks, search, lookup, and finish), then read the resulting observation the environment returns, and use that observation to produce the next thought. Each cycle’s observation is grounded fact, not the model’s own belief, so a wrong intermediate assumption gets corrected by what actually comes back rather than compounding silently across steps. On HotpotQA multi-hop question answering and the Fever fact-verification benchmark, ReAct produced more interpretable trajectories and reduced the hallucination and error-propagation the paper found “prevalent in chain-of-thought reasoning.” On the interactive decision-making benchmarks ALFWorld and WebShop, ReAct prompting — with only one or two in-context examples, no fine-tuning — beat imitation- and reinforcement-learning baselines by 34 and 10 absolute percentage points respectively. One month later, ChatGPT launched (November 30, 2022) and made conversational prompting the default way people met a language model at all; ReAct’s contribution was to show that the same loop powering a conversation could, with a small change of what counted as an “action,” power a task.

Toolformer: skip the prompt engineering, learn it from data. ReAct’s loop still depended on hand-written prompts showing the model how and when to act. “Toolformer: Language Models Can Teach Themselves to Use Tools” (Schick et al., 2023), submitted in February 2023, trained a model to decide “which APIs to call, when to call them, what arguments to pass, and how to best incorporate the results into future token prediction,” starting from only a handful of human-written examples per tool. The method samples plausible API calls at many positions in ordinary text, executes each candidate against a real tool — a calculator, a question-answering system, two search engines, a machine-translation system, a calendar — and keeps only the calls whose result measurably reduces the model’s loss on the tokens that follow. The model is then fine-tuned on its own text with the surviving calls stitched in, so calling a tool becomes just another thing the model has learned to do mid-sentence. The result was a GPT-J-based model with substantially improved zero-shot performance across the tasks its tools covered, competitive with much larger models on those tasks, without degrading the model’s general language modeling ability — self-supervision, not a labeled demonstration set, is what taught it to reach for a calculator instead of guessing an arithmetic answer.

From single tools to open-ended agents. Through 2023 the open-source community pushed the ReAct loop toward full autonomy — projects such as AutoGPT chained the thought-action-observation cycle into long unsupervised runs with self-generated subgoals, with uneven reliability. “Voyager: An Open-Ended Embodied Agent with Large Language Models” (Wang et al., 2023), submitted in May 2023, gave that ambition a more disciplined architecture inside Minecraft, using GPT-4 purely through black-box API calls — no fine-tuning at all. Voyager combines three pieces: an automatic curriculum that proposes the next goal given the agent’s current state, so exploration keeps expanding rather than looping; an ever-growing skill library of executable JavaScript programs, each one a reusable, composable behavior retrieved by embedding similarity when a similar situation recurs; and an iterative prompting loop that feeds execution errors and environment feedback back to GPT-4 until a generated program runs cleanly, then verifies with a self-check before committing the skill to the library. Against the strongest prior systems, including ReAct- and AutoGPT-style baselines, Voyager obtained 3.3 times more unique items, traveled 2.3 times farther, and unlocked key tech-tree milestones up to 15.3 times faster — and, distinctively, it could drop its learned skill library into a brand-new world and solve novel tasks immediately, where the baselines had to relearn from scratch.

SWE-bench: measuring agents against real work. Voyager showed an agent accumulating skill in a game world; the question of whether the same loop could do economically real work needed its own yardstick. “SWE-bench: Can Language Models Resolve Real-World GitHub Issues?” (Jimenez et al., 2023), submitted in October 2023, built one from 2,294 task instances drawn from real GitHub issues and their merged pull requests across 12 popular Python repositories. Each instance hands a model an issue description and the surrounding codebase and asks it to produce a patch; the patch is graded by running the repository’s own test suite, including tests the original human pull request added specifically to verify the fix, so grading is objective rather than a judgment call. A task counts as resolved only if the patch makes those tests pass. Tasks routinely require understanding and coordinating changes across multiple files and functions at once, well past the single-function scope of Codex’s HumanEval. The paper’s headline number was a gap, not a win: the best system tested, built on Claude 2, resolved only 1.96% of the instances — real repositories, at their actual size and messiness, were far harder for an agent than a curated coding benchmark had suggested.

Productizing the loop: computer use and MCP. Two announcements in late 2024 turned the research loop into shipped infrastructure. On October 22, 2024, Anthropic introduced computer use in public beta (Anthropic, 2024): an upgraded Claude 3.5 Sonnet that perceives a computer desktop through periodic screenshots and acts on it with the same primitives a person uses — moving a cursor, clicking, typing, scrolling — rather than through an application-specific API. On the OSWorld benchmark, screenshot-only, that model scored 14.9% against a 7.8% next-best system, rising to 22.0% when given more steps per task; on SWE-bench Verified, a 500-instance human-validated subset OpenAI’s Preparedness team released with the original SWE-bench authors that August, the same model reached 49.0%, up from 33.4% for its immediate predecessor and ahead of the prior best publicly available system’s 45%. A month later, on November 25, 2024, Anthropic introduced the Model Context Protocol, or MCP (Anthropic, 2024): an open specification for the observation and action side of the loop itself, so that connecting a model to Google Drive, Slack, GitHub, Git, Postgres, or any other system no longer required a bespoke integration per model per tool — one protocol, SDKs in Python and TypeScript, and a shared library of servers other tools could plug into instead of each reinventing the wire format ReAct and Toolformer had first shown a model could speak.

The Mathematics

ReAct’s contribution is best stated as a change to the action space a language model policy is allowed to choose from. Let $\mathcal{A}$ be an environment’s real action set — the moves that actually change the world, such as search[query] or click[button] — and let $\mathcal{L}$ be the space of natural-language strings. ReAct augments the space the policy samples from to $\hat{\mathcal{A}} = \mathcal{A} \cup \mathcal{L}$, and defines the policy over trajectories rather than single steps:

$$\hat{a}_t \sim \pi_\theta\!\left(\hat{a}_t \mid c_t\right), \qquad c_t = \left(o_1, \hat{a}_1, \dots, o_{t-1}, \hat{a}_{t-1}, o_t\right)$$

Symbol	Meaning
$\mathcal{A}$	The environment’s real action space, e.g. search, click
$\mathcal{L}$	The space of natural-language thought strings
$\hat{\mathcal{A}}$	Augmented action space, $\mathcal{A} \cup \mathcal{L}$
$\pi_\theta$	The language model, used as a policy over $\hat{\mathcal{A}}$
$c_t$	Context: the trajectory of observations and actions up to step $t$
$o_t,\ \hat{a}_t$	The observation and chosen action at step $t$

Toolformer’s contribution is a criterion for deciding, without any human label, which candidate tool calls belong in a model’s training data. For a position $i$ in a text and a candidate API call $c_i$, define $L_i(z)$ as the model’s loss on the tokens following position $i$ when the text is prefixed with $z$. Compare the loss with no useful call against the loss once the call’s actual result $r_i$ is inserted:

$$L_i^{-} = \min\!\left(L_i(\varnothing),\, L_i\big(e(c_i,\varepsilon)\big)\right), \qquad L_i^{+} = L_i\big(e(c_i, r_i)\big)$$ $$L_i^{-} - L_i^{+} \geq \tau_f$$

Symbol	Meaning
$L_i(z)$	Model loss on tokens after position $i$, given prefix $z$
$e(c_i, r)$	Text serialization of API call $c_i$ with result $r$ inserted
$\varepsilon$	An empty, unhelpful result
$\varnothing$	No call inserted at all
$r_i$	The API’s actual response to the sampled candidate call $c_i$
$\tau_f$	Filtering threshold an API call’s loss reduction must clear

What It Enabled

The practical effect was a new axis along which a language model deployment could be judged: not only “how good are its answers” but “how many steps can it complete unsupervised before something breaks.”

	Chat-only (2020–2022)	Tool-augmented (2022–2023)	Autonomous agent (2023–2026)
Action space	None — text in, text out	Fixed small tool set per task	Open-ended: filesystem, shell, browser, screen
Feedback loop	None within a turn	Observation after each tool call	Multi-hundred-step loops with self-correction
Verification	Human reads the answer	Tool result checked against expectation	Test suites, unit tests, task completion checks
Representative system	InstructGPT, early ChatGPT	ReAct, Toolformer	Voyager, SWE-bench agents, computer use

The SWE-bench story is also a story about how fast the ceiling moved once agentic scaffolding — not just the base model — became something labs optimized deliberately:

Benchmark	Date	Scope	Headline result
SWE-bench (Jimenez et al., 2023)	Oct 2023	2,294 instances, 12 Python repos	Best system, on Claude 2, resolves 1.96%
SWE-bench Verified	Aug 2024	500 human-validated instances	Released by OpenAI Preparedness with the original authors
Claude 3.5 Sonnet, upgraded (Anthropic, 2024)	Oct 2024	Same 500-instance Verified set	49.0%, up from 33.4% the prior release

Three threads carry forward from here. First, the same act-observe-revise structure, turned inward rather than out toward tools, is what reasoning models use when they extend their own chain of thought at inference time — the loop this chapter externalizes against a real environment, chapter 20 runs internally against the model’s own draft reasoning. Second, agent benchmarks proliferated in SWE-bench’s wake — OSWorld for desktop control, and the broader evaluation methodology this chapter’s benchmarks feed into is catalogued in measuring minds and the frontier. Third, MCP turned “which tools can this model use” from a one-off integration decision into an interoperability question the whole industry now answers once, not per model.

Key Papers

Paper	Authors	Year	Link	Why it matters
Evaluating Large Language Models Trained on Code	Chen et al.	2021	arXiv:2107.03374	Codex: a language model that writes and executes working code; became GitHub Copilot
ReAct: Synergizing Reasoning and Acting in Language Models	Yao et al.	2022	arXiv:2210.03629	Interleaved thought-action-observation loop; +34 points on ALFWorld, +10 on WebShop
Toolformer: Language Models Can Teach Themselves to Use Tools	Schick et al.	2023	arXiv:2302.04761	Self-supervised loss filter teaches a model which API calls to insert, with no labels
Voyager: An Open-Ended Embodied Agent with Large Language Models	Wang et al.	2023	arXiv:2305.16291	GPT-4 plus a growable skill library; 3.3x items, 2.3x distance vs prior SOTA in Minecraft
SWE-bench: Can Language Models Resolve Real-World GitHub Issues?	Jimenez et al.	2023	arXiv:2310.06770	Real GitHub issues graded by real test suites; best 2023 system resolved 1.96%
Introducing computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku	Anthropic	2024	Anthropic	Screenshot-and-coordinate control of a real desktop; SWE-bench Verified to 49.0%
Introducing the Model Context Protocol	Anthropic	2024	Anthropic	One open protocol for tool and data connections, replacing per-model custom integrations

PM Lens