Chapter 24 · Section 24.3 · 105 min

ReAct and reasoning with actions

ReAct is a way to run a language model as an agent that thinks between its actions. Before each tool call, the model writes a short note to itself, called a thought. After each call, it reads the result, called an observation, before it thinks again. A thought is plain text that no tool runs. It holds the plan and the facts found so far. Almost every agent you will build or be asked about in an interview, from a search assistant to a coding agent, runs some version of this cycle of thought, action, and observation.

The idea matters because the two simpler designs fail in opposite ways. Take a question that needs two facts in a row, such as the city where the architect of some library was born. A model that only reasons writes a fluent chain of steps from what its weights remember, the chain of thought of §18.1. If one of the two facts is missing from its memory, it fills the gap with something plausible, and nothing in the chain can notice. A model that only acts calls a search tool at every turn with no reasoning in between. It fetches true facts. Then, with no plan written down anywhere, it answers with the first fact and forgets that it needed a second.

Yao et al. (2023) named the method ReAct, for reasoning plus acting, and measured both failures on HotpotQA, a benchmark of questions that each need two Wikipedia facts combined. Among the chain-of-thought runs that failed, 56% failed because the model made up a fact or a reasoning step. Interleaving thoughts with searches brought that share to 0%. It left failures of a new kind. Of ReAct's failures, 47% were reasoning errors, many of them loops that repeat an earlier thought and action, and 23% were searches that returned nothing useful.

This section builds on the agent loop of §24.1, so here it is in brief. The model writes text, and a parser pulls an action out of it. The environment, everything outside the model that the agent acts on, runs the action and returns an observation as text. The loop repeats until the model writes a final answer or a step budget, a cap on the number of turns, runs out. Every token of the finished episode is tagged by who wrote it with a source mask $m_i$ , which is 1 for a token the model sampled and 0 for a token that the prompt or the world supplied. §24.2 made the parsed action reliable.

Here we answer two questions. What should the model write besides the action? And how is a model trained to act well? The first half is about running agents. We define ReAct precisely, build one from scratch with error recovery and loop detection, follow one task through it token by token, and measure in a small simulation what thoughts and actions each contribute. The second half is about training. A model can label its own text with useful tool calls, it can imitate good episodes, and it can learn from its own episodes by reinforcement learning. There we prove the fact that every agent-training system rests on. Tokens written by the environment carry no learning signal for the model, and putting them in the loss teaches the model to expect whatever tool results came with success. We end with the decision an agent makes at every turn, whether a tool call is worth its cost.

Three ways to answer one question

Here is a small world to reason about, five encyclopedia pages invented for this section. There is one tool, search[entity]. It returns the first five sentences of the page whose title is exactly entity. If no title matches, it returns up to five similar titles instead. That is the Wikipedia search interface ReAct used. ReAct's agent had two more actions, lookup[keyword], which returns the next sentence of the current page that contains the keyword, and finish[answer], which ends the episode with an answer.

Vela Library (Tarsko)  The Vela Library is a public library in Tarsko. It was designed by the
                       architect Ines Moreau and opened in 1998.
Ines Moreau            Ines Moreau is an architect known for libraries and museums. She was born
                       in Quebec City in 1961.
Tarsko                 Tarsko is a port town known for its shipyards.
Moreau Bridge          The Moreau Bridge is a steel bridge in Lyon, opened in 1907.
Quebec City            Quebec City is the capital of the province of Quebec, in Canada.

✏️ One question answered three ways

The question is "In which city was the architect of the Vela Library born?" It needs two hops. The first goes from the library to its architect, and the second from the architect to her birthplace.

Chain of thought (reason only)
  Thought   The Vela Library was designed by Ines Moreau. Moreau is a French name, and the
            Moreau Bridge is in Lyon, so she was most likely born in Lyon.
  Answer    Lyon                                                              wrong, invented

Act (act only)
  Action 1       search[Vela Library]
  Observation 1  Could not find [Vela Library]. Similar: ['Vela Library (Tarsko)'].
  Action 2       search[Vela Library (Tarsko)]
  Observation 2  The Vela Library is a public library in Tarsko. It was designed by the
                 architect Ines Moreau and opened in 1998.
  Action 3       finish[Ines Moreau]                                          wrong, one hop short

ReAct (reason and act)
  Thought 1      I need the architect of the Vela Library, then where that person was born.
  Action 1       search[Vela Library]
  Observation 1  Could not find [Vela Library]. Similar: ['Vela Library (Tarsko)'].
  Thought 2      There is no exact page. The closest title is Vela Library (Tarsko).
  Action 2       search[Vela Library (Tarsko)]
  Observation 2  The Vela Library ... was designed by the architect Ines Moreau ...
  Thought 3      The architect is Ines Moreau. Now I need where Ines Moreau was born.
  Action 3       search[Ines Moreau]
  Observation 3  Ines Moreau is an architect ... She was born in Quebec City in 1961.
  Thought 4      Quebec City is the birthplace, so that is the answer.
  Action 4       finish[Quebec City]                                          right

Each trace shows one mechanism. The chain of thought recalled the first fact, invented the second, and had no way to check either. The act-only agent fetched the right page for the first hop but never wrote down that a second hop was needed. At turn 3 it decided from the raw pages alone and answered with the architect, one hop short. ReAct's first thought is a plan. Its second thought turns a failed search into a repair. Its third pulls out of a page the name that the next search needs. Every fact in its final thought came from an observation.

The widget steps through these traces, plus a fourth in which ReAct repeats a failed search until a loop detector stops it. Press Play or move the slider to reveal one step at a time, and watch where an error enters. The switch on the right, who wrote each token, boxes every token of the context and colors green the ones the model wrote. Only those tokens are trained, as the second half of this section proves.

The pattern is short enough to memorize. A thought changes only the context. An action changes the world, or at least queries it. An observation brings the world back into the context. Chain of thought has only thoughts, so it reasons well from premises that may be false. The act-only agent has actions and observations, so it gets true premises and reasons poorly about them. ReAct has all three. That is the one-paragraph answer to the interview question "why does ReAct work?"

A thought is an action that only changes the context

ReAct's authors state the idea as a change to what the model may do at each step. Besides the environment's actions, the model may now write any string at all, and such a string does nothing except sit in the context. The training half of this section depends on knowing exactly which text counts as what, so here is the precise statement. A policy is the rule that picks the next step given everything so far, here the language model with parameters $\theta$ .

📐 ReAct step and trajectory

Let $\mathcal{A}$ be the environment's actions (tool calls and finish[answer]), $\mathcal{O}$ its observations, and $\mathcal{Z}$ the set of natural-language strings. An element $z \in \mathcal{Z}$ is a thought. A ReAct policy $\pi_\theta$ with parameters $\theta$ acts in the augmented action space

\hat{\mathcal{A}} = \mathcal{A} \cup \mathcal{Z}.

A thought is sent to no tool, leaves the environment's hidden state unchanged, produces no observation, and only extends the context $c$ , as $c \leftarrow (c, z)$ . An action $a \in \mathcal{A}$ is executed and returns an observation $o \in \mathcal{O}$ . In the dense format used for question answering, every turn is one thought followed by one action, so an episode of $T$ turns for task $x$ is the trajectory

\tau = (x, z_1, a_1, o_1, z_2, a_2, o_2, \dots, z_T, a_T).

The last action $a_T$ is finish[answer], which ends the episode without an observation. If the step budget $B$ runs out first, the episode instead ends after $o_B$ with no answer. In the sparse format used for long decision tasks, the policy decides at each turn whether to think before it acts.

In plain words, a thought is an action whose only effect is on what the agent reads next.

The model writes both the thoughts and the actions, so both have $m_i = 1$ , and observations have $m_i = 0$ . A trained ReAct model therefore learns to write thoughts in the same way it learns to write actions.

🔬 A thought is a latent variable

Write $h_t$ for the history at turn $t$ , everything in the context so far, and $\pi_\theta(a_t \mid h_t)$ for the probability of an action given it, as in §24.1. That probability is a sum over every thought that could come before the action,

\pi_\theta(a_t \mid h_t) = \sum_{z \in \mathcal{Z}} \pi_\theta(z \mid h_t)\,\pi_\theta(a_t \mid h_t, z).

Sampling a thought and then an action draws from this sum, and the thought shapes the action through the second factor. Nobody computes the sum, since $\mathcal{Z}$ holds every possible string. The point is conceptual. A thought is an unobserved choice the policy makes on the way to an action. A training method that rewards only actions still trains the thoughts, through the samples it rewards.

Why a thought helps

Writing a thought helps for two different reasons. The first is computation. A Transformer does a fixed amount of work for each token it writes, so every extra token buys another pass through the network (§18.1). A thought spends that work on the decision at hand, for example on pulling the name "Ines Moreau" out of a page before choosing the next query.

The second reason is memory. The context is the agent's only record of the episode. An observation is long, noisy, and about the world rather than about the task. A thought writes down the plan and the facts that matter in a few tokens, so later turns read them instead of re-deriving them from raw pages. The act-only trace above failed for lack of memory, not of computation.

Observations do the opposite job. They replace a recalled fact, which can be wrong with no warning, by a fetched fact, which is wrong only when its source is. That is why made-up facts vanished from ReAct's failures. It is also why ReAct depends on something new, the quality of its tools. An interviewer who asks "what does a thought buy you that a longer prompt does not?" wants these two answers, computation spent on the current decision and a short written memory of the plan.

What ReAct measured

ReAct prompted PaLM-540B, a 540-billion-parameter language model from Google, with a few hand-written example trajectories in the prompt. That is few-shot prompting, and the model's weights were never changed. HotpotQA got 6 examples and FEVER got 3, because more examples did not help. FEVER asks whether a claim is supported, refuted, or not decidable from Wikipedia. Every baseline used prompts built from the same example trajectories with parts removed.

Standard removes thoughts, actions, and observations, so the model answers at once.
CoT (chain of thought) removes actions and observations, so the model reasons and then answers.
CoT-SC samples 21 chains of thought at temperature 0.7 and takes the majority answer. This is the self-consistency of §18.2.
Act removes the thoughts.

HotpotQA is scored by exact match, the share of answers equal to the reference string after light normalization such as lower-casing.

📌 ReAct's question-answering results with PaLM-540B

prompting method	HotpotQA exact match	FEVER accuracy
Standard	28.7	57.1
CoT	29.4	56.3
CoT-SC (21 samples)	33.4	60.4
Act	25.7	58.9
ReAct	27.4	60.9
CoT-SC, then ReAct	34.2	64.6
ReAct, then CoT-SC	35.1	62.0
supervised state of the art	67.5	89.5

ReAct beats Act on both tasks and beats CoT on FEVER, but on HotpotQA it trails CoT and even Standard prompting. The two combinations, defined below, are the best prompting methods on both tasks. The last row is a model trained on each task's full training set (Table 1 of the paper).

ReAct alone losing to chain of thought on HotpotQA is the result interviewers ask about, and the paper's error analysis explains it. The authors drew 50 correct and 50 incorrect trajectories from each of ReAct and CoT, 200 in all, and labeled each one by hand.

		ReAct	CoT
success	correct reasoning and facts	94%	86%
	right answer from made-up reasoning or facts	6%	14%
failure	reasoning error, including loops	47%	16%
	search returned nothing useful	23%	none
	made-up fact or step	0%	56%
	right answer that did not match the label exactly	29%	28%

Read the two columns as two different machines. CoT reasons freely and invents facts. ReAct reasons under a constraint, since every thought must lead to an action and every result must be read. Its facts are grounded, but its reasoning is less flexible. Its signature failure is to repeat its previous thought and action and never leave the loop. An uninformative search also derails it badly, because the next thought has nothing to work with. The two machines fail on different questions, which is why combining them works.

Each rule uses a cheap signal that one method is failing, running out of steps or a split vote, to hand the question to the other method. With these rules, ReAct plus CoT-SC matched the accuracy of 21-sample CoT-SC while drawing only 3 to 5 samples. In practice this is the first design lesson of agents. Pair a method that grounds facts with a method that reasons freely, and route between them on a signal you can observe.

🔬 ReAct on long decision-making tasks

The same paper tested two decision-making benchmarks, with two example trajectories in each prompt for ALFWorld and one for WebShop. ALFWorld is a text game of household tasks, such as finding an object, cleaning or heating it, and putting it somewhere. There the best of six ReAct prompts succeeded on 71% of tasks, and the average of the six on 57%. The best Act prompt reached 45%. BUTLER, an imitation-learning agent trained on 100,000 expert trajectories per task type, reached 37%.

WebShop is a simulated online store with 1.18 million real products, in which the agent must buy an item that matches a written request. ReAct's success rate was 40.0%, against 30.1% for Act, 29.1% for imitation learning, 28.7% for imitation learning plus reinforcement learning, and 59.6% for human experts. In these long tasks the thoughts were sparse, written only where the model chose to plan or to check its progress.

📜 Where ReAct came from

WebGPT (Nakano et al. 2021) fine-tuned GPT-3 to answer questions by driving a text browser with commands such as Search <query>, Clicked on link <id>, Find in page: <text>, Quote: <text>, Back, and End: Answer. It learned first by imitating human demonstrations. Then it sampled many answers and kept the best one, as scored by a reward model trained on human preferences. Its best answers were preferred to the human demonstrators' answers 56% of the time. It had actions and learned rewards, but no free-form thoughts.

MRKL systems (Karpas et al. 2022) proposed a router that sends each input to an expert module, such as a calculator, a database, or a language model. Self-Ask (Press et al. 2022) had the model pose and answer its own follow-up questions, and noticed that a search engine could answer those follow-ups instead. Inner Monologue (Huang et al. 2022) fed a robot's sensor feedback back into its planner's prompt. ReAct tested that style as an ablation, with thoughts limited to the state of the environment and the current subgoal, and it reached 53% on ALFWorld against ReAct's 71%. ReAct's contribution was the general format, free-form thoughts interleaved with any actions, and it is still the default design.

A ReAct agent from scratch

Python cells in this book run in your browser with no network, so no cell can call a real model. The policies below are short scripts with the same interface as a model, a context string in and text out. The loop is real, and only the decisions are hand-written. Each script reads the observations and decides what to do, as a model would. First come the five pages, the search tool, and a parser for one step.

Python · cell 1

That parser handles the tidy text our scripts write, and it breaks on what models actually produce. Real outputs number their steps (Thought 3:), put brackets inside arguments, and spread a thought over several lines. Sometimes they run past the action and write an Observation: line of their own. That is an invented tool result, and the harness, the program that runs the loop, must never trust it. Text-based ReAct implementations prevent it with a stop sequence, a string at which the API stops generating (§6.4), here Observation. A robust parser also ignores everything after the first action line.

Warm-up

Parse a ReAct step

Write parse_react_step(text). It returns (thought, action, arg), or None if the text holds no valid action.

The action line. It is the first line that, after optional leading spaces, starts with Action: or with Action <n>: for a step number $n$ , then has name[arg], and ends with ] (trailing spaces allowed). The name is letters, digits, or underscores. Return action as name in lower case. Return arg as everything between the first [ after the name and the last ] of that line, so an argument may contain brackets.
The thought. It is the text after the first Thought: or Thought <n>: that comes before the action line, with leading and trailing whitespace removed. It may span several lines. If there is no thought, return the empty string.
Ignore everything after the action line.
If no line qualifies as an action line, return None.

import re

def parse_react_step(text):
    m = re.search(r"Action:\s*(\w+)\[(.*?)\]", text)
    if m is None:
        return None
    return "", m.group(1).lower(), m.group(2)

Tests, hints, verified solutions, and AI help, with Pro at $7.99 a month.

The loop

Now the loop. It keeps the episode as a list of segments, each tagged with who wrote it, because that tag becomes the training mask. A failed tool call becomes an observation, as in §24.1, so the loop never crashes. An optional loop detector replaces the third identical action in a row with a warning.

Python · cell 2

The scripted policy reads the last observation. If the search failed, it picks the closest similar title. If a page names an architect, it searches for that person. If a page gives a birthplace, it answers.

Python · cell 3

The printout is the ReAct trace of the example box, now produced by running code.

One task, end to end

Here is what happens to the task at each turn, in numbers. Training sees the episode as tokens, so we need a tokenizer. A toy one that splits words and punctuation is enough here. Real tokenizers split words into subword pieces (§2.2), but only the proportions matter for this count.

Python · cell 4

The next cell follows the episode turn by turn. At each turn the policy reads the whole context so far and writes one thought and one action, and the world writes back one observation.

Python · cell 5

✏️ One episode, turn by turn

The goal enters the context as a 14-token question. Each row is one turn of the loop, with token counts from the toy tokenizer.

turn	action	policy wrote	world returned	policy read first
1	`search[Vela Library]`	25	23	14
2	`search[Vela Library (Tarsko)]`	28	26	62
3	`search[Ines Moreau]`	24	23	116
4	`finish[Quebec City]`	21	0	163

At turn 1 the policy reads only the question and writes a plan and a search. The search misses, and the 23-token error message names the right title. At turn 2 the policy reads 62 tokens and retries with that title. At turn 3 it reads the architect's name in the page and searches for her. At turn 4 the parser finds finish, the loop returns its argument, and nothing more is appended. That is the stop condition. The step budget of 7 was never reached.

The finished episode has 184 tokens. The policy wrote 98 of them ( $m_i = 1$ ), and the question and the world wrote 86 ( $m_i = 0$ ). Across its four turns the policy read 355 tokens, almost twice the length of the episode, because every turn re-reads everything before it (§24.1).

Even in this tiny episode the world wrote almost half the tokens. Real pages run to thousands of tokens per observation, so in a long real episode nearly every token has $m_i = 0$ . That fact drives the cost of agents and, as the second half shows, the way they must be trained.

Error recovery belongs in the loop and in the policy

The loop above makes four decisions that every production agent makes, and each has a reason.

Errors are observations. An unknown action, a malformed output, and a tool exception all come back as text. The exact message matters, because the model reads it. The message Error: unknown action 'serch'. Valid actions are ['search'] and finish. tells the model how to fix the call, and a bare "error" does not. The Model Context Protocol makes this a rule for arguments that fail validation, which go back to the model as tool errors rather than protocol errors (§24.6).
Retrying is the policy's decision. The loop never retries a tool on its own. It reports what happened, and the model chooses whether to retry, rephrase, or give up, because only the model knows whether the same call can succeed. Automatic retries of a call that timed out on the network, with backoff and idempotency keys, are a different layer, covered in §24.6.
Loops are detected by repeated pairs of action and normalized argument. A loop is ReAct's most common reasoning error. The cheapest detector is exact repetition after normalizing the argument (lower case, one space between words), so search[Vela Library] and search[vela library] count as the same call. Three identical actions in a row is a common threshold. An agent can also alternate between two failing calls, A, B, A, B, which no test of consecutive repeats catches, so the harness also compares each action with the one two steps back. On a loop, the harness injects a warning instead of executing the call, or, more aggressively, forbids the repeated action. Near-repeats, such as the same query with one word changed, need a fuzzier test, for example a threshold on string similarity.
Back off after a step limit. When the budget runs out without an answer, the trajectory is evidence that tools are not helping, and ReAct's back-off rule hands the question to self-consistency.

Agents often add one more kind of thought, a reflection. It is written after a failed attempt, says what went wrong, and stays in the context for the next attempt (Shinn et al. 2023 call this Reflexion). It is one more element of $\mathcal{Z}$ , and §24.4 works out what it adds over a plain retry. Whether a model can correct itself this way without an outside signal has a careful answer in §26.1. With a real error message or a failing test as the signal, reflection helps. Without one, it often does not.

Interview

A ReAct agent with loop detection and a training mask

This is the ReAct loop as an interviewer asks for it, with the loop detector a production harness needs and the training mask at the end. It leaves out two guards that a general loop also needs, a separate message for unknown tools and truncation of long observations, which Exercise 24.1c covers. Write react_agent(policy, tools, question, max_steps, tokenize), which returns (answer, segments, tokens, mask). The setup provides parse_action(text), a robust parser that returns (thought, action, arg) or None.

Segments. segments is a list of (source, text) pairs. It starts as [("prompt", f"Question: {question}\n")].
One step. At step t = 1, 2, ..., call out = policy(context) once, where context is the concatenation of all segment texts so far, and append ("policy", out).
Finishing. If parse_action(out) gives the action finish, the episode ends with its argument as the answer. Append nothing more.
The key of a step. Every other step has a key, (action, normalize(arg)). Here normalize lower-cases the argument, collapses every run of whitespace to one space, and strips the ends. A malformed output has no key and gives the observation "Error: could not parse an action.".
Loops. A step is a loop in two cases. Either its key equals the keys of the two previous steps (three identical actions in a row), or its key equals the key from two steps back and the previous key equals the key from three steps back (a cycle A, B, A, B, caught at its fourth step). A loop gets the observation "Warning: repeated action. Try something different." and runs no tool. A malformed step is never part of a loop. Record every step's key, warned or not.
Running the tool. Otherwise run tools[name](arg) with the argument as written, since normalization is only for comparing. Any exception, including a KeyError for an unknown name, gives f"Error: {type(e).__name__}: {e}", and success gives str(result).
Observations and the budget. Append ("env", f"\nObservation {t}: {obs}\n"). If max_steps calls pass without finish, the answer is None.
Tokens and mask. tokens concatenates tokenize(text) over the segments in order. mask is a list of ints, 1 for every token of a "policy" segment and 0 for every other token.

def react_agent(policy, tools, question, max_steps, tokenize):
    raise NotImplementedError

Tests, hints, verified solutions, and AI help, with Pro at $7.99 a month.

What each part buys, in a simulation

One trace proves nothing about rates. To see how the mechanisms combine, we run every method on 200 generated two-hop questions of the form "Where was the director of Film $k$ born?" against a noisy search tool. The failure modes are HotpotQA's, in miniature.

The world. Each film page names a director, and each director's page names a birth city. Each director's page is missing with probability 0.25 (63 of the 200 here), and then a search for it returns only unrelated titles. Each page has a disambiguated title such as "Ada Cruz (director)" with probability 0.3, and then an exact search misses but the list of similar titles shows the right page.
The model's memory. Chain of thought recalls each fact $f$ correctly with its own probability $r_f$ , drawn from a $\operatorname{Beta}(2, 2)$ distribution with mean 0.5. When recall fails, the model states its favorite wrong value with probability 0.7 and a random one otherwise, and it is equally confident either way. The favorite wrong value is what stops majority voting from improving forever, as §18.2 showed for models with a preferred wrong answer.
The agents. The ReAct and Act policies read each observation and act on it. Each has one slip probability that stands in for a measured failure. After a failed search, the policy repeats the same search with probability 0.5 instead of reading the similar titles. Once it has repeated itself, it keeps repeating until a warning arrives, because greedy decoding gives the same output for the same context. The Act policy, with no plan written down, answers with the director's name after the first hop with probability 0.35.

These probabilities are invented. The point is not the particular numbers but how the mechanisms combine.

Python · cell 6

The agent is one script for both ReAct and Act. With think=False it writes no thoughts and can lose track of the goal.

Python · cell 7

Python · cell 8

✏️ What the simulation shows

method	accuracy	what limits it
CoT	31.5%	every failure is a confident wrong answer
CoT-SC	44.0%	the favorite wrong answers win many votes
Act	33.0%	answers with the director on 29.5% of questions
ReAct	48.5%	51.5% of runs end with no answer
ReAct + loop warning	68.0%	pages that do not exist
ReAct, then CoT-SC	73.0%	runs stuck in a loop fall back to voting
CoT-SC, then ReAct	54.0%	split votes go to plain ReAct, which has no fallback
ReAct + warning, then CoT-SC	81.5%	questions both methods miss

The pattern is ReAct's, in miniature. Chain of thought fails only by confident wrong answers, the analogue of the 56% share of made-up facts. Plain ReAct edges past CoT-SC, and its failures are the analogue of ReAct's loops and useless searches, runs stuck repeating a failed search or looking for a page that does not exist. Each fix targets one of those failures. The loop warning turns most stuck runs into answers, and the back-off to CoT-SC answers many of the questions the tools cannot. Because the two methods fail on different questions, using both fixes gives 81.5%, far above either alone.

CoT-SC then ReAct does less well here, for two reasons. The vote is split on most questions, so most of them go to plain ReAct, whose stuck runs then have no fallback. And some clear majorities are favorite wrong answers, which never ask for help. In practice, this is the argument for detecting loops and for keeping a fallback. An interviewer who asks how you would raise an agent's success rate wants to hear both.

Toolformer, a model that labels its own data

Prompting a frozen model with example trajectories, as ReAct did, teaches the format of tool use but not the judgment of when a tool helps. Toolformer (Schick et al. 2023) asked whether a model can learn that judgment from plain text, with no human labels of where calls belong. The idea fits in one sentence. Let the model propose calls inside ordinary text, run them, and keep only the calls whose results make the following text easier for the model to predict. Then fine-tune the model on the text with those calls written in. The model's own loss is the teacher.

First, what "easier to predict" means. The loss is the negative log-likelihood, minus the log-probability the model gives each actual next token, measured in nats (natural-log units). One nat saved on a token means the model's probability for it grew by a factor of $e \approx 2.72$ .

The pipeline has four steps, run separately for each tool. The tools were a question-answering model, a Wikipedia search, a calculator, a calendar, and a translator.

Sample positions. Show the model a prompt with a few examples of calls written into text. For every position $i$ of a text $x_1, \dots, x_n$ , compute the probability $p_i$ that the model starts a call there. Keep the positions with $p_i > \tau_s$ , at most the top $k$ of them.
Sample calls. At each kept position, sample up to $m$ candidate calls $c_i$ from the model.
Execute. Run each call to get a result $r_i$ , a string.
Filter, then fine-tune. Keep a call only if its result lowers the model's loss on the next tokens by at least $\tau_f$ , as defined below. Write the kept calls and their results into the text, and fine-tune the model on the new corpus with the ordinary next-token loss.

📐 Toolformer's filtering criterion

Write a call and its result as the text $e(c_i, r_i)$ , of the form [Calculator(12 * 4) -> 48], and write $e(c_i, \langle\rangle)$ for the call with an empty result, where $\langle\rangle$ is the empty string. (The paper writes $\varepsilon$ for the empty string, a symbol this chapter uses for an error rate.) A prefix of $\langle\rangle$ alone means no call at all. For a prefix $\mathbf{z}$ and the model's log-probabilities $\log p(x_j \mid \mathbf{z}, x_{1:j-1})$ , the weighted loss of the text from position $i$ on is

L_i(\mathbf{z}) = -\sum_{j = i}^{n} w_{j - i}\,\log p(x_j \mid \mathbf{z}, x_{1:j-1}), \qquad w_t = \frac{\tilde w_t}{\sum_{s \ge 0} \tilde w_s},\quad \tilde w_t = \max(0,\ 1 - 0.2\,t).

The two losses compared are

L_i^+ = L_i\big(e(c_i, r_i)\big), \qquad L_i^- = \min\big(L_i(\langle\rangle),\ L_i(e(c_i, \langle\rangle))\big),

and the call is kept if and only if $L_i^- - L_i^+ \ge \tau_f$ . Here $\tau_s$ and $\tau_f$ are thresholds, not trajectories. The defaults are $\tau_s = 0.05$ , $\tau_f = 1.0$ , $k = 5$ positions, and $m = 5$ calls per position. The calculator and the translator ran on a small, filtered subset of the texts, so for them the paper set $\tau_s = 0$ , $k = 20$ , $m = 10$ , and $\tau_f = 0.5$ . The weights are indexed from $t = 0$ , the token right after the call, and normalized over all $s \ge 0$ , so $\tilde w = (1, 0.8, 0.6, 0.4, 0.2, 0, \dots)$ sums to 3 and only the next five tokens count.

Three details of the definition are easy to get wrong. First, the call is given as a prefix $\mathbf{z}$ before the whole text, not inserted at position $i$ . The model has not yet seen calls in the middle of text, and inserting one there would disturb its predictions for reasons that have nothing to do with the result. Second, the weights fall to zero after five tokens, so a result is credited only for helping the text right after it. Third, $L_i^-$ takes the better of two baselines, no call at all and the call without its result, so a call is never credited for information that was already in its own wording.

✏️ The criterion by hand, five tokens

Take one candidate call and the log-probabilities of the next five tokens under the three prefixes.

prefix	$t = 0$	$t = 1$	$t = 2$	$t = 3$	$t = 4$
no call, $\langle\rangle$	$-6.2$	$-0.4$	$-0.9$	$-1.5$	$-2.0$
call without result, $e(c, \langle\rangle)$	$-6.0$	$-0.5$	$-0.9$	$-1.4$	$-2.1$
call with result, $e(c, r)$	$-0.3$	$-0.5$	$-0.9$	$-1.5$	$-2.0$

The weights are $w = (1, 0.8, 0.6, 0.4, 0.2)/3 = (0.333, 0.267, 0.2, 0.133, 0.067)$ . Then

$L(\langle\rangle) = (6.2 + 0.8 \cdot 0.4 + 0.6 \cdot 0.9 + 0.4 \cdot 1.5 + 0.2 \cdot 2.0)/3 = 8.06/3 = 2.687$ ,
$L(e(c, \langle\rangle)) = (6.0 + 0.4 + 0.54 + 0.56 + 0.42)/3 = 7.92/3 = 2.640$ ,
$L^+ = (0.3 + 0.4 + 0.54 + 0.6 + 0.4)/3 = 2.24/3 = 0.747$ .

So $L^- = \min(2.687, 2.640) = 2.640$ , and $L^- - L^+ = 1.893 \ge 1$ , so the call is kept. Almost all of the gain is the first token, whose log-probability rose by 5.7 nats once the result was visible.

The hand computation found the saving as the difference of two weighted sums. Taken token by token, that difference says what the threshold $\tau_f$ really asks of a result. Each token contributes its weight times the log of a ratio, its probability with the result in view over its probability without it.

∂ Proof

Substitute the definitions,

L_i^- - L_i^+ = L_i(\mathbf{z}^-) - L_i(e(c_i, r_i)) = -\sum_t w_t \log p(x_{i+t} \mid \mathbf{z}^-, \cdot) + \sum_t w_t \log p(x_{i+t} \mid e(c_i, r_i), \cdot),

and combine the two sums term by term. ∎

In plain words, the criterion counts how many nats the result saves on the next few tokens. Each log-ratio is the model's own estimate of how much the result tells it about the next token. So the criterion is a measure of information, judged by the model on this one text. Two consequences are worth carrying into an interview. A threshold of $\tau_f = 1$ is demanding. If the whole saving falls on the first token, whose weight is $1/3$ , that token's log-probability must rise by 3 nats, a factor of $e^3 \approx 20$ . And the criterion measures usefulness for prediction, not truth. A result that is wrong but fits the text, or right but irrelevant to the next five tokens, is judged by its effect on the loss alone.

The cell below computes the criterion for five candidate calls in one sentence. The scorer is a toy language model. With probability 0.5 it copies an earlier token of its context, chosen uniformly, and otherwise it guesses one of 1,000 words uniformly. Copying is a crude stand-in for what a real model does with a result in its context, and it is enough to make the criterion behave as it does at scale.

Python · cell 9

Read the five rows. The right calculation is kept and the wrong one is dropped, since a wrong result makes the next tokens harder to predict. The calendar's result has nothing to do with "1998" and raises the loss. The question about eggs is the instructive case. Its wording alone, without the result, already lowers the loss, because the words "eggs" and "pack" can be copied. $L^-$ uses that better baseline, so the call is credited only for what its result adds, which is still enough. The question about the shop gains 0.95 nats, below the default threshold of 1 and above the 0.5 used for the calculator and the translator. The widget lets you move the threshold and how many tokens after the call carry weight.

Core

Toolformer's filter

Write two functions.

toolformer_weights(L, rate=0.2) returns the weights $w_0, \dots, w_{L-1}$ . Each is $\tilde w_t = \max(0, 1 - \text{rate}\cdot t)$ divided by $\sum_{s \ge 0}\tilde w_s$ , the sum over all $s$ , not only the first $L$ .
toolformer_filter(lp_none, lp_empty, lp_result, tau_f=1.0, rate=0.2) takes three arrays of shape (n_calls, L). Row $c$ of each holds the log-probabilities of the continuation tokens $x_i, \dots, x_{i+L-1}$ of candidate call $c$ . The first array has them under no call, the second under the call without its result, and the third under the call with its result. Return (L_plus, L_minus, keep). L_plus and L_minus are float arrays of shape (n_calls,) holding $L^+$ and $L^-$ for each call, with the weights of toolformer_weights(L, rate). keep is a boolean array of the same shape, True where $L^- - L^+ \ge \tau_f$ .

import numpy as np

def toolformer_weights(L, rate=0.2):
    return np.array([max(0.0, 1 - rate * t) for t in range(L)])

def toolformer_filter(lp_none, lp_empty, lp_result, tau_f=1.0, rate=0.2):
    raise NotImplementedError

Tests, hints, verified solutions, and AI help, with Pro at $7.99 a month.

📌 Toolformer's results with GPT-J (6.7B parameters)

On the T-REx subset of LAMA, a test of completing factual sentences, the score went from 31.9 for GPT-J to 53.5 for Toolformer, mostly through the question-answering tool. That is above OPT-66B (30.1) and GPT-3 with 175B parameters (39.8). On the math word problems of ASDiv, SVAMP, and MAWPS, it went from 7.5, 5.2, and 9.9 to 40.4, 29.4, and 44.0, mostly through the calculator, against 14.0, 10.0, and 19.8 for GPT-3. Language modeling did not suffer from the calls. With calls disabled, perplexity on WikiText and CCNet (10.3 and 10.5) matched GPT-J fine-tuned on the same texts without calls. In a family of models from 124M to 6.7B parameters, the ability to benefit from the tools appeared only at about 775M, except for the Wikipedia search tool, which smaller models could already use.

The paper is candid about its limits, and they define what came next.

No chains. Calls are sampled independently for each tool, so the training data has no chains, in which the result of one call is the input of the next.
No interaction. The model cannot browse a list of search results or refine a query the way ReAct does.
Sample inefficiency. More than a million documents gave only a few thousand useful calculator calls.
Sensitivity. The decision to call depended on the exact wording of the input.
Cost is ignored. The decision to call does not weigh what the call costs to run.

The evaluation also allowed at most one call per input, so that the model could not get stuck calling forever.

Toolformer answers "when does a call help this text?" well, and "how should a sequence of calls solve this task?" not at all. The second question needs whole episodes as training data.

Fine-tuning on trajectories

The simplest way to teach a model ReAct is to show it many successful ReAct episodes and fine-tune on them. This is supervised fine-tuning, training on example outputs with the ordinary next-token loss, as in §14.3, applied to whole trajectories.

ReAct itself did a first version. It generated trajectories with the 540B model, kept 3,000 that reached a correct answer, and fine-tuned PaLM-8B and PaLM-62B to write whole trajectories, thoughts, actions, and observations alike. Fine-tuned ReAct became the best of the four methods at both sizes. PaLM-8B fine-tuned on ReAct beat every prompting method at 62B, and PaLM-62B fine-tuned on ReAct beat every prompting method at 540B. Fine-tuning on CoT or Standard trajectories did much worse. The authors read this as the difference between memorizing facts, possibly made-up ones, and learning a skill for finding them. Keeping only the attempts that succeed and training on them is the bootstrapping recipe that §26.3 studies for reasoning (Zelikman et al. 2022).

FireAct (Chen et al. 2023) made this the main object of study, with a Google search tool on HotpotQA.

Fine-tuned agents also learned to adapt the length of their trajectories, answering some simple questions in a single round.

Fine-tuning on trajectories differs from fine-tuning on chats in four ways that matter in practice.

Most of the text is the world's. Observations dominate the tokens. In supervised learning, whether to train on them is a choice, not a law. Training on observations is maximum-likelihood learning of the environment's text, a model of the world. FireAct found that masking them had no consistent effect either way (its Appendix A.7). But training on them teaches the model to write tool results, which the stop sequence or the message format must then prevent, and it spends capacity on text the model will never be asked to write. The usual practice is to mask observations, as §14.3 masks tool results. In reinforcement learning, below, the choice stops being a choice.
Every step of a successful trajectory is imitated, including the wasted ones. A trajectory that reached the right answer after two pointless searches teaches the pointless searches too. Filtering by success selects episodes, not steps.
The model never sees its own mistakes. Demonstrations come from a stronger model or a careful human, so they rarely contain the states that the student's own errors lead to, and the student never learns how to recover from them. Its cost is worked out next.
Format and judgment are learned together. Few-shot prompting has to teach the format in the context, which is why small models prompted with ReAct did worst of all. Fine-tuning moves the format into the weights, which frees the context for the task.

Why cloned agents compound their errors

Behavior cloning is supervised learning of a policy from an expert's demonstrations, one pair of state and expert action at a time. Fine-tuning on successful trajectories is behavior cloning. A cloned agent is accurate on the states its expert visited. After its first mistake it is somewhere the expert never went, and there its error rate can be far higher. This mismatch between the states trained on and the states visited is called covariate shift. It is the agent-sized version of the exposure bias of §3.5, where a model trained only on correct prefixes must continue its own imperfect ones. Once the first mistake happens, mistakes stop being rare.

§3.5 stated the quadratic bound below for translation and named its fix, and §14.1 applied it to supervised fine-tuning. Here we prove it, show that it is tight, and prove the linear bound that training on the student's own states achieves. Agents are where the difference matters most, because their episodes run for many steps and the first bound grows with the square of the number of steps.

Here are numbers first. Suppose a cloned agent errs on 1% of the expert's states, and after its first mistake every remaining step is wasted. Over a 30-step task, the chance that step $t$ is already off track is $1 - 0.99^t$ , and the expected number of wasted steps is $\sum_{t=1}^{30}(1 - 0.99^t) = 4.23$ . Now suppose instead that the agent errs on 1% of its own states, and that each mistake costs one step before it is back on track. Then it wastes at most $30 \times 0.01 = 0.3$ steps on average. Same error rate, fourteen times the waste. Two things changed, whether a mistake can be repaired and on whose states the error rate is measured, and the theorem below separates them.

∴ Behavior cloning compounds quadratically

Take a task of $T$ steps. At step $t$ the agent is in state $s_t$ (for a language agent, the history $h_t$ ), takes action $a_t$ , and pays $\operatorname{cost}(s_t, a_t) \in [0, 1]$ , for example 1 for every step spent off track. Write $J(\pi)$ for the expected total cost of a policy $\pi$ , and $d_t^{\pi}$ for the distribution of $s_t$ when $\pi$ acts. The expert $\pi^*$ is deterministic. The student $\hat\pi$ , possibly random, disagrees with it with probability $\varepsilon_t = \mathbb{P}_{s \sim d_t^{\pi^*}}\big(\hat\pi(s) \ne \pi^*(s)\big)$ on the expert's states, and with probability $\varepsilon'_t = \mathbb{P}_{s \sim d_t^{\hat\pi}}\big(\hat\pi(s) \ne \pi^*(s)\big)$ on its own states. Their averages over the $T$ steps are $\varepsilon = \frac1T\sum_t \varepsilon_t$ and $\varepsilon' = \frac1T\sum_t \varepsilon'_t$ . Behavior cloning minimizes $\varepsilon$ , because its data are the expert's states. Finally, $Q^*_t(s, a)$ is the expected cost from step $t$ to the end of taking $a$ in state $s$ and following the expert afterward.

(Ross and Bagnell 2010) $J(\hat\pi) \le J(\pi^*) + T^2\varepsilon$ .
The quadratic is attained, up to a factor of 2. Suppose the expert pays nothing, the student errs with probability $\varepsilon$ at every step while it is still on the expert's path, and it pays 1 at the step of its first mistake and at every step after. Then $J(\hat\pi) - J(\pi^*) = \sum_{t=1}^{T}\big(1 - (1-\varepsilon)^t\big) = \frac{T(T+1)}{2}\varepsilon + O(T^3\varepsilon^2)$ .
(Ross, Gordon, and Bagnell 2011) Suppose one wrong action followed by the expert's play raises the cost to go by at most $u$ , that is, $Q^*_t(s, a) - Q^*_t(s, \pi^*(s)) \le u$ for every $t$ , $s$ , and $a$ . Then $J(\hat\pi) \le J(\pi^*) + uT\varepsilon'$ .

∂ Proof of the three parts

Part 1. Run the student and the expert on the same random draws of the world, so they are in the same state for as long as they have taken the same actions. Let $E_t$ be the event that they agree at steps 1 to $t$ . Before the first disagreement, the shared state is the expert's own state at that step. So a first disagreement at step $r$ requires the student to disagree with the expert at the expert's state $s_r$ , which is distributed as $d_r^{\pi^*}$ , and that has probability $\varepsilon_r$ . A union bound gives $\mathbb{P}(\text{not } E_t) \le \sum_{r \le t}\varepsilon_r \le T\varepsilon$ . On $E_t$ the two pay the same cost at step $t$ , and off it their costs differ by at most 1, so their expected costs at step $t$ differ by at most $T\varepsilon$ . Summing over the $T$ steps gives $T^2\varepsilon$ .

Part 2. The cost at step $t$ is 1 exactly when at least one of the first $t$ steps erred, which has probability $1 - (1-\varepsilon)^t = t\varepsilon + O(t^2\varepsilon^2)$ , and $\sum_{t=1}^{T} t = T(T+1)/2$ .

Part 3. Let $\pi_{1:t}$ play the student for $t$ steps and the expert afterward, so that $\pi_{1:T} = \hat\pi$ and $\pi_{1:0} = \pi^*$ . The difference telescopes,

J(\hat\pi) - J(\pi^*) = \sum_{t=1}^{T}\big(J(\pi_{1:t}) - J(\pi_{1:t-1})\big).

The two policies in the $t$ -th term act alike for $t - 1$ steps, so both pay the same costs up to then and both reach $s_t \sim d_t^{\hat\pi}$ . From there one takes the student's action and the other the expert's, and both follow the expert. The term is therefore $\mathbb{E}_{s \sim d_t^{\hat\pi}}\big[Q^*_t(s, \hat\pi(s)) - Q^*_t(s, \pi^*(s))\big]$ . It is 0 where the actions agree and at most $u$ where they differ, so it is at most $u\varepsilon'_t$ . Summing over $t$ gives $uT\varepsilon'$ . ∎

In plain words, part 1 charges each mistake for every step that remains, because nothing guarantees that the student can find its way back. Its first mistake comes after about $1/\varepsilon$ steps, so in a long episode many steps come after it. Part 3 charges each mistake only $u$ , the price of one wrong step that the expert could repair. But it counts mistakes where the student actually goes, so it is only as good as the student's training on its own states. The numbers at the start of this subsection are parts 2 and 3 at $\varepsilon = 0.01$ and $T = 30$ , and part 1 allows up to 9 wasted steps. When mistakes cannot be undone, $u$ can be as large as $T$ and nothing escapes the quadratic, one more reason §24.4 asks whether actions can be reversed.

Part 3 says what to fix. The student must be trained on the states it reaches, with labels for what to do there. Ross, Gordon, and Bagnell turned that into an algorithm.

For language agents each remedy has a direct form. Recovery demonstrations patch the data by hand. The DAgger recipe uses a stronger model as the expert. It runs the student, has the teacher write the right next step at states taken from the student's own trajectories, and fine-tunes on those corrections. On-policy distillation is the same idea token by token, with the student sampling and the teacher scoring every token it wrote. An interviewer who asks why supervised fine-tuning alone plateaus for agents wants to hear covariate shift, the $T^2\varepsilon$ against $uT\varepsilon'$ contrast, and one of these remedies.

Reinforcement learning on the student's own episodes, the subject of the rest of this section, goes one step further. It needs no expert at all, only a reward, and its training states are the student's own by construction, so the covariate shift is gone. What it gives up is the dense label, one right action for every state, in exchange for one number per episode. That is why assigning credit across turns becomes the hard problem below.

Reinforcement learning with environment tokens

Supervised trajectories teach a model to imitate a good agent. Reinforcement learning teaches it to succeed. The model samples its own episodes, and each episode gets a reward, one number such as an exact-match check, a passing test, or a verified final state. Training makes the high-reward episodes more likely. Everything in §15.3 to §18.4 applies, with one new fact to establish first. An agent's episode contains tokens that the policy did not write.

Start with the smallest case. The policy writes the token search, the world writes hit, and the policy writes answer. The probability of this episode is a product of three factors,

\pi_\theta(\texttt{search}) \cdot P_{\text{env}}(\texttt{hit} \mid \texttt{search}) \cdot \pi_\theta(\texttt{answer} \mid \texttt{search}, \texttt{hit}).

The middle factor belongs to the world, and changing $\theta$ cannot change it. So when we take the gradient of the log-probability, the middle term contributes nothing. That is the whole idea, and the rest of this part makes it exact.

Serialize a finished episode into tokens $w_1, \dots, w_N$ , as §24.1 did. The source mask is $m_i = 1$ if $w_i$ was sampled from $\pi_\theta$ and $m_i = 0$ if it came from the task, the chat template, or the environment. The prefix decides which source writes token $i$ , since the policy writes until it emits its stop token and the harness then writes the observation. Let $P_{\text{env}}(w_i \mid w_{\lt i})$ be the probability that the outside world produces token $w_i$ after the prefix $w_{\lt i}$ when it is the world's turn.

This conditional does not depend on $\theta$ , and here is why. The world's hidden state $s$ , such as the contents of a database, enters only through the posterior $\mathbb{P}(s \mid w_{\lt i})$ . The policy sees only the prefix, so its probabilities $\pi_\theta(\cdot \mid w_{\lt i})$ are the same factor in $\mathbb{P}(s, w_{\lt i})$ for every $s$ , and they cancel when we normalize over $s$ . The chain rule then splits the probability of the episode into the policy's tokens and the world's,

p_\theta(w_{1:N}) = \prod_{i=1}^{N} \pi_\theta(w_i \mid w_{\lt i})^{m_i}\, P_{\text{env}}(w_i \mid w_{\lt i})^{1 - m_i}.

∴ Environment tokens carry no policy gradient

Let $R$ be the episode's reward and $J(\theta) = \mathbb{E}_{w \sim p_\theta}[R(w)]$ the expected reward. Then

\nabla_\theta \log p_\theta(w_{1:N}) = \sum_{i=1}^{N} m_i\, \nabla_\theta \log \pi_\theta(w_i \mid w_{\lt i}), \qquad \nabla_\theta J = \mathbb{E}_{w \sim p_\theta}\Big[R(w) \sum_{i=1}^{N} m_i \nabla_\theta \log \pi_\theta(w_i \mid w_{\lt i})\Big],

and a baseline $b(w_{\lt i})$ may be subtracted from $R$ in the term of any policy token without changing the expectation.

∂ Proof

Take the log of the factorization. The terms with $m_i = 0$ are $\log P_{\text{env}}(w_i \mid w_{\lt i})$ , which do not depend on $\theta$ , so their gradient is zero. The terms with $m_i = 1$ give the sum shown. The score-function identity of §15.3, $\nabla_\theta J = \mathbb{E}[R\,\nabla_\theta \log p_\theta]$ , gives the second formula. For the baseline, condition on the prefix $w_{\lt i}$ of a policy token. Then $w_i \sim \pi_\theta(\cdot \mid w_{\lt i})$ , and the expected-score lemma of §15.4 gives $\mathbb{E}[\nabla_\theta \log \pi_\theta(w_i \mid w_{\lt i}) \mid w_{\lt i}] = 0$ . So $\mathbb{E}[b(w_{\lt i})\nabla_\theta \log \pi_\theta(w_i \mid w_{\lt i})] = 0$ . ∎

In plain words, the world's tokens are context, never targets. They shape the gradient through attention, since every later policy token is conditioned on them, but they contribute no terms of their own. This is the "environment drops out" step of §15.3, applied token by token inside one sequence. The one assumption to check is that the world's text does not depend on $\theta$ . A simulated user played by a frozen copy of the model satisfies it. A user played by the policy itself does not, and then those tokens are the policy's own moves in a two-player game.

Leaving the world's tokens in the loss teaches wishful thinking

The mask is often described as a trick for stability. It is more than that. Leaving environment tokens in the loss changes what the estimator estimates.

∴ Training on environment tokens is biased toward wishful thinking

The unmasked estimator adds the terms $R(w)\sum_i (1 - m_i)\nabla_\theta \log \pi_\theta(w_i \mid w_{\lt i})$ . Their expectation is not zero in general, because the expected-score lemma needs $w_i \sim \pi_\theta$ , and here $w_i \sim P_{\text{env}}$ . Fix a prefix whose next token is the world's. Let the model's prediction of that token be $\pi_\theta(\cdot \mid w_{\lt i}) = \operatorname{softmax}(\mathbf{u})$ over candidate tokens with logits $\mathbf{u}$ , and let $\bar R(o) = \mathbb{E}[R \mid w_{\lt i}, w_i = o]$ be the expected reward given that the world writes $o$ . Then the added term pushes the logits $\mathbf{u}$ in the direction

\sum_{o} P_{\text{env}}(o \mid w_{\lt i})\,\bar R(o)\,\big(\mathbf{e}_o - \operatorname{softmax}(\mathbf{u})\big),

where $\mathbf{e}_o$ is the one-hot vector of $o$ . For a nonnegative reward with $\mathbb{E}[R \mid w_{\lt i}] > 0$ , this push vanishes only when the model's prediction of the world equals the reward-tilted distribution

\pi_\theta(o \mid w_{\lt i}) = \frac{P_{\text{env}}(o \mid w_{\lt i})\,\bar R(o)}{\mathbb{E}[R \mid w_{\lt i}]}.

∂ Proof

Condition on the prefix and use $\nabla_{\mathbf{u}} \log \operatorname{softmax}(\mathbf{u})_o = \mathbf{e}_o - \operatorname{softmax}(\mathbf{u})$ (§0.2). The expected added term is $\sum_o P_{\text{env}}(o)\,\bar R(o)\,(\mathbf{e}_o - \operatorname{softmax}(\mathbf{u}))$ . Its $o$ -th entry is $P_{\text{env}}(o)\bar R(o) - \pi_\theta(o)\sum_{o'} P_{\text{env}}(o')\bar R(o')$ . That is zero for every $o$ exactly when $\pi_\theta(o) = P_{\text{env}}(o)\bar R(o)/\mathbb{E}[R \mid w_{\lt i}]$ . ∎

In plain words, with a constant reward the extra term is ordinary maximum-likelihood learning of the world, harmless if wasteful. With a real reward it teaches the model to expect the observations that came with success. If searches that found the answer earned reward and searches that failed did not, the model learns that searches find the answer. It learns to predict good news, and a model that predicts tool results it never received is a model that makes them up.

A baseline makes this worse, not better. With the usual baseline $b = \mathbb{E}[R \mid w_{\lt i}]$ , the coefficient of $\operatorname{softmax}(\mathbf{u})$ in the added term is $\mathbb{E}[R \mid w_{\lt i}] - b = 0$ . The term becomes the constant vector $\sum_o P_{\text{env}}(o \mid w_{\lt i})(\bar R(o) - b)\,\mathbf{e}_o$ . It never vanishes unless $\bar R$ is the same for every observation, and it pushes the logits of the above-average observations up without limit.

The cell checks both theorems exactly on a world small enough to enumerate. The policy chooses one of three queries $a$ . The world returns a hit or a miss, with probabilities that depend on the query. Then the policy answers. The reward is 1 only if the search hit and the policy used the result. The same model also has logits for predicting the observation token, as any language model does. Those logits never influence what happens, so the true gradient of $J$ with respect to them is exactly zero.

Python · cell 10

The masked estimator matches the true gradient to five decimals, as the first theorem says. The unmasked estimator adds a nonzero push on logits that do not affect the reward at all. Follow that push for a few hundred steps, holding the rest of the policy fixed, and watch what the model comes to believe about its search tool.

Python · cell 11

An interviewer who asks "which tokens go in the agent's RL loss?" wants the answer and both halves of its proof, the factorization that makes the world's tokens drop out, and the bias toward good news that appears when they are left in.

The multi-turn GRPO objective

With the gradient settled, the training objective is the group-relative policy optimization (GRPO) of §18.4, applied to whole episodes. Here it is in one paragraph. For one prompt, sample $G$ responses with rewards $r_1, \dots, r_G$ . Each response's advantage is its reward minus the group mean, divided by the group's standard deviation, so a response is pushed up when it beat its siblings. Each token's update uses the probability ratio between the current policy and the policy that sampled it. That ratio is clipped to a small range as in PPO (§15.6), so one batch cannot move the policy far. In an agent, a response is an episode.

📐 Multi-turn GRPO with masked environment tokens

For a task $x$ , sample $G$ episodes from $\pi_{\theta_{\text{old}}}$ , interacting with the environment. Episode $j$ is serialized as tokens $w_{j,1}, \dots, w_{j,N_j}$ with source mask $m_{j,i}$ , and it earns reward $R_j$ . Its advantage is

\hat A_j = \frac{R_j - \bar R}{\operatorname{std}(R_1, \dots, R_G) + \epsilon_{\text{num}}}, \qquad \bar R = \frac{1}{G}\sum_{l=1}^{G} R_l,

with the sample standard deviation (dividing by $G - 1$ , as in §18.4) and a small constant $\epsilon_{\text{num}}$ . The same advantage applies to every policy token of episode $j$ . With the per-token ratio $\rho_{j,i}(\theta) = \pi_\theta(w_{j,i} \mid w_{j,\lt i})/\pi_{\theta_{\text{old}}}(w_{j,i} \mid w_{j,\lt i})$ and clip range $\epsilon$ , the objective to maximize is the clipped surrogate averaged over policy tokens only,

J(\theta) = \mathbb{E}\Bigg[\frac{1}{\sum_{j=1}^{G}\sum_{i=1}^{N_j} m_{j,i}} \sum_{j=1}^{G}\sum_{i=1}^{N_j} m_{j,i}\,\min\Big(\rho_{j,i}(\theta)\hat A_j,\ \operatorname{clip}\big(\rho_{j,i}(\theta), 1 - \epsilon, 1 + \epsilon\big)\hat A_j\Big)\Bigg].

In plain words, run the same task $G$ times and score each run against its siblings. Push every token the policy wrote in a run up or down by that run's score, clipped as in PPO, and ignore every token the world wrote.

The normalization here is one average over all policy tokens of the group, the token-level choice of DAPO (Yu et al. 2025). The two alternatives are GRPO's original average within each episode and a constant divisor (Liu et al. 2025, Dr. GRPO), with the length and difficulty biases that §18.4 derived for each. Agents make this choice matter more than reasoning did, because episode lengths vary by an order of magnitude, for example a three-turn success next to a thirty-turn failure. Under a per-episode average, each policy token of the short episode weighs ten times as much as one of the long episode. Some trainers also add a KL penalty, a term that keeps the policy close to a fixed reference model. If one is used, it is masked the same way, and Search-R1, below, masks its retrieved tokens in the KL term too.

Interview

Masked multi-turn GRPO loss

Write multiturn_grpo_loss(logp, logp_old, mask, rewards, group, clip_eps=0.2, eps_num=1e-6).

Inputs. logp and logp_old have shape (B, T). They hold the log-probabilities of the tokens of $B$ episodes, padded to length $T$ , under the current policy and under the sampling policy. mask has shape (B, T), 1 at policy tokens and 0 at environment tokens and padding. The log-probabilities at masked positions may be arbitrary garbage. rewards has shape (B,). group has shape (B,) and holds integer task ids. Episodes with the same id are rollouts of the same task, at least two per task, in any order.
Advantages. adv holds each episode's GRPO advantage within its own group, using the sample standard deviation (ddof=1) plus eps_num.
Loss. loss is a float, the negative of the definition's surrogate with the token average pooled over the whole batch, as DAPO does when a batch holds several tasks, $-\sum_{j,i} m_{j,i}\min(\rho\hat A_j, \operatorname{clip}(\rho, 1 - \epsilon, 1 + \epsilon)\hat A_j)/\sum_{j,i} m_{j,i}$ .

Return (loss, adv).

import numpy as np

def multiturn_grpo_loss(logp, logp_old, mask, rewards, group, clip_eps=0.2, eps_num=1e-6):
    rewards = np.asarray(rewards, float)
    adv = (rewards - rewards.mean()) / (rewards.std() + eps_num)
    ratio = np.exp(logp - logp_old)
    surr = ratio * adv[:, None]
    return float(-surr.mean()), adv

Tests, hints, verified solutions, and AI help, with Pro at $7.99 a month.

The loss in PyTorch

In a real trainer the log-probabilities come from the model, and autograd does the rest. The block below computes the masked multi-turn loss from logits and checks two things. At $\theta = \theta_{\text{old}}$ the ratio is 1, so the gradient with respect to each policy position's logits is $-\hat A_j(\mathbf{e}_{w} - \operatorname{softmax})/\sum m$ , the masked policy gradient of the theorem. And the gradient at every environment position is exactly zero.

🔥 In PyTorchfor a machine with PyTorch

What masking measured at scale

Search-R1 (Jin et al. 2025) trained Qwen2.5 models to interleave <think>, <search>, and <answer> blocks, with retrieved passages inserted between <information> tags. The reward was exact match, and the optimizer was PPO or GRPO. It masks the retrieved tokens from the policy loss and from the KL term, and it tested the choice. With Qwen2.5-7B-base and PPO, the average exact match over seven question-answering benchmarks was 0.431 with the mask and 0.343 without it, and the mask won on every one of the seven. Overall, Search-R1 improved on retrieval-augmented baselines by 24% (Qwen2.5-7B) and 20% (Qwen2.5-3B) in relative terms.

ReTool (Feng et al. 2025) trained a model to run code during long reasoning, with the interpreter's output masked from the loss in the same way. Starting from Qwen2.5-32B-Instruct, it reached 67.0% on AIME 2024, a high-school mathematics competition, after 400 RL steps. A text-only RL baseline, fine-tuned on text reasoning first, reached 40.0% after 1,080 steps. Starting instead from DeepSeek-R1-Distill-Qwen-32B, a model distilled from a reasoning model, ReTool reached 72.5%.

One practical detail decides whether the theorem actually holds in code. The log-probabilities in the loss must be those of the exact tokens the policy sampled. Suppose the trainer re-tokenizes the episode's text. A tokenizer may merge characters across the boundary between the policy's output and an observation, so the "policy tokens" in the loss are not the tokens the policy produced. Their log-probabilities are then off-policy in a way that no importance ratio corrects. Agent RL frameworks therefore carry token ids, not strings, from the sampler to the trainer.

Credit across many turns

The group advantage gives every token of an episode the same number. In a 30-turn episode with one reward at the end, that is a blunt instrument. A run that wasted ten turns and then succeeded pushes up the wasted turns too, and only many episodes average out which turns mattered. Deciding which turns earned the reward is the credit assignment problem. There are four answers, in increasing order of machinery.

Better group baselines. LOOP (Chen et al. 2025) trained a 32B agent in AppWorld, a simulation of nine everyday apps driven by API calls. It used PPO with no value network, so only one copy of the model sits in memory. Each episode's baseline was the mean reward of the other $K - 1 = 5$ rollouts of its task, the leave-one-out estimator of §18.4, and the ratio was clipped per token. The reward was the fraction of the task's unit tests passed. LOOP completed 71.3% of the tasks in AppWorld's normal test split and beat the much larger OpenAI o1 agent by 9 points. The agents learned to read API documentation before calling, to avoid unwarranted assumptions, and to recover from failed calls.
A critic over turns. ArCHer (Zhou et al. 2024) runs two learners. A high-level critic, trained off-policy like a Q-function on whole turns, estimates the value of each turn. A low-level token policy is trained within each turn against that value. Turn-level values give each turn its own advantage, and the authors report about 100 times the sample efficiency of on-policy methods such as PPO on their agent tasks.
Process rewards. A process reward model scores intermediate steps, as in §18.6, and for an agent a step is a turn. Judging an agent's step is harder than judging a reasoning step, because its quality depends on a hidden state of the world.
Shaped rewards. These are small rewards for intermediate events, such as a well-formed call, a successful tool execution, or a correct sub-answer, studied for tool use by ToolRL (Qian et al. 2025). Every shaped reward is a new target for reward hacking, earning reward without doing the task. An agent paid for successful calls learns to make easy calls, and one paid for format learns format.

Multi-turn RL is also less stable than single-answer RL, because a policy that stops varying its behavior leaves its groups with identical rewards and no signal. The deeper box describes the measured failure.

🔬 The echo trap

RAGEN (Wang et al. 2025) named this failure the echo trap. The agent settles into one fixed, repetitive way of phrasing its reasoning, the reward variance within groups collapses, and gradient norms spike. Recall from §18.4 that a group with identical rewards carries no signal. With 0/1 rewards and success probability $p$ , a group of $G$ is mixed with probability $1 - p^G - (1 - p)^G$ . RAGEN's stabilized variant keeps only the tasks whose rollouts vary most in reward, adds a critic, and stabilizes the gradient, for example by clipping the ratio differently above and below 1. RAGEN also found that without rewards that look at the reasoning itself, multi-turn RL produced shallow strategies or made-up thoughts. The reward pays for outcomes, and the thoughts are free to drift.

Rollouts wait on tools

In RL for reasoning, generating a batch means running the model. In RL for agents, each rollout also waits for tools, such as a search engine, a code sandbox, or a browser, with latencies from milliseconds to minutes and long tails. If a training step waits for the whole batch, the slowest rollout sets the pace. Large systems therefore generate rollouts asynchronously, training on whatever episodes are ready. That makes training slightly off-policy, since some episodes were drawn by an older copy of the policy, and PPO's ratio corrects for it. The box gives the numbers.

🔬 Why agent RL generates asynchronously

A quick estimate shows how bad that is. Suppose rollout durations were independent exponentials with mean $\mu$ . Then the slowest of $n$ takes $\mu H_n$ on average, where $H_n = 1 + 1/2 + \dots + 1/n \approx \ln n + 0.577$ is the harmonic number (§24.6 proves this when it times parallel tool calls). A batch of 512 would take about $6.8\mu$ , while most of the hardware sat idle waiting for the last few. Real durations are sums of many calls and less extreme, but the direction holds.

In an asynchronous system, rollout workers keep sampling with a recent copy of the weights, finished episodes stream into a buffer, and the trainer consumes whatever is ready. The cost is that episodes come from a policy a few updates old, so training is slightly off-policy, learning from samples that an older policy drew. PPO's per-token ratio $\rho$ is exactly the correction for that (§15.6), and the clip bounds how far a stale sample can move the policy. Systems also cap staleness and discard episodes generated more than a few versions ago, because the correction degrades as the two policies drift apart.

Every environment must also reset cheaply, since one task needs $G$ fresh copies of its world per step. For software that means a container or a microVM, a small virtual machine that starts in a fraction of a second, per rollout, the subject of §24.5.

🔬 Systems details that matter more for agents than for single answers

Tool workers and caches. ReTool's sandbox ran as a pool of workers that take jobs as capacity frees up. To save memory during rollouts, it kept the cached keys and values computed before each code execution and computed new ones only for the interpreter's output.

The unit of the ratio. §18.4 showed that a per-token ratio reweights one sampled token at one context, which corrects nothing by itself and adds noise. GSPO (Zheng et al. 2025) replaces it with one length-normalized ratio per response, the geometric mean of its token ratios, clipped once. An agent episode makes the case stronger, since it holds thousands of policy tokens spread over many turns, each with its own noisy ratio, and staleness adds to each. By the factorization above, the environment's factors cancel in the episode's likelihood ratio, which is the product of the policy tokens' ratios. So GSPO's ratio for episode $j$ averages the log-ratios over its policy tokens only,

s_j(\theta) = \exp\Big(\frac{\sum_i m_{j,i}\log\rho_{j,i}(\theta)}{\sum_i m_{j,i}}\Big).

The sampler is not the trainer. The sampler and the trainer disagree even with identical weights, because the inference engine's kernels, precision, and batching differ from the trainer's. So the tokens were not drawn from the distribution the trainer computes. Systems therefore multiply each policy token's term by a capped importance weight, computed from the log-probabilities that the sampler recorded,

\min\!\Big(\frac{\pi_{\text{train}}(w_i \mid w_{\lt i})}{\pi_{\text{sampler}}(w_i \mid w_{\lt i})},\ C\Big).

This is truncated importance sampling (Yao et al. 2025, introduced in §18.4). The cap $C$ bounds the variance at the price of a small bias. Qi et al. (2025) used $C = 3$ when they compared this and other corrections, and found that running both engines in FP16 removed much of the mismatch at its source (§8.5). Either way, the sampler's log-probabilities must travel with the token ids from the rollout workers to the trainer.

Environments at scale

Frontier labs train agents on thousands of environments, most of them synthesized. An environment here is a task, the tools an agent may call, a simulated or real world those tools act on, and a check that decides whether the task was done.

The published recipes share three moves. First, tools and users are simulated by language models, so that thousands of tasks can run in parallel and reset for free, with real sandboxes kept where fidelity matters, as in coding. Second, every task comes with a check, such as a rubric, a verification function, or unit tests. Third, tasks are filtered to the right difficulty, kept only if the current model solves them sometimes, so that every group of rollouts has mixed rewards and carries signal. The deeper box gives the numbers for two published recipes.

🔬 Two published recipes for agent environments

Kimi K2 (Kimi Team 2025) built its tool-use data from more than 3,000 real tools of the Model Context Protocol and more than 20,000 synthetic ones. It generated thousands of agents with different system prompts and tool sets, wrote tasks with explicit grading rubrics, and simulated both the users and the tools. Its tool simulator keeps state between calls, which makes it a model of the world, and adds controlled randomness to produce partial failures. A language model acting as judge kept only the trajectories that met each task's rubric, and real sandboxes replaced simulation for coding. The resulting model has 1 trillion total parameters, of which 32 billion are active per token. It scored 65.8 on SWE-bench Verified (real GitHub issues graded by the repository's tests) and 66.1 on Tau2-Bench (customer-service tasks with a simulated user), without extended thinking.

DeepSeek-V3.2 (DeepSeek-AI 2025) synthesized 1,827 environments with 4,417 tasks for a general agent. Each environment is a database, a set of tool functions, a task, and a verification function, all written by an agent and checked against a solution. It kept only the tasks the model solved at least once in 100 tries, so every task gives some learning signal. With real environments for code (24,667 tasks mined from GitHub issues), search (50,275 synthesized questions), and a code interpreter (5,908), that is about 85,000 prompts for agent RL. It also keeps the model's reasoning across tool calls within a turn and discards it only when a new user message arrives. Discarding it at every tool result forced the model to re-derive its whole plan each time.

The 50,275 search questions were built by inversion, the method described below for BrowseComp, run by agents. A question-writing agent explored rare entities with search. Other agents, with different checkpoints and prompts, answered each question. A verifying agent kept only the questions whose reference answer checked out and whose candidate answers were all verifiably wrong. So every kept question defeated the answering agents and can still be graded. The model scored 51.4 on BrowseComp, and 67.6 with the context management of §25.2.

Browsing agents are the most visible result of this recipe. Their benchmark, BrowseComp (Wei et al. 2025), is a set of 1,266 questions from OpenAI that are hard to find and easy to check, and the deeper box after its results shows how such questions are built.

🔬 How BrowseComp questions are built

Human trainers started from a seed, such as a person, an event, or a paper, and wrote an inverted question. It stacks several properties, each with a large search space, until the seed is the only thing that fits all of them. The paper's sample asks for an EMNLP paper from 2018 to 2023 whose first author did their undergraduate degree at Dartmouth and whose fourth author did theirs at the University of Pennsylvania. Each question had to defeat the models of the time and five simple searches, and the answer is a short string that a grader compares with the reference.

The inversion is also a data recipe. A fact chosen first comes with a checkable answer for free, and the stacked constraints set the difficulty. The same property invites a hack. An answer that is easy to check is easy to find once it leaks, so an agent graded on a public benchmark can search for the benchmark itself. That is why BrowseComp ships a canary string, a unique marker that lets crawlers filter it out, and asks that its examples never be posted online.

Step back from the individual systems. Each one trained a model on its own episodes against a reward that a program, a rubric, or a reference answer could check. The table puts their numbers side by side, with what each one trained, so you can quote them in an interview with their context.

📌 Agent RL, measured

system	what was trained	result
Search-R1	search interleaved with reasoning, retrieved tokens masked	+24% (7B) and +20% (3B) over retrieval baselines; 0.431 with the mask against 0.343 without
ReTool	code execution inside reasoning, interpreter output masked	AIME 2024 67.0% in 400 steps, against 40.0% in 1,080 for text-only RL
LOOP	AppWorld agent, PPO with a leave-one-out baseline, no critic	71.3% of normal test tasks, 9 points above OpenAI o1
Kimi K2	synthetic tools, users, and rubric tasks, then RL	65.8 SWE-bench Verified, 66.1 Tau2-Bench, no extended thinking
DeepSeek-V3.2	1,827 synthesized environments plus real code and search	about 85,000 agent RL prompts; BrowseComp 51.4
Deep Research	RL on hard browsing tasks	BrowseComp 51.5%, against 1.9% for GPT-4o with browsing

ReAct inside frontier models

ReAct began as a prompt format. In frontier models it has become a trained behavior inside the reasoning itself. OpenAI's system card for o3 and o4-mini (April 2025) states that the models use tools within their chains of thought, for example searching the web or running Python while they think. It also states that they are trained with large-scale reinforcement learning on chains of thought. Anthropic's API documents interleaved thinking, in which Claude reasons about each tool result before it decides what to do next, between tool calls within one assistant turn. The thinking blocks are passed back with the tool results so that the reasoning can continue. Both are ReAct's cycle of thought, action, and observation, with the thoughts now long, learned by RL, and hidden from or summarized for the user.

When a tool call is worth it

Every agent in this section, prompted or trained, makes one decision over and over. The agents above search until they can answer, but a real agent must decide at every turn whether a call is worth its cost in latency, tokens, and money, or whether to answer from what it already knows. A two-line model makes the decision exact and shows why calibration matters for tool use. An agent is calibrated if, among the answers it gives with confidence 0.7, about 70% are right.

Start with numbers. Suppose the agent is 60% sure of its own answer, a search tool is right 90% of the time, and a call costs 0.1 in units where a correct answer earns 1. Answering alone earns 0.6 on average. Calling and using the tool's answer earns $0.9 - 0.1 = 0.8$ . So call. If the agent were 85% sure, answering alone would earn 0.85, more than 0.8, so it should not call.

∴ The value of a tool call

Suppose the agent has an answer of its own that is correct with probability $c$ , and $c$ is calibrated. A tool call costs $\lambda > 0$ in units of reward and returns an answer that is correct with probability $q$ , independently of the agent's. A correct final answer earns 1. (In this box $\lambda$ is a cost and $q$ an accuracy, not the failure hazard and repair probability of §24.1.)

If the agent submits the tool's answer whenever it calls, calling is optimal if and only if $q - c > \lambda$ , that is, if and only if $c \lt q - \lambda$ .
If the agent may compare the two answers and submit either one, and two wrong answers never coincide, then the expected reward after calling is $\max(q, c) - \lambda$ , so the rule is the same as in 1.
Suppose an overconfident agent reports $\hat c = g(c)$ , with $g$ increasing and $g(c) \ge c$ , and applies the rule of 1 to $\hat c$ . Then it calls only when $c \lt g^{-1}(q - \lambda)$ , which is never more often than optimal. Its expected regret per question is $\mathbb{E}\big[(q - \lambda - c)\,\mathbb{1}[g^{-1}(q - \lambda) \le c \lt q - \lambda]\big] \ge 0$ .

∂ Proof of the three parts

Part 1. Not calling earns $c$ in expectation and calling earns $q - \lambda$ , so calling is better exactly when $q - \lambda > c$ .

Part 2. After calling, the two answers agree only if both are right, because wrong answers never coincide. That happens with probability $cq$ . They disagree with probability $1 - cq$ . Then the agent's own answer is the right one with probability $c(1 - q)$ and the tool's with probability $q(1 - c)$ . So the best choice on disagreement is the tool's answer if $q(1 - c) > c(1 - q)$ , which simplifies to $q > c$ . The expected reward is therefore $cq + \max\big(q(1-c), c(1-q)\big) - \lambda$ . That equals $q - \lambda$ when $q \ge c$ and $c - \lambda$ when $c > q$ , that is, $\max(q, c) - \lambda$ . Calling beats not calling if and only if $\max(q, c) - c > \lambda$ . With $\lambda > 0$ this needs $q > c$ , and then it reads $q - c > \lambda$ .

Part 3. The agent calls if and only if $g(c) \lt q - \lambda$ , which for increasing $g$ means $c \lt g^{-1}(q - \lambda)$ . And $g(c) \ge c$ gives $g^{-1}(q - \lambda) \le q - \lambda$ . So the overconfident agent calls on a subset of the questions where calling is optimal. On the questions it skips, those with $g^{-1}(q-\lambda) \le c \lt q - \lambda$ , it earns $c$ instead of $q - \lambda$ . ∎

In plain words, call the tool when it is better than you by more than it costs, and the honesty of your own confidence decides whether you can follow that rule. Comparing the two answers buys nothing extra in this model. Overconfidence makes an agent skip exactly the calls it needed, which is how a fluent model ends up answering from a wrong memory with a search tool right there.

✏️ Overconfidence, priced

Let the agent's true confidence be uniform on $[0, 1]$ across questions, the tool be right with $q = 0.9$ , and a call cost $\lambda = 0.1$ , so the optimal rule calls when $c \lt 0.8$ . The optimal policy calls 80% of the time and earns

\mathbb{E}[\max(c, 0.8)] = 0.8 \cdot 0.8 + \int_{0.8}^{1} c\,\mathrm{d}c = 0.64 + 0.18 = 0.82

per question. An agent that reports $\hat c = \sqrt{c}$ calls only when $\sqrt{c} \lt 0.8$ , that is, when $c \lt 0.64$ . It calls 64% of the time and earns $0.64 \cdot 0.8 + \int_{0.64}^{1} c\,\mathrm{d}c = 0.512 + 0.2952 = 0.8072$ . The regret is $\int_{0.64}^{0.8}(0.8 - c)\,\mathrm{d}c = (0.8 - 0.64)^2/2 = 0.0128$ per question, which matches $0.82 - 0.8072$ .

The model is simple on purpose, and its lesson survives the complications. Calibration (§19.8) is not only an honesty property. It is what makes a tool policy work. And when a call has a real cost, training should reward correctly skipping a call as well as making a correct one. That is why tool-use training data includes examples where the right move is not to call (§24.2), and why RL rewards for tool use often charge for each call.

Research

When is a tool call worth it?

Work with a population of questions, given as an array c of the agent's true probabilities of answering correctly on its own, each question weighted equally. The tool is right with probability q and costs lam. The agent calls on a question if and only if its reported confidence is below a threshold t, and when it calls it submits the tool's answer. Write three functions.

tool_call_utility(c, t, q, lam, reported=None) returns the mean reward per question. reported (default c) is the confidence the agent uses for the decision, while the reward of not calling is always the true c.
best_call_threshold(c, q, lam, grid) returns the threshold in grid with the highest utility for a calibrated agent, the first one if several tie.
overconfidence_regret(c, q, lam, g) returns the utility of the optimal calibrated policy (threshold q - lam on c) minus the utility of an agent that applies the same threshold to g(c).

import numpy as np

def tool_call_utility(c, t, q, lam, reported=None):
    raise NotImplementedError

def best_call_threshold(c, q, lam, grid):
    raise NotImplementedError

Tests, hints, verified solutions, and AI help, with Pro at $7.99 a month.

Where it breaks

Each way of producing an agent has its own data, its own cost, and its own typical failure.

method	data it needs	what it learns	cost	typical failure
ReAct prompting	a few written trajectories	the format only, from the context	nothing to train, long prompts	loops, derailment by bad searches
Toolformer	plain text plus tools that run	when a single call helps the next tokens	sampling and running many calls	no chains, no interaction, sensitive to wording
Trajectory fine-tuning (FireAct)	hundreds to thousands of good trajectories	the format and habits of a strong agent	cheap, offline	imitates wasted steps, never sees its own errors
Multi-turn RL (Search-R1, LOOP)	tasks, environments, and a checker	whatever earns reward	rollouts wait on tools, many environments	reward hacking, unstable groups, credit over turns

In practice the rows build on each other. The published recipes (ReTool, Kimi K2, DeepSeek-V3.2) fine-tune on trajectories before RL. RL then starts from a policy that can already finish some tasks, so its groups have mixed rewards and carry signal.

⚠️ How ReAct agents and their training go wrong

Trusting the model's own observations. Without a stop sequence, or a message format that separates turns, a model writes Observation: lines itself and then reasons from them. Parse only up to the first action, and let only the harness write observations.
Uninformative observations. A search that returns nothing useful leaves the next thought with nothing to work on, and ReAct agents often never recover (23% of ReAct's failures). Return informative failures, such as similar titles, the error message, or the valid options, never a bare "no results".
Observations that flood the context. One web page can be 20,000 tokens. Truncate it with a marker, return the relevant part, or let the agent page through results, because every later turn re-reads the context (§24.1). Managing the context over long runs is the subject of §25.2.
Leaving observations in the RL loss. The estimator becomes biased toward predicting rewarding observations, and the model learns to make up tool results. Mask them, in the KL term too, and carry token ids rather than re-tokenized text into the trainer.
Rewards that can be earned without doing the task. A search agent graded on a public benchmark may find the benchmark's answers online, and a coding agent graded by tests may edit the tests (§18.3, and §24.5 for software). Block the shortcuts in the environment, and check final states rather than claims.
Dividing by the wrong count. Averaging the loss over all positions, padding and observations included, silently rescales each episode by its share of policy tokens.

🧭 What to remember

ReAct lets the policy act in $\hat{\mathcal{A}} = \mathcal{A} \cup \mathcal{Z}$ , where $\mathcal{Z}$ holds the thoughts. A thought changes only the context, an action queries or changes the world, and an observation brings the world back. Thoughts supply computation and a written plan, and observations supply facts that recall cannot.
On HotpotQA, chain of thought fails mostly by making up facts (56% of failures) and ReAct by loops and useless searches (47% and 23%). So ReAct alone trails chain of thought, and the back-off combinations beat both (35.1 and 34.2, against 33.4 for CoT-SC).
A robust loop parses only up to the first action, turns every failure into an informative observation, detects repeated actions, and backs off after a step limit. A call is worth making when the tool beats the model by more than its cost, $q - c > \lambda$ , which needs calibrated confidence.
Toolformer keeps a self-sampled call when its result saves at least $\tau_f$ nats on the next five tokens, $L_i^- - L_i^+ \ge \tau_f$ , with $L_i^-$ the better of no call and the call without its result. It teaches single helpful calls, not chains or interaction.
Fine-tuning on successful trajectories teaches format and habits (FireAct, +77% for Llama-2-7B), with behavior cloning's blind spot for the student's own mistakes. Its cost can grow as $T^2\varepsilon$ with the horizon, against $uT\varepsilon'$ when the student is trained on its own states, as in DAgger and in RL.
Environment tokens carry no policy gradient, $\nabla_\theta \log p_\theta(w_{1:N}) = \sum_i m_i\nabla_\theta\log\pi_\theta(w_i \mid w_{\lt i})$ , and leaving them in the loss biases the model toward expecting rewarding observations. Multi-turn GRPO gives each episode one group-relative advantage and applies it to the policy's tokens only. At scale, the hard parts are credit over turns, unstable groups, rollouts that wait on tools, and building thousands of checkable environments.
Next, §24.4 asks when it pays to think ahead instead of step by step, and finds that search over actions needs a way to undo them.

Yao et al. (2023). ReAct: Synergizing Reasoning and Acting in Language Models. The augmented action space, the ablations against Act and CoT, the error analysis of Table 2, and the back-off rules; read the appendix trajectories.
Schick et al. (2023). Toolformer: Language Models Can Teach Themselves to Use Tools. Self-supervised tool-use data from the model's own loss, with an honest limitations section that set the agenda for later work.
Chen et al. (2023). FireAct: Toward Language Agent Fine-tuning. The systematic study of fine-tuning on agent trajectories, including data scale, mixtures of methods, and observation masking.
Ross, Gordon, and Bagnell (2011). A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning. DAgger, with the quadratic bound for behavior cloning and the linear bound for training on the learner's own states; Section 2 is short and worth reading in full.
Jin et al. (2025). Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning. Multi-turn RL with retrieved-token masking, its ablation, and the dynamics of learning when to search.
Chen et al. (2025). Reinforcement Learning for Long-Horizon Interactive LLM Agents. LOOP, a critic-free PPO for a stateful multi-app environment, with an analysis of the behaviors RL taught.
Wei et al. (2025). BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents. How inverted questions are built, checked, and graded, and the gap between browsing models and agents trained with RL.
DeepSeek-AI (2025). DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models. Section 3.2 describes thinking across tool calls and the synthesis of verifiable agent environments at scale.

Made for larger screens

ReAct and reasoning with actions

Three ways to answer one question

A thought is an action that only changes the context

Why a thought helps

What ReAct measured

A ReAct agent from scratch

Parse a ReAct step

The loop

One task, end to end

Error recovery belongs in the loop and in the policy

A ReAct agent with loop detection and a training mask

What each part buys, in a simulation

Toolformer, a model that labels its own data

Toolformer's filter

Fine-tuning on trajectories

Why cloned agents compound their errors

Reinforcement learning with environment tokens

Leaving the world's tokens in the loss teaches wishful thinking

The multi-turn GRPO objective

Masked multi-turn GRPO loss

The loss in PyTorch

What masking measured at scale

Credit across many turns

Rollouts wait on tools

Environments at scale

ReAct inside frontier models

When a tool call is worth it

When is a tool call worth it?

Where it breaks

Read more

Watch next