How a language model becomes an agent that calls tools, plans, writes and runs code, operates a computer, and works with other agents, built from the agent loop up and taken to the depth a frontier lab interview expects.

An agent is a language model that works in a loop. It writes a request to run a tool, such as a web search or a database query. A program runs the tool and hands the result back as text. The model reads the result and writes again, until it gives a final answer or runs out of budget. This loop is how a model fixes a bug, books a meeting, or answers a question from live data instead of from memory. It is the object the whole chapter studies.

The loop rests on four ideas from earlier in the book. A model follows instructions and examples written into its prompt, with no change to its weights (§6.8), which is how it learns what tools exist. Retrieval-augmented generation put fetched documents into the prompt (§7.6), but there a fixed program decided what to fetch, once. A model can write out its reasoning before it answers (§18.1). And generating text can be treated as a policy, a rule that picks the next token given the text so far, trained with rewards (§15.7). An agent lets the model decide, turn after turn, what to look up or do next, and this chapter works out what that one change costs and buys.

The loop raises one hard question, and this section answers it. Suppose a model gets each step of a task right 95% of the time, which sounds excellent. A task of 20 steps, any one of which can ruin the result, then succeeds only $0.95^{20} = 0.358$ of the time. Two tasks in three fail. At 100 steps it succeeds 0.6% of the time. Yet in February 2026 the best measured model finished, about half the time, software tasks that take a skilled person around 12 hours. Such tasks have far more than 20 steps. The simple calculation is missing something, and by the end of the section you will know what.

We first follow the loop through two real tasks, a weather lookup and a data question. Then we state it precisely, as a policy acting in a world it can only partly see, and write it in code that never crashes. We count its tokens, which grow with the square of the number of turns. We work out how errors compound and what checking and repairing buys back. Then we read the measured time horizons of frontier agents through a small theorem, and end with how much freedom to give an agent.

The loop in plain words

A tool is a function that a program runs on the model's behalf, such as a search, a calculator, or a SQL query. An agent uses tools by running four steps over and over.

1. The model writes some text.
2. A parser, a small program, looks in that text for an action, such as a call to a tool with its arguments.
3. If it finds one, the environment runs the action and returns a result as text, called an observation.
4. The observation is appended to the context, and the model writes again, now seeing what happened.

The loop ends when the model writes a final answer or when a budget runs out. The environment means everything outside the model that the agent acts on, the tools and the files, databases, web pages, and people behind them.

Four things are new compared with plain text generation, and the next sections take them up in turn.

• The text must be turned into an action by a parser, and parsing can fail. §24.2 makes it reliable.
• Some tokens in the context are written by the world, not by the model. That matters for training, which §24.3 works out.
• Actions can change the world, so they cannot always be undone (§24.4 and §24.5).
• Every turn re-reads a context that keeps growing, so cost grows faster than the number of turns. §24.6 puts a price on it.

One piece of background makes the transcripts below readable. A chat template is the fixed format that turns a conversation into one string of tokens (§14.3). A common one, Qwen's, writes the tool definitions into the system message. It writes a tool call as an assistant turn with JSON between <tool_call> tags, and the tool's result as a separate turn. The model's turn ends when it emits an end-of-turn token or a stop sequence, a string that halts sampling (§6.4).

One task, followed turn by turn

The weather lookup is tiny. Real agent tasks are longer, and some of their costs only show at that length. Here is a question of the kind data agents answer every day, followed through all eight turns.

The cell below recomputes those totals from the table, so you can see where each number comes from.

The widget plays the same episode. The bar on top is the context, and each column on the right is the input one turn reads, colored by who wrote it. Step through it and watch three numbers. The context grows every turn. The total of input tokens read grows faster, because every turn re-reads everything before it. And the share of tokens the model actually wrote stays small. Lower the step budget below 8 to see the episode stop before its answer.

What changes when a model meets an environment

Reinforcement learning has the vocabulary for this, so here it is in brief (§15.1). A Markov decision process, or MDP, models a sequence of decisions. At each step the agent sees a state $s$ and picks an action $a$. The world moves to a new state $s'$, drawn from a transition distribution $P(s' \mid s, a)$, and pays a reward. A policy $\pi(a \mid s)$ is a rule for choosing actions. The goal is the largest expected total reward.

§15.7 cast text generation as an MDP. The state $s_t = (x, y_{\lt t})$ is the prompt $x$ plus the tokens $y_{\lt t}$ generated so far. The action is the next token, and the transition appends it. That process is unusually friendly. The model sees the whole state. The transition is known and deterministic. And the policy can go back to any earlier prefix for free. An agent breaks each of these. The table below takes the rows of that section's comparison that matter here, adds a row for resets, and adds a column for agents.

Each changed row has a price.

• Hidden state. The agent acts on what it has observed, so it can be wrong about the world without knowing it. A file it read ten turns ago may have changed. This is the formal reason agents need verification steps.
• Unknown, random transitions. Part of an episode's probability now belongs to the environment, so the model cannot compute it by itself. Training must work with sampled episodes. Evaluation must run each task several times to see the spread, which is the pass$^k$ metric near the end of this section.
• No reset. Search methods branch from a saved prefix, and §18.6 does that freely over reasoning steps. When actions touch the world, branching needs a way to undo them. That is why §24.4 finds that search over actions needs reversibility or a simulator, and why training environments are sandboxes that can be saved and restored.
• Environment tokens in the context. The model reads text it did not write. Training on that text would teach the model to imitate the environment, which is how a model learns to invent tool results. Trusting it blindly is how instructions hidden in a web page or a file take control of an agent, the prompt injection of §19.3.
• Side effects. A wrong token in a chat costs a wrong sentence. A wrong action can cost money, data, or trust. That is why autonomy is a design decision rather than a default.

The agent as a policy over histories

Now the picture becomes definitions. They are short, and the rest of the chapter uses each of them many times.

First the environment, in plain words. The world has a true state that the agent never sees directly. Each action changes the world, possibly at random, and returns a piece of text about the new state. A decision process of this kind is called partially observable.

The agent cannot see the world, so all it can act on is what has happened so far. That is the task, its own earlier actions, and what came back from each. This record is called the history.

In the weather transcript, $x$ is the question about Paris, $a_1$ is the call with "unit": "celsius", $o_1$ is the error message, and "I need the live temperature." is the thought $z_1$. Two remarks make the definition earn its keep.

First, the policy sees the history, never the state, and that is enough. The best possible agent would keep a belief, a probability for each possible state of the world given everything observed so far, $b_t(s) = \mathbb{P}(s \mid h_t)$. The belief is computed from the history alone, so a policy over histories can do anything a policy over beliefs can. The thoughts carry no new evidence about the world, because the agent wrote them from what it already had, so $b_t$ is the same with or without them. Some chat templates drop the reasoning of earlier turns, and the history the model sees is then the version without thoughts. The context is therefore the agent's evidence about the world, not the world itself.

Evidence can go stale, as when a file changes after it was read. It can also be lost, when a long context is cut or summarized (§25.2). Good agents re-read before they write and verify before they finish for exactly this reason.

Second, $\pi_\theta(a_t \mid h_t)$ is the probability of the action, not of the text. Many different texts parse to the same action, with different thoughts or different whitespace in the JSON, and the action's probability is the sum over all of them. Some probability also goes to texts that parse to $\bot$. So a malformed output is a real event that the loop must handle, not a bug to ignore.

The definitions count turns. A turn is one call to the model. One turn may request several tool calls at once, and one tool call may itself take minutes. Mixing these units is a common source of wrong numbers, and the pitfall box below returns to it.

The token-level view

Training needs the episode at the level of tokens. Serialize a finished episode into one sequence of tokens $w_1, \dots, w_N$, exactly as the chat template would show it to the model. Then mark each token by who wrote it.

The end-of-turn token that closes an assistant turn was sampled, so it has $m_i = 1$, and that is how the model learns to stop.

In plain words, train on what the model wrote, and mask what the world returned. Supervised fine-tuning already uses this rule, with a loss that skips the prompt (§14.3), and here it covers a whole episode. §24.3 proves it is also the right rule for reinforcement learning. The probability of an episode splits into a factor from the policy's tokens and a factor from the environment's. The environment's factor does not depend on $\theta$, so only tokens with $m_i = 1$ carry gradient. In the data question above, $m_i = 1$ on 385 of 2,306 tokens.

Environments as programs

In code, an environment is an object with two methods. reset(task) puts the world in a start state and returns the first observation. step(action) runs an action and returns the observation, the reward (for most agent tasks zero until the episode ends), whether the episode is over, and any extra information. This shape comes from OpenAI Gym (Brockman et al. 2016), a library of reinforcement-learning environments, and much RL code still follows it. Tools are the functions behind step, and every observation is a string.

The hard part is reset. For a coding agent it means a fresh container with the repository at a known commit. For a browser agent it means a fresh browser profile. For an agent that sends email it may not exist at all. That is why agents are trained and evaluated in sandboxes (§24.5).

Writing the loop

Python cells in this book run in your browser with no network, so no cell can call a real language model. The policy below is a short script with the same interface as a model, messages in and text out. The loop is real, and only the decisions are hand-written. The script reads the last observation and reacts to it, as the model in the weather transcript did. First come the tool and the parser $g$.

Now the scripted policy and the loop. The loop is the part to study. It does three things a naive loop does not. A tool exception is caught and turned into an observation. A malformed output gets an error observation instead of a crash. And the step budget bounds everything.

The transcript it prints is the one in the example box, produced by a real loop, with a shorter system prompt and the <answer> tags that the parser looks for around the final answer. Now build the token source mask for it. Real tokenizers split text into subwords (§2.2), but a toy tokenizer that splits on words and punctuation is enough to see the proportions. Role headers belong to the template, so they get $m_i = 0$ even inside an assistant turn.

Here the policy wrote 105 of the 196 tokens, more than half. In a real agent the split runs the other way. A single file read or web page can be thousands of tokens of observation for a tool call of twenty tokens, so most of a long episode has $m_i = 0$.

The same loop against a real model differs only in where the text comes from and in the message format. Here is the loop written by hand with the Anthropic Python SDK. The model name is the one current as this is written (October 2026) and will change. The SDK also ships a tool runner that writes this loop for you.

Three details carry over from the toy to every production loop. Each tool result carries the id of the call it answers (tool_use_id), because one turn can contain several calls. A failed tool is reported to the model with an error flag, never dropped. And the loop has a budget and reports why it stopped.

import inspect

def run_agent(policy, tools, task, max_steps, max_obs_chars):
    raise NotImplementedError

What the loop costs in tokens

The widget showed the context growing. Here is the count, first with tiny numbers. Take a prompt of 100 tokens and a loop that adds 10 tokens per turn. Turn 1 reads 100 tokens, turn 2 reads 110, and turn 3 reads 120. So three turns read 330 input tokens.

In general, let the initial prompt (system prompt, tool definitions, and task) be $P_0$ tokens. Let each turn add $s$ tokens (a local symbol here, not a state), the model's output plus the observation. Turn $t$ reads the prompt and the $t - 1$ earlier turns, which is $P_0 + (t-1)s$ input tokens. Using $\sum_{t=1}^{T}(t-1) = T(T-1)/2$, a loop of $T$ turns reads

input tokens in total. With $P_0 = 100$, $s = 10$, and $T = 3$ that is $300 + 30 = 330$, as counted by hand.

In plain words, re-reading the prompt costs an amount linear in the number of turns, and re-reading the growing transcript costs an amount quadratic in it. For long runs the transcript term dominates. The cell checks the formula at a realistic scale.

This is the first number an interviewer expects in an agent design question. Doubling the turns roughly quadruples the transcript term, so long agent runs are dominated by re-reading, not by writing.

Workflows and agents

Not every system that calls a model several times is an agent. A widely used distinction from Anthropic's engineering team (Anthropic 2024) draws the line at who decides the control flow, the order in which steps run.

The same post names five common workflow patterns.

• Prompt chaining runs fixed steps in sequence, each on the previous output, with checks between them.
• Routing classifies an input and sends it to a specialized prompt or model.
• Parallelization runs subtasks at once, either different pieces (sectioning) or the same task several times for a vote.
• Orchestrator-workers has one model split a task into pieces for worker models. It is already halfway to an agent, because the split is decided at run time.
• Evaluator-optimizer loops a generator and a critic until the critic is satisfied.

The agent is the general case, the loop above with the model in charge. The advice that comes with the distinction is the most useful sentence in agent engineering. Find the simplest system that works, and add autonomy only when the task needs it. The trade-offs are concrete.

Many production systems are workflows with one agentic step inside, for example a fixed pipeline that hands one subproblem to a bounded agent loop. The reliability arithmetic below is the quantitative reason to keep the agentic part small.

Why long tasks fail

Now the arithmetic from the opening, done properly. Start by hand. A task has three steps, and each goes right with probability 0.9. The task succeeds only if all three do, which happens with probability $0.9 \times 0.9 \times 0.9 = 0.729$. Every extra step multiplies by another 0.9.

In general, model a task as $T$ steps, each of which succeeds or fails. A step is whatever unit can fail on its own, often one turn, so with no retries the episode lasts $T$ turns. Write $p$ for the probability that a step succeeds and $\varepsilon = 1 - p$ for its error rate.

Proof. The task succeeds if and only if all $T$ steps succeed, and by independence the probability of that is the product, $p^T$. Setting $p^{T} = 1/2$ and taking logs gives $T \ln p = -\ln 2$, so $T_{1/2} = \ln 2/\ln(1/p)$. The bounds are proved in the derivation below. ∎

In plain words, a task's success decays exponentially in its length, and the half-life is about $0.69/\varepsilon$ steps. Halving the error rate doubles the length of task you can finish half the time. At $p = 0.95$ the half-life is 13.5 steps, at $p = 0.99$ it is 69, and at $p = 0.999$ it is 693.

This is why "95% accurate" means little for an agent until you know the task length. An interviewer who asks for a success estimate wants $p^T$, the half-life, and the assumptions behind them.

import math

def task_success(p, T):
    return 1 - T * (1 - p)          # first-order guess, wrong for long tasks

def half_life(p):
    raise NotImplementedError

Detection and repair

The theorem assumed every error is fatal, which is the pessimistic extreme. Real agents run tests, read error messages, and check their work. The weather transcript showed one error caught and fixed, and the data question showed another. So we model checking and fixing directly.

Proof. A step ends badly exactly when an error occurs and is not both detected and repaired, which has probability $\varepsilon(1 - dq)$. So it ends well with probability $1 - \varepsilon(1-dq)$, and independence across steps gives $p_{\text{eff}}^T$ as before. A step uses one turn, plus one more when an error occurs and is detected, which has probability $\varepsilon d$. So the expected number of turns per step is $1 + \varepsilon d$. ∎

In plain words, only errors that go unnoticed or unfixed kill a task, so the effective error rate is $\varepsilon(1 - dq)$. Detection multiplies the error rate, so it acts on success exactly as accuracy does. Success is about $e^{-T\varepsilon(1 - dq)}$. That is why verification is worth as much as accuracy, and why the error that kills a long task is the one nobody noticed. This is the first half of the answer to the opening puzzle. Agents that run tests and read error messages are not at the mercy of their raw error rate.

That model allows one repair attempt per error. Another common case is a retry loop under a step budget. A detected failure costs one turn, and the agent tries the same subtask again. Now the step budget $B$ matters, because retries use it up.

Here $k$ is the number of retries. $T$ counts subtasks, and the episode uses between $T$ and $B$ turns, so the episode length and the task length are no longer the same number. The proof counts the orders in which retries can fall.

In plain words, a little slack in the budget buys most of what retries can give. Take $T = 20$, $p = 0.95$, and $d = 0.8$, so $r = 0.04$. Finishing in exactly 20 turns succeeds 0.358 of the time, within 22 turns 0.766, and within 25 turns 0.811. That is already the limit, $(0.95/0.96)^{20} = 0.811$. Undetected failures are what hold the limit below 1.

At 98% per step, 50 steps succeed 36% of the time. With the checker and repair, the effective error rate falls from 0.02 to $0.02 \times 0.19 = 0.0038$, and 50 steps succeed 83% of the time. A raw per-step accuracy of 99.62% would give the same. The simulation matches the formula to within sampling noise at every length.

The widget draws the same curves for your own numbers. The dots mark the half-life of each curve.

import math
import numpy as np

def p_finish(T, B, p, d):
    raise NotImplementedError

def simulate_finish(T, B, p, d, n=20000, seed=0):
    raise NotImplementedError

Where the simple model breaks

Every assumption in these theorems is false in some useful way. Knowing which way is what an interviewer is probing.

Errors are not independent. Steps differ in difficulty, and errors cluster on the hard ones. Two kinds of clustering pull in opposite directions.

• Within a task, uneven steps hurt. Take a task with one coin-flip step and nineteen certain ones. It succeeds half the time. Its average per-step success is $\bar p = (0.5 + 19)/20 = 0.975$, and $0.975^{20} = 0.603$ would overestimate it. In general $\prod_t p_t \le \bar p^{\,T}$, because the geometric mean of positive numbers never exceeds their arithmetic mean.
• Across tasks, uneven tasks help. Suppose half the tasks have per-step success 1 and half have 0.95, again an average of 0.975. Overall success at 20 steps is $0.5 \times 1 + 0.5 \times 0.95^{20} = 0.679$, above the 0.603 that the average predicts. In general, if each task has its own per-step success $p_{\text{task}}$, overall success is $\mathbb{E}[p_{\text{task}}^T]$. The function $p \mapsto p^T$ curves upward (it is convex), and for such a function the average of the values is at least the value at the average. That is Jensen's inequality, and here it reads $\mathbb{E}[p_{\text{task}}^T] \ge (\mathbb{E}\,p_{\text{task}})^T$. Easy tasks keep succeeding at lengths where the average predicts failure.

The next part of the section measures this second effect in real agents. The cell checks both numbers.

Not every error is fatal, and not every step matters. Some mistakes are harmless, such as a wasted search. Some can be recovered later, such as a bug caught by a test ten steps on. Some are catastrophic, such as deleting the database. The detection and repair model is the first refinement. Practitioners go further by making catastrophic actions rare and gated, so that the steps that can be fatal are few and checked.

The number of steps is not fixed. An agent chooses how many steps to take, and a confused agent takes more. So task length is better measured in a unit the agent does not control, such as the time a skilled human needs. The measurements below do exactly that.

What practitioners do with all this is consistent.

• They add verification wherever a check is cheap (run the tests, re-read the file, validate the call).
• They cut steps by giving the agent better tools.
• They checkpoint long tasks, so that a failure costs one segment rather than everything (§25.4).
• They put a human approval in front of the few actions that cannot be undone.

Measured time horizons

How do real agents fare? The most informative measurement is the time horizon of METR, a nonprofit that evaluates AI capabilities (Kwa et al. 2025). §19.4 showed how it is computed, so here is the method in brief. Skilled people are timed on a suite of software, machine learning, and cybersecurity tasks, from a few seconds to many hours long. Each model then attempts every task several times as an agent. A model's 50% time horizon $t_{50}$ is the length of task, in human time, that it completes half the time, and its 80% horizon $t_{80}$ is the length it completes four times in five. Both are read off a logistic curve fitted to success against the logarithm of the human time $t$,

with $\sigma(z) = 1/(1 + e^{-z})$ the logistic sigmoid and $\beta \gt 0$ a slope. At $t = t_{50}$ the argument is 0 and success is 0.5, and $t_{80}$ is where the fitted success reaches 0.8. §19.4 writes the horizon at success rate $u$ as $h_u$, so its $h_{0.5}$ and $h_{0.8}$ are our $t_{50}$ and $t_{80}$. We avoid $h$ here because $h_t$ already names the agent's history. The numbers below come from the current version of the suite, METR-Horizon-v1.1.

Read the horizon carefully. It measures how hard a task the agent can handle, in human time. It is not how long the agent runs, and it is not a claim about all jobs.

• The tasks are well specified and self-contained. They are closer to what a capable contractor new to a codebase could do than to messy work with years of context, and models do worse on "messier" tasks of the same length.
• The human times likely overestimate how long an expert who knows the work would take.
• METR's own page says that measurements above 16 hours are unreliable with this task suite, and that the intervals are wide. The page is no longer actively updated, and its last update was in September 2026.

What the horizons imply for a product that needs 99% reliability, and how to measure the horizon of your own agent, are the subject of long-running agents in §25.4. Here we read the numbers.

The last line of the fact box is a puzzle, and the reliability model of this section solves it. Suppose an agent fails at a constant rate per minute of human work. Call that rate the hazard $\lambda$. Then success on a task of length $t$ is $S(t) = e^{-\lambda t}$, the continuous-time version of $p^T$. Here $t$ is the human length of a task in minutes, not a turn index. This is the half-life model of Ord (2025), and it predicts a fixed ratio between the horizons. Setting $e^{-\lambda t_{50}} = 1/2$ and $e^{-\lambda t_{80}} = 0.8$ gives $t_{50} = \ln 2/\lambda$ and $t_{80} = \ln 1.25/\lambda$, so

The measured ratios in the table, from 0.10 to 0.25, are all lower. The logistic form is not what lowers them. A logistic fitted to data from a single constant hazard gives a ratio between about 0.3 and 0.4. In the whole series only two models, both from before 2023, come out above 0.322. They are davinci-002 and gpt-3.5-turbo-instruct, whose horizons are under a minute and whose fits are the noisiest. So the 80% horizon is shorter than any single per-minute failure rate allows. Why?

Because tasks differ. Some tasks of a given length are easy for the agent per minute and some are hard, exactly the "uneven tasks" effect above. Ord notes that averaging exponentials with different rates thickens the tail of the curve. The theorem below shows that this always pushes the ratio down. This is the second half of the answer to the opening puzzle. Long tasks are not all equally hard per step, so a suite of them is not a sequence of identical coin flips.

The idea of the proof is short. For a single hazard, $-\ln S(t)$ is a straight line in $t$. For a mixture it bends downward, so success falls to 80% early relative to when it falls to 50%.

In plain words, a single per-minute failure rate fixes the 80% horizon at a third of the 50% horizon. Any spread in how hard tasks are per minute makes the 80% horizon shorter still. The success curve then drops steeply at first, from hard tasks the agent fails quickly, and keeps a fat tail of easy tasks it goes on solving. METR's ratios of 0.10 to 0.25 say that frontier agents are not failing at a steady rate. They solve some long tasks reliably and fail some short ones.

The widget below draws both cases with every task length divided by its curve's own 50% horizon, so both curves pass 50% at the same point. The thin gray lines are single tasks, and the thick curve is their average. Widen the spread of hazards and watch the 80% point move left, into the shaded band of ratios METR measured.

In practice the 80% horizon is the number to plan with. If you need a task done reliably, the 50% horizon overstates what you can hand off by a factor of four to ten.

import numpy as np
from scipy.optimize import minimize_scalar, brentq
from sklearn.linear_model import LogisticRegression

def hazard_mle(t, y):
    return np.mean(y) / np.mean(t)        # a rate from counts, not a per-task likelihood

def logistic_horizons(t, y):

Running the same task again

The time horizon averages over tasks. A second measurement asks how reliably an agent solves the same task again, which matters as soon as a user depends on it. The τ-bench benchmark (Yao et al. 2024) tests this in customer service. A language model plays a customer who talks to the agent. The agent follows a written domain policy and calls tools against a database. Each conversation is graded by comparing the final database state with the goal and checking that the agent told the user what it had to.

Its reliability metric is pass$^k$, the probability that $k$ independent attempts at a task all succeed, averaged over tasks (§22.5),

where $p_{\text{task}}$ is the agent's success probability on that task. To estimate it, run each task $n$ times and count the $c$ successes. Among all sets of $k$ of the $n$ trials, the fraction in which every trial succeeded is $\binom{c}{k}/\binom{n}{k}$. That fraction is an unbiased estimate of $p_{\text{task}}^k$ for $k \le n$, as §22.5 proves. Contrast pass@$k$ from §18.2, the chance that at least one of $k$ attempts succeeds. pass@$k$ rises with $k$, and pass$^k$ falls.

The agent in those runs sampled at temperature 0. Most of the variation came from the simulated customer, who sampled at temperature 1 and phrased the same request differently each time. So the same agent, on the same task, gave different results from one run to the next. That is the property a user notices first, and it is why an interviewer asks for pass$^k$ and not only average success.

Autonomy is a design decision

Reliability numbers tell you how often an agent will be wrong. They do not tell you how much freedom to give it. Feng, McDonald, and Zhang (2025) argue that an agent's level of autonomy should be chosen deliberately, as a design decision separate from how capable the agent is and from the environment it runs in. Their five levels are named by the role left to the user.

A different ladder, from "AI as a tool" to "AI as an agent", appears in the levels of AGI of Morris et al. (2023), which also lists autonomy as its own dimension, though in Morris et al. higher capability unlocks higher autonomy. Feng et al. go further and argue that a capable agent can sensibly run at any level. What both share is the distinction itself. Capability is what an agent could do, and autonomy is what you let it do.

The same agent can sensibly run at different levels in different places. Take a coding agent that can edit code, run tests, open pull requests, and deploy. In a production repository it should run at level 4. It writes code and runs the test suite on its own, opens a pull request, and stops, because a human approves the merge and the deploy. In a disposable sandbox with no credentials, practicing on training tasks, it can run at level 5. Nothing it does there can hurt anyone, and an off switch is enough.

The arithmetic of this section says where approvals pay. An approval gate is a detector with $d$ close to 1 for the steps it guards, and it costs human attention. So put gates in front of the few steps whose failure is both likely to slip past automated checks and impossible to undo, such as deploys, payments, deletions, and messages to people. Do not put them in front of every file edit. §24.6 turns this into a permission system, with risk classes, approval gates, and defenses against injected instructions (building on §19.3).

In PyTorch

The token source mask is the piece of this section that reaches training. Here it is in a few lines, for a serialized episode whose policy-written spans are known. The loss is the mean negative log-likelihood over policy tokens only. The gradient confirms that positions written by the environment contribute nothing to it. In a real model those positions still shape the gradient through attention, as context for later tokens, exactly as prompts do in supervised fine-tuning (§14.3).

Read more

• Anthropic (2024). Building effective agents. The workflow and agent distinction, the five workflow patterns, and the advice to start simple, from the team that builds Claude's agents.
• Yao et al. (2023). ReAct: Synergizing Reasoning and Acting in Language Models. The paper that made the think, act, observe loop the default design; read it with §24.3.
• Sumers et al. (2024). Cognitive Architectures for Language Agents. The CoALA vocabulary of memories, internal and external actions, and decision procedures, with a survey organized by it.
• Kwa et al. (2025). Measuring AI Ability to Complete Long Software Tasks. The time-horizon methodology, with human baselines, the logistic fit, and the messiness analysis. The current numbers are on METR's time-horizons page.
• Ord (2025). Is there a half-life for the success rates of AI agents?. The constant-hazard model, its 0.322 prediction, and the note that mixing task difficulties thickens the tail, which our theorem makes exact.
• Yao et al. (2024). τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains. Simulated users, state-based grading, and the pass$^k$ reliability metric.
• Feng, McDonald, and Zhang (2025). Levels of Autonomy for AI Agents. Autonomy as a design decision with five user roles, and autonomy certificates. Compare Morris et al. (2023), Levels of AGI for Operationalizing Progress on the Path to AGI.
• Lilian Weng (2023). LLM Powered Autonomous Agents, and Chip Huyen (2025), Agents. Two readable overviews of planning, memory, and tool use, good for a second view of the whole chapter.

Watch next

▶

▶