Chapter 24 · Section 24.5 · 100 min

Code agents and computer use

A code agent is a language model that acts by writing programs and running them. A computer-use agent acts on a screen instead, by clicking and typing the way a person does. Both put the model in front of a real machine, with files, processes, a network, and the power to break things. This section shows why a program is often a better action than a single tool call, how to run model-written code safely, how coding agents are measured and trained, and how screen agents see and point.

Interviewers ask about this area because it moved faster than almost any other. SWE-bench is a benchmark of 2,294 real GitHub issues, each graded by running the repository's own tests. In October 2023 the best model tested on it fixed 1.96% of the issues (Jimenez et al. 2023). That model was handed a few retrieved files and asked to write a fix in one shot. In the leaderboard's February 2026 entries, Claude Opus 4.5 fixed 76.8% of SWE-bench Verified, a subset of 500 issues that people screened by hand. It ran inside a harness, the program around the model that offers it tools and runs them, and this harness offered nothing but a shell. The run cost $0.75 per issue on average.

Three things changed between those two numbers. First, models learned to act. They search the repository, open files, run the code, read the error, edit, and run the tests again. Second, the commands they are given got better. With the model held fixed, changing a single command moved the share of issues fixed by as much as 7.7 points (Yang et al. 2024). Third, coding agents are now trained with reinforcement learning, in thousands of isolated copies of real repositories at once.

Here is the setting, restated from earlier in the chapter. §24.1 defined an agent as a loop. The model writes text, a parser turns the text into an action, the environment runs the action and returns an observation as text, and the model writes again. In symbols the agent is a policy $\pi_\theta(a_t \mid h_t)$ , a probability distribution over the next action $a_t$ given the history $h_t = (x, z_1, a_1, o_1, \dots, z_{t-1}, a_{t-1}, o_{t-1})$ . The history holds the task $x$ , and for each earlier turn $i$ the model's thought $z_i$ (the text it wrote around the action), the action $a_i$ , and its observation $o_i$ .

Until now each action was a JSON tool call, a small structured request that names one tool and its arguments. §24.2 showed how to check such calls and how to make them always well formed. §24.3 interleaved them with written reasoning and trained the policy on them. §24.4 showed that searching over them needs a way to undo them.

In this section the action becomes a program and the environment becomes a computer. Four consequences organize what follows. Code composes tools, so one action can do the work of many turns. Code can do anything, so it must run in a sandbox. Real software work needs benchmarks that grade by running tests, and interfaces shaped for the model, and those benchmarks double as training environments. And when the only interface is a screen, the agent must look at pixels and point at them, which costs many tokens and can miss.

Code as the action space

Start with a small task that every agent builder meets in some form. A user asks "How much do we owe Acme in late-delivery fees?" Three tools are available.

list_orders(customer, cursor) returns one page of three order ids, plus a cursor that says where the next page starts. The agent cannot guess the cursor. It must read it from the page.
get_order(id) returns the promised and actual delivery days and the order's value.
late_fee(value, days_late) applies the contract, 1% of the value per day late, capped at 10%.

Acme has seven orders, so there are three pages.

A turn is one call to the model. With JSON tool calls the agent cannot ask for page 2 until it has read page 1, because the cursor for page 2 is inside page 1's result. So paging takes three turns. Then the agent can fetch all seven orders in one turn of parallel calls, compute the fees for the late ones in one more turn, and answer in a sixth. Every order record, with its long shipping log, lands in the context and is read again by every later turn.

Written as code, the same work is one action. A while loop walks the pages, and a for loop fetches the orders and adds up the fees. The only thing that enters the context is the one line the program prints.

The difference in turns is not an accident of this example. A tool-calling agent must see a result before it can write any call that needs it. A program never waits for the model, because the interpreter passes each result to the next call through a variable.

To state this precisely, draw the calls as a graph. Put an arrow from call $u$ to call $v$ whenever $v$ 's arguments cannot be written without $u$ 's result. Such a graph has no cycles, since no call can need its own result, so it is a directed acyclic graph. Its critical path is the longest chain of arrows, as in the task graphs of §24.4, and $L_{\text{cp}}$ is the number of calls on it.

∂ Proof

A turn is one call to the model, and it writes the arguments of every tool call it issues from the context it was given. The results of a turn's calls are appended only after that turn ends. If $v$ depends on $u$ , then $u$ 's result must already be in the context of the turn that issues $v$ , so $v$ is issued in a strictly later turn than $u$ . Along a path $c_1 \to c_2 \to \dots \to c_{L_{\text{cp}}}$ the turn indices strictly increase, so the last call is issued in turn $L_{\text{cp}}$ or later. The answer needs that call's result, so it comes in turn $L_{\text{cp}} + 1$ or later.

For part 2, the interpreter evaluates all the calls inside one action in an order that respects the arrows, passing results through variables, so none of them waits for the model. The model needs one turn to write the program and one to read its output and answer. ∎

In plain words, tool calling pays one model call per level of dependency, and code pays one per program. The condition in part 2 matters. Sometimes the next step needs the model's judgment about a result, such as reading a document to decide what to search for next. Then code cannot compute the dependency and saves nothing. Code wins when the composition is mechanical, such as pagination, filtering, joins, arithmetic, and retries.

Two turns also assumes the program runs correctly the first time. Each failed run costs one more turn, with the error message as the observation. That is still far fewer turns than a long chain of calls.

In the Acme task the critical path is three page calls, then one get_order, then one late_fee. So $L_{\text{cp}} = 5$ , and the bound says at least six turns. The cells below run both agents. The tools count their calls, and a rough rule of four characters per token measures what each agent reads. Every turn also reads a system prompt with the tool definitions, of $P_0 = 1{,}000$ tokens, as in the token count of §24.1. As everywhere in this chapter, the policies are short scripts with the same interface as a model. The loops are real, and only the decisions are hand-written.

Python · cell 1

The JSON agent reads every result so far and decides its next batch of calls. Independent calls go out in parallel in one turn, as every major API allows. So this is the best a tool-calling agent can do.

Python · cell 2

The code agent writes one program. The executor runs it in a fresh namespace that holds only the tools. It returns what the program printed, or the exception it raised, as the observation.

Python · cell 3

✏️ The Acme task, as tool calls and as one program

Both agents make the same 14 tool calls (three list_orders, seven get_order, four late_fee) and reach the same answer, four late orders and $279.00 in fees. You can check the fees by hand. Order A2 is 3 days late on $800, so 3% gives $24. A4 is 6 days late on $2,000, $120. A6 is 3 days late on $1,500, $45. A7 is 15 days late, capped at 10% of $900, $90.

	turns	tool calls	input tokens read
JSON tool calls	6	14	9,374
one code action	2	14	2,131

The JSON agent needs six turns, exactly the bound $L_{\text{cp}} + 1 = 6$ . It reads about 4.4 times as many tokens, because the seven shipping logs enter its context and every later turn reads them again. The code agent's two turns read only the task, the program, and the one line it printed.

Make the logs real database rows or web pages and the gap keeps growing, since each result in the tool-calling agent is read again by every later turn. The published evidence matches the theorem, including its condition.

📌 Code actions win when calls compose

The paper that proposed code actions, under the name CodeAct (Wang et al. 2024), built M³ToolEval, 82 tasks that each need several calls to several tools. gpt-4-1106-preview succeeded on 74.4% of them with code actions, against 53.7% with plain-text actions and 52.4% with JSON. It needed 5.5 turns on average with code, against 7.7 with text and 7.6 with JSON.
On API-Bank, where each task is one atomic call, the same model did better with JSON (82.7% against 76.7% for code). With nothing to compose, code buys nothing.
Anthropic's programmatic tool calling lets the model write code that calls tools inside a sandbox, and only the final output returns to the model. It cut average token use on complex research tasks from 43,588 to 27,297, or 37% (Anthropic, November 2025).

Code as an action is a different idea from code as reasoning. In program-aided reasoning the model writes one program to compute an answer, so the interpreter does the arithmetic the model would get wrong (§18.1). A code agent writes programs that act on an environment across many turns, reads the errors, and revises. The interpreter's error messages are among the most useful observations an agent ever gets, because they name the file, the line, and the reason.

Code also has real costs, which is why production systems offer both kinds of action.

Checking before running. A JSON call can be checked against its schema before it runs, forced to be well formed (§24.2), and shown to a person for approval. A program cannot be checked that way in general. Whether it will call a given tool or touch a given file is not known until it runs. So approval checks must move into the tools themselves and into the sandbox's permissions.
Partial failure. A program that crashes after three of seven writes leaves the world half-changed. The tools a program calls should be idempotent, meaning that running a call twice has the same effect as running it once, so a retry repeats no side effects. §24.6 takes this up.
Isolation. A JSON call can only invoke the tools you listed. A program can do whatever its interpreter can, which is the subject of the next part.

🔬 Why no checker can predict what a program will do

Programs are written in a Turing-complete language, one that can express any computation. So the question "will this program ever call the tool send_email?" is undecidable, meaning no algorithm answers it correctly for every program. Here is the argument. Take any program $P$ that uses no tools, and wrap it as "run $P$ , then call send_email". The wrapper calls the tool if and only if $P$ halts. A checker that always answered the first question correctly would therefore decide whether any program halts, and Turing proved that no algorithm can do that.

So real systems do not try to predict what a program will do. They enforce the rule at the moment of the call, inside the tool or the sandbox, where the question is no longer about the future.

	JSON tool calls	code actions
turns for dependent calls	at least $L_{\text{cp}} + 1$	two, if the dependencies are computable
intermediate results	all enter the context and are read again	only what the program prints
check before running	schema validation, constrained decoding, approval per call	not possible in general; checks move into tools and permissions
failure	one call fails, the rest are untouched	a crash mid-program can leave partial side effects
isolation needed	the tools' own permissions	a sandbox for arbitrary code
best for	single calls, high-stakes actions, models that code poorly	pagination, filtering, joins, loops, data analysis, software work

In practice, give an agent code actions when its tools are cheap to compose and their results are large. Keep JSON calls for the few actions that a person must approve one at a time. An interviewer will ask you to count the turns for a concrete task, and then ask what you give up by switching to code.

Sandboxes

A code action can do anything the interpreter can, and the interpreter can do anything the operating system lets it. So the question is never whether to isolate model-written code, only how strongly. A sandbox is an environment that runs code under limits the code cannot lift. Before choosing one, list what can go wrong.

The model's own mistakes. An agent that runs rm -rf in the wrong directory, force-pushes to a shared branch, or drops a database table needs no adversary. Practitioners report such mistakes as far more common than attacks.
Injected instructions. A repository file, an issue comment, or a web page can contain text that the agent reads as an instruction. §19.3 showed why this risk is built into any model that reads untrusted data. It calls the dangerous combination the lethal trifecta, an agent that holds private data, reads untrusted content, and has a way to send data out.
Reward hacking. During reinforcement learning the policy is paid by a grader, and if it can reach the grader it can game it. Policies have been seen exiting before the assertions run, patching the test framework, and editing the tests (§18.3). Baker et al. 2025 caught a frontier reasoning model doing this during training on coding tasks, by having a second model read its chain of thought (§18.1).
Exfiltration. Any secret in the environment, such as an API key in an environment variable or a cloud credential on the disk, is one print away from the context and one network request away from the outside world.
Resource abuse. An infinite loop, a fork bomb (a program that copies itself until the machine runs out of processes), a 50 GB file, or a cryptocurrency miner.
Escape. Code that breaks out of the isolation and reaches the host machine, and through it every other user's sandbox.

The isolation boundary is where the options differ most, so here they are from weakest to strongest. The differences all come down to the kernel, the core of the operating system. Every other program asks the kernel for files, memory, and network access, through requests called system calls.

A process shares everything with the host except what you take away. You take things away with resource limits (rlimits) and a filter on system calls (seccomp).
A container gives the code its own view of files, processes, and the network (Linux namespaces) and caps its resources (cgroups). But it still runs on the host's kernel, so a bug in that kernel is an escape.
gVisor puts a kernel written in user space between the code and the host. The code's system calls are served by gVisor rather than by the host kernel (gVisor documentation).
A virtual machine gives the code its own kernel behind hardware virtualization (KVM on Linux). A small host program, the virtual machine monitor, emulates its devices. A microVM such as Firecracker (Agache et al. 2020) strips the virtual hardware down until a VM starts almost as fast as a container.

boundary	how it isolates	what an escape needs	startup	typical use
process with rlimits and seccomp	limits and a system-call filter in the host kernel	a hole in the filter or the kernel	milliseconds	a second layer inside one of the others
container	namespaces and cgroups, shared host kernel	a host kernel bug	typically under a second	single-tenant CI, trusted code
gVisor	a user-space kernel intercepts system calls	a gVisor bug, then a host bug through its narrow interface	close to a container	multi-tenant code execution
microVM (Firecracker)	its own guest kernel, KVM hardware virtualization	a bug in the small virtual machine monitor or KVM	under 125 ms to application code	serverless, RL rollouts at scale
full VM	its own kernel and full virtual hardware	a bug in the hypervisor, the layer that runs the VMs	seconds or more	computer-use agents that need a whole desktop

Stronger isolation costs startup time and compatibility. At the scale of agents it costs little else.

The other four parts of the definition are where most real incidents are prevented. Deny network access by default. When a task needs packages, route traffic through a proxy that allows only a package mirror. Mount the workspace read-write and the rest of the filesystem read-only, and keep hidden tests and graders outside the sandbox entirely. Pass no credentials through environment variables, because the first thing a curious program prints is os.environ. And limit everything, since an accidental infinite loop is far more common than an exploit.

Production sandboxes stack two layers. The outer one is a container or microVM per episode, one attempt at one task from start to finish. The inner one is a process with hard limits. The deeper box writes both out.

🔬 The two layers, written out

Here is the inner layer, a runner that executes one code action in a fresh directory with an empty environment and hard limits. It is correct for Linux. It is a second layer, never the only one.

Two details in it trip people up. First, the CPU limit does not stop a program that sleeps or waits on a lock. So the wall-clock timeout is separate, and it must kill the whole process group, because killing only the child leaves its children running. Second, RLIMIT_NPROC counts every process of the user, not of the sandbox. So each sandbox must run as its own user for the limit to mean anything.

The outer layer is a container or a microVM per episode. With Docker and gVisor's runtime (runsc), one episode's sandbox looks like this.

# One sandbox per episode: gVisor's runtime, no network, read-only root, a writable workspace,
# no capabilities, an unprivileged user, and hard limits. Hidden tests are never mounted here.
docker run --rm \
  --runtime=runsc \
  --network none \
  --read-only --tmpfs /tmp:rw,size=512m \
  --mount type=bind,src="$PWD/workspace",dst=/workspace \
  --workdir /workspace \
  --user 1000:1000 --cap-drop ALL --security-opt no-new-privileges \
  --pids-limit 256 --memory 4g --memory-swap 4g --cpus 2 \
  swe-task-image:latest \
  bash -lc 'python -m pytest -x -q'

Training raises a different problem, which is scale. Reinforcement learning on agents often uses group baselines, as in GRPO (group relative policy optimization, §18.4). The policy makes $G$ attempts at the same task, called rollouts, and each rollout is scored against the average of its group. §24.3 applies this to agents. Every rollout needs its own sandbox, and all rollouts in a group must start from the same state.

✏️ Sandboxes for one training step

One step runs 512 tasks with $G = 8$ rollouts each, so it needs $512 \times 8 = 4{,}096$ sandboxes. Building each environment from scratch, by cloning the repository and installing its dependencies, can take minutes. So systems build each environment once, save a snapshot of it, and restore copies. They use copy-on-write container layers or microVM snapshots, where a copy shares the snapshot's data and stores only what it changes.

At Firecracker's 150 creations per second, 4,096 sandboxes take $4{,}096 / 150 \approx 27$ seconds on one host, and under a second across 32 hosts. Their virtual machine monitors add about $4{,}096 \times 5$ MB, or 20 GB, on top of whatever memory the guests use.

For a language model alone, going back to any earlier state was free, since it only meant reusing an earlier prefix of text (§15.7). For a code agent, that reset has become an infrastructure system of snapshots and clones.

Warm-up

Stay inside the sandbox

A file tool that runs outside the sandbox must refuse paths outside the workspace, and this check is a classic place for security bugs. A symlink (symbolic link) is a file that points to another path, so a path inside the workspace can lead out of it. Implement is_within(root, path, symlinks), which returns True exactly when path resolves to root or to something under it.

root is an absolute directory, possibly with a trailing slash. It contains no symlinks itself.
path is absolute, or relative to root.
symlinks maps absolute link paths to their targets. A relative target is relative to the directory that contains the link.
Resolve the path the way the operating system does, one component at a time from the left. Skip empty components and .. Let .. move to the parent, never above /. Whenever the path built so far is a key of symlinks, replace it with its target and continue. So a .. that follows a link applies to the link's target.
Allow at most 40 link expansions, as Linux does. If resolving needs a 41st, assume a loop and return False. When in doubt, refuse.

def is_within(root, path, symlinks):
    raise NotImplementedError

Tests, hints, verified solutions, and AI help, with Pro at $7.99 a month.

A real system avoids checking paths with string logic when it can. It opens files relative to a directory handle and lets the kernel refuse any path that leaves it, as Linux's openat2 does with the flag RESOLVE_BENEATH. Or it runs the file tool inside the sandbox, where paths outside the workspace do not exist. But the exercise is the check that interviewers ask for, and any file tool that runs outside the sandbox must get it right.

The toy coding agent later in this section runs its code with much weaker protection than any of these, and it is worth seeing why that protection is weak. Its executor runs code in a Python namespace whose built-in functions leave out open and __import__. So the obvious ways to read a file or load a module fail.

Python · cell 4

The toy repository has a bug. Its median returns the upper of the two middle elements of an even-length list instead of their mean. The first line reproduces it (the answer is 3, not 2.5), and reading the test file or importing os fails.

That is useful hygiene against an agent's mistakes, but it is not a security boundary. Every Python object carries a reference to its class. From any object a program can walk the class hierarchy back to objects that reach the full interpreter. That is why projects that restrict Python from inside the same process have a long history of escapes. A namespace limits what an honest program can do by accident. Only an operating-system boundary limits what a hostile one can do.

⚠️ Three sandbox mistakes that look like security

Removing built-ins and calling it a sandbox. Restrictions inside the same process can be walked around in Python, or in any language whose objects can inspect themselves. Use a process, container, gVisor, or VM boundary, and treat namespace tricks as hygiene only.
Checking paths by string prefix. "/workspace/x".startswith("/work") is true. Also, .. and symlinks resolve after your check, and a file can be swapped for a symlink between the check and the open. Compare whole components after resolution, or open relative to a directory handle without following links.
Leaving secrets in the environment. A token in an environment variable, a cloud metadata endpoint on the network, or an SSH key in the home directory is available to every line the model writes. Start from an empty environment and add only what the task needs.

An interviewer who asks you to design a sandbox wants all five parts of the definition, a reason for the boundary you chose, and the reset mechanism for training at scale.

Software engineering agents

Coding was the first domain where agents became widely used at work, for a reason the previous parts make clear. A repository comes with a cheap and mostly trustworthy reward, its test suite, and it fits naturally in a sandbox. This part covers how SWE-bench turns repositories into graded tasks, what the interface between model and repository should look like, where the numbers stand, and how coding agents are trained.

How SWE-bench is built and graded

Jimenez et al. 2023 built SWE-bench in three stages. A pull request is a proposed change to a repository, which maintainers review and merge. The authors collected about 90,000 pull requests from 12 popular Python repositories. They kept merged ones that resolve a GitHub issue and change the repository's test files, since a contributor who adds tests has written down what "fixed" means. Then they kept only those where the repository installs, the tests run, and at least one test fails before the fix and passes after it. That left 2,294 task instances.

Each instance comes with the repository at the commit just before the fix, the issue text, and two lists of tests. The tests that check the fix come from the pull request's changes to the test files, so the agent's copy of the repository does not contain them in their final form. Every instance has at least one such test, and 40% have two or more. A median of 51 existing tests check that nothing else broke.

📐 SWE-bench task instance and resolution

A task instance is a repository at a base commit, an issue text, and two sets of tests from the reference pull request. $\mathcal{T}_{\text{F2P}}$ (FAIL_TO_PASS) holds the tests that fail at the base commit and pass with the reference fix. $\mathcal{T}_{\text{P2P}}$ (PASS_TO_PASS) holds the tests that pass both before and after. The agent sees the repository and the issue but not the pull request's tests, so $\mathcal{T}_{\text{F2P}}$ is hidden from it, while the tests in $\mathcal{T}_{\text{P2P}}$ are ordinary tests it can read and run. It submits a patch.

To grade, apply the patch, run the tests, and read each test's status $\operatorname{st}(t)$ from the log. A test missing from the log has status "missing". The patch resolves the instance if and only if it applies and

\operatorname{st}(t) \in \{\text{PASSED}, \text{XFAIL}\} \ \ \forall t \in \mathcal{T}_{\text{F2P}} \quad\text{and}\quad \operatorname{st}(t) \in \{\text{PASSED}, \text{XFAIL}, \text{SKIPPED}\} \ \ \forall t \in \mathcal{T}_{\text{P2P}}.

The resolve rate is the fraction of instances resolved.

In plain words, every test that proves the fix must pass, every old test must still pass or be skipped, and a test you cannot find in the log counts against you. XFAIL is the status that pytest, Python's standard test runner, gives a test that is marked as an expected failure and does fail. It counts as passing.

Skipped tests follow different rules in the two sets, which is how SWE-bench's grading code treated them when we read it in October 2026. A skipped old test is not a regression, but a skipped fix test proves nothing. The grading code notes that without this rule, a patch that made every FAIL_TO_PASS test skip would count as resolved.

✏️ Grading one instance by hand

An instance has $\mathcal{T}_{\text{F2P}} = \{$ test_float_repr, test_float_roundtrip $\}$ and $\mathcal{T}_{\text{P2P}} = \{$ test_int_card, test_long_string, test_comment $\}$ . After the agent's patch, the log reads

test_float_repr       PASSED
test_int_card         PASSED
test_long_string      FAILED
test_comment          SKIPPED

test_float_roundtrip is missing from the log, so it counts as failed, and that alone means the instance is not resolved. test_long_string broke, which also fails it, since the patch fixed one thing by breaking another. test_comment was skipped, which is fine for a PASS_TO_PASS test.

So two independent reasons say "not resolved". A grader that made two common mistakes, checking only FAIL_TO_PASS and ignoring tests missing from the log, would have said "resolved".

The grading rule is strict, but it can still be wrong. Issues sometimes contain the fix. Tests can be too weak to notice a wrong patch, or so specific that they reject a right one. And frontier models have seen many of the reference fixes during training. §22.5 measures each of these and explains why OpenAI stopped reporting SWE-bench Verified in February 2026.

One more defense lives in the harness. Before running the hidden tests, SWE-bench's evaluation script restores the test files that the hidden tests change, and only then adds the hidden tests. So an agent's edits to those files are overwritten. It cannot undo edits to other files the tests depend on, such as a conftest.py, the file where pytest looks for shared test setup. A patched conftest.py can even print fake PASSED lines. So the grading code also compares the log with the exit code of the test command, and it rejects a run that exited with an error while reporting no failure. Graders also flag patches that touch test infrastructure. An interviewer who asks how SWE-bench is graded will follow up with exactly these failure modes.

🔬 The SWE-bench family

The original 2,294 instances have grown a family, and each member inherits the grading definition above.

SWE-bench Lite has 300 instances chosen to be cheaper to run.
SWE-bench Verified has 500 that human developers screened for a clear issue and fair tests. OpenAI released it in August 2024, and it was the standard number until its own problems caught up with it.
SWE-bench Multilingual has 300 tasks from 42 repositories in 9 languages. SWE-bench Multimodal has 617 JavaScript tasks from 17 libraries whose issues contain images (Yang et al. 2024).
SWE-Bench Pro (Deng et al. 2025) has 1,865 harder problems from 41 repositories. They are split into a public set from 11 repositories, a held-out set from 12, and a commercial set from 18 proprietary codebases, so that part of it cannot have been seen in training (§22.3).
Terminal-Bench 2.0 (Merrill et al. 2026) moves from patches to whole tasks in a terminal, 89 of them, each with its own environment, a human-written solution, and tests. Frontier agents scored under 65% at its release.

Core

Grade a SWE-bench run

Implement grade_run(instances, logs, patches, costs), the grader for a batch of task instances, by the definition above.

instances[id] is a dict with lists "FAIL_TO_PASS" and "PASS_TO_PASS".
logs[id] maps test names to statuses ("PASSED", "FAILED", "ERROR", "SKIPPED", "XFAIL"), or is None if the patch did not apply.
patches[id] is the list of file paths the patch modifies, or None.
costs[id] is the dollars spent on the instance.

Return a dict with four entries.

"resolved" maps each id to a bool, by the definition above.
"resolve_rate" is the fraction resolved.
"cost_per_resolved" is the total cost of all instances divided by the number resolved, or float("inf") if none was.
"touches_tests" is the sorted list of ids whose patch modifies a test file. A path is a test file if one of its directories is named test or tests, or its file name starts with test_, ends with _test.py, or is conftest.py.

def grade_run(instances, logs, patches, costs):
    raise NotImplementedError

Tests, hints, verified solutions, and AI help, with Pro at $7.99 a month.

The agent-computer interface

A human engineer does not fix a bug through a bare terminal. They use an editor with line numbers and a search that lists matches by file. They also use a linter, a program that checks code without running it and underlines a syntax error the moment it appears. The SWE-agent project (Yang et al. 2024) asked whether a model needs tools of the same kind, and its answer has shaped coding agents since.

SWE-agent's interface had four parts.

Search. The commands find_file, search_file, and search_dir list every match as a file and line, return at most 50 results, and ask for a more specific query when there are more.
A file viewer. It shows 100 lines at a time with line numbers, the file's path and length, and how many lines lie above and below the window.
An edit command. It replaces a range of lines and then shows the edited region. A linter checks the edit first, and an edit that introduces a syntax error is discarded, with the error shown to the agent.
A history processor. It collapses every observation older than the last five into a single line.

The agent wrote a short note of reasoning and then one command per turn, the ReAct format of §24.3. Each run had a budget of $4. The paper then removed or changed one component at a time, with GPT-4 Turbo on SWE-bench Lite. An experiment of this kind, which measures what each part is worth by taking it away, is called an ablation.

Two patterns stand out. First, more context was not better context. The whole-file viewer and the full history both lost to their condensed versions. Second, a well-meant interface can do worse than none. Iterative search, modeled on an editor's "find next", scored below plain grep. The authors saw why in the transcripts. Agents called "next" until they had looked at every match, which could use up the budget or the context window.

In the widget's illustrative episode, the context at the last turn grows only about 2.5 times, from 7,913 to 20,013 tokens. With the last five observations kept, at most five of the large file views are in the context at once. The tokens read over the whole episode grow more than four times, and the resolve rate falls from 18.0% to 12.7%. Keep everything as well and the two costs compound.

The paper never measured that combination, since it changed one component at a time. Single changes say nothing about how changes combine. When an interviewer asks how to design a coding agent's tools, the lesson to give is that short, informative observations beat more context.

A history processor that keeps calls and results paired

The history processor is the cheapest of the four parts to build. It is also a common interview task, because the obvious version breaks the conversation. In the chat formats of the major APIs, an assistant message carries its tool calls, each with an id, and each result names the id it answers. In OpenAI's format a result is a tool message, and in Anthropic's it is a tool-result block. Both APIs reject a request in which a call has lost its result or a result has lost its call. So a history processor may shorten old results, but it must never drop them.

Parallel calls add a trap. Their results can come back in any order, so the command behind a result must be found by its id, never by its position.

There is also a cost that is easy to miss. A prompt cache stores the model's work on the start of a request. A later request reuses that work only up to the first message where the two requests differ (§24.6). A sliding window rewrites an old message on every turn, so the reusable part stops at the message that was just collapsed.

Core

A history processor

Implement the two functions of a history processor like SWE-agent's, over chat messages. A message is a dict with "role" ("system", "user", "assistant", or "tool") and "content". An assistant message may have "tool_calls", a list of {"id", "cmd"}. A tool message has "tool_call_id", the id of the call it answers. The setup defines msg_tokens(m) and an example episode, make_episode().

collapse_history(messages, k) returns (new_messages, tokens).

The newest k tool messages stay exactly as they are. Treat any k below 1 as 1, so the newest tool message is never collapsed.
Every older tool message gets the content f"(output of {cmd} omitted, {n} lines)", where cmd is the command of the call with its tool_call_id and n = len(content.splitlines()). If that placeholder is not shorter than the original content, keep the original.
Every message keeps its position, role, and ids, and system, user, and assistant messages are never touched. Do not modify the input.
Raise ValueError if a tool message's id matches no earlier tool call.
tokens is the sum of msg_tokens over the new list.

first_change(prev, new) returns the index of the first message at which two requests differ. If one list is a prefix of the other, it returns the length of the shorter one. It is the number of leading messages a prompt cache could reuse.

def collapse_history(messages, k):
    return [dict(m) for m in messages], sum(msg_tokens(m) for m in messages)   # placeholder: collapses nothing

def first_change(prev, new):
    raise NotImplementedError

Tests, hints, verified solutions, and AI help, with Pro at $7.99 a month.

With $k = 2$ the example episode shrinks from 833 tokens to 339. On the next turn the request first differs from the previous one at message 6, the file view that has just left the window. Everything from there on must be written to the cache again. A real transcript has thousands of tokens after that point. So a processor that collapses one observation per turn pays a cache write on most of the context every turn.

Production systems therefore clear old results in batches, and only when a clear removes enough tokens to pay for the rewrite. §25.2 derives that trade-off. It also gives the general form of this exercise, which clears tool results but leaves a reference to each, such as the file path, so the agent can fetch it again.

What still holds in 2026

Are these lessons still true when a harness with only a shell reaches 76.8% on SWE-bench Verified? The specific numbers belong to GPT-4 Turbo. Stronger models, trained heavily on tool use and able to use long contexts, need less help from the interface. But the principles have survived in production harnesses.

Observations are short and informative, and long outputs are truncated with a note. Claude Code limits a tool response to 25,000 tokens by default (Anthropic 2025).
Error messages say what to do next.
Guardrails stop mistakes from compounding, which matters because a long task fails if any one of its many steps fails (§24.1).
Small details matter. Anthropic's SWE-bench agent made mistakes with relative file paths until the tool required absolute ones (Anthropic 2024).

The edit command most agents use today replaces an exact string rather than a range of line numbers. The agent gives the old text and the new text, and the edit succeeds only if the old text occurs exactly once. Line numbers go stale the moment an earlier edit adds a line, while a unique string is its own address. The uniqueness rule turns an ambiguous edit into an error the agent can fix by quoting more of the surrounding code. Anthropic's text editor tool uses this convention, with a "view" command that shows line numbers for reading.

Interview

The editor tool

Implement the two commands of a string-replacement editor, a windowed view and a unique-match str_replace, the way coding-agent interviews ask for them.

view(text, start=1, window=100) shows lines start to start + window - 1, counting from 1 and stopping at the last line.

Format each shown line as f"{i:6d}\t{line}".
If lines are hidden above, the first output line is "(k more lines above)". If lines are hidden below, the last is "(k more lines below)".
An empty file gives "(empty file)", and a start outside 1..n gives a message starting with "error". Use text.splitlines().

str_replace(text, old, new) returns (new_text, message).

If old occurs exactly once, replace it and return a message that starts with "ok" and contains "line L", the line where the match starts.
Otherwise return the text unchanged with a message that starts with "error". For no match, the message contains "no match". For several, it contains "k matches" and "line L" for every match.
An empty old is an error. Overlapping occurrences count, so "aa" occurs twice in "aaa".

def view(text, start=1, window=100):
    raise NotImplementedError

def str_replace(text, old, new):
    raise NotImplementedError

Tests, hints, verified solutions, and AI help, with Pro at $7.99 a month.

A small coding agent, end to end

Now put the pieces together. The cells below are a small coding agent on the toy repository from the sandbox part, whose median returns the upper middle element instead of the mean of the two middle ones. The agent has a summarized search, a windowed viewer, a unique-match editor, and the restricted executor for running code. Its scripted policy does what good agents do, in this order.

Find the function the issue names.
Open it.
Reproduce the bug before touching anything.
Edit.
Check that the reproduction now gives the right answer, and submit.

Then the grader runs the three tests in tests/test_core.py, which the agent never opened. In SWE-bench terms, test_median_even is the FAIL_TO_PASS test and the other two are PASS_TO_PASS tests.

Python · cell 5

The loop gives every action an observation, and the grader is the SWE-bench rule in miniature.

Python · cell 6

The reproduction step is the one to notice. An agent that edits before reproducing cannot tell a fix from a coincidence. One that writes and runs a reproduction has a check that tells it when to stop. §24.1 showed that such checks are worth as much as accuracy over a long task, because a long task fails only on the errors that are not caught and repaired. Interviewers who ask you to design a coding agent listen for exactly this step.

A fixed pipeline, and the 2026 numbers

In mid-2024 a paper asked whether an agent loop is needed at all. Recall the distinction of §24.1. In a workflow, code fixes the sequence of steps and the model only fills in content. In an agent, the model chooses the next step. Agentless (Xia et al. 2024) replaced the agent loop with a workflow of three fixed phases.

Localization narrows the repository to files, then to classes and functions, then to the lines to edit, using prompts and embedding retrieval.
Repair samples candidate patches as search-and-replace edits, ten for each of four sets of locations, so 40 per issue with GPT-4o.
Validation writes tests that reproduce the issue, runs them and the existing tests on every candidate, and discards the failures. It normalizes the survivors, ignoring whitespace and comments, and submits the most common one by majority vote.

Agentless resolved 32.00% of SWE-bench Lite (96 issues) at $0.70 per issue, more than every open-source agent of the time. The reason is that errors compound over steps (§24.1). With 2024's models, every extra decision an agent made was another chance to wander off. A pipeline that makes the decisions in code, samples many candidates, and lets tests choose among them removed most of those chances.

As models became more reliable per step, the balance moved back toward agents, which can explore, run anything, and recover. But the pipeline's central idea survived inside them, which is to sample several attempts and let tests choose.

By February 2026, comparing models fairly meant holding the harness fixed. The SWE-bench leaderboard's "bash only" view runs every model in the same minimal agent, mini-SWE-agent, whose only tool is a shell.

📌 SWE-bench Verified, bash-only harness (entries of February 2026, read October 20, 2026)

model	resolved	cost per instance	model calls per instance	cost per resolved issue
Claude Opus 4.5 (high)	76.8%	$0.75	32.9	$0.98
Gemini 3 Flash (high)	75.8%	$0.36	56.1	$0.47
MiniMax M2.5 (high)	75.8%	$0.073	60.5	$0.097
Claude Opus 4.6	75.6%	$0.55	28.9	$0.73

The cost per resolved issue is the cost per instance divided by the resolve rate, since every attempt is paid for and only the successes count. For the first row, $0.75 / 0.768 = 0.98$ . The best entries with harnesses of their own reach 79.2% (SWE-bench leaderboards).

Read all of these numbers with the benchmark's caveats (§22.5). OpenAI stopped reporting Verified in February 2026, citing flawed tests and contamination (OpenAI 2026). Epoch AI's own runs, in a harness of its own, top out at 83.5% for Claude Opus 4.7 in April 2026 (Epoch AI 2026).

The four rows are within 1.2 points of each other and a factor of ten apart in price. That is the trade-off between cost and accuracy that §22.5 asks every agent evaluation to report. The leaderboard also publishes per-instance results, and two more facts come out of them (our analysis of those files).

Failures cost more than successes. For Claude Opus 4.5, unresolved instances cost $1.16 on average against $0.63 for resolved ones, with a median of 38.5 model calls against 26.5. That is what you would expect if a stuck agent keeps trying until its budget runs out.
The models fail on different issues. At least one of the four models resolves 428 of the 500 instances, 85.6%. All four resolve only 323.

That gap between "some attempt succeeds" and "the one attempt we submit succeeds" is the subject of the next part.

Sampling many patches and choosing one

§18.2 separated two questions about many samples. Coverage asks whether any of $n$ samples is right. Selection asks whether you can pick the right one. Coding agents live on that distinction.

With the hidden tests in hand, coverage is pass@ $n$ , the probability that at least one of $n$ candidate patches resolves the issue. At deployment the hidden tests are not available. A system must choose with what it has. That may be reproduction tests it wrote itself, the existing tests, a learned verifier (a model that scores patches), or a vote. What it achieves is best-of- $n$ , the probability that the candidate it submits is correct.

Here is a simple model of a verifier. It gives every candidate a noisy score, and the system submits the candidate with the highest score. A correct patch scores $\delta + \sigma\xi$ and a wrong one scores $\sigma\xi$ . Here $\xi$ is an independent standard normal draw for each candidate, $\delta > 0$ is the margin by which correct patches score higher on average, and $\sigma$ is the noise.

Issue $i$ has its own probability $p_i$ that one sample is correct. On real benchmarks these probabilities pile up near 0 and near 1, with many issues almost always solved and many almost never. The cell draws 500 such issues and measures both curves, with $\delta = 1$ .

Python · cell 7

Coverage climbs from 0.399 to 0.786 at 32 candidates. With $\sigma = 0.5$ the verifier keeps much of that gain, reaching 0.721. With $\sigma = 1$ it reaches 0.607, about half of the gap, and with $\sigma = 3$ only 0.475. Every curve starts at pass@1 (up to sampling noise), because with one candidate there is nothing to choose. None can exceed pass@ $n$ , because a verifier cannot select a correct patch that nobody wrote.

This is why SWE-Gym (Pan et al. 2024) trained a verifier on agent transcripts alongside the agent, and reached 32.0% on SWE-bench Verified with it. It is also why every system that samples heavily spends as much effort on selection as on generation. When an interviewer asks how to spend more compute at test time, give this answer. More samples raise the ceiling, and a better verifier decides how close you get to it. The exercise computes these curves exactly.

Research

Test-time scaling with a verifier

Compute the two curves of the cell above exactly, for the verifier model just described. Implement bon_curve(p, n_max, margin, sigma), which returns two arrays of length n_max, bon and cov, where entry n - 1 refers to $n$ candidates.

Issue $i$ 's candidates are independently correct with probability p[i].
A correct candidate's score is distributed as $\mathcal{N}(\delta, \sigma^2)$ and a wrong one's as $\mathcal{N}(0, \sigma^2)$ , independently, with $\delta$ = margin. Best-of- $n$ submits the highest-scoring candidate.
cov[n - 1] is the mean over issues of pass@ $n$ , and bon[n - 1] is the mean over issues of the probability that the submitted candidate is correct.

Compute both exactly, with no Monte Carlo sampling. Handle sigma == 0, where any correct candidate outscores every wrong one, and stay accurate as sigma shrinks toward 0.

import numpy as np

def bon_curve(p, n_max, margin, sigma):
    raise NotImplementedError

Tests, hints, verified solutions, and AI help, with Pro at $7.99 a month.

Training coding agents

A repository with tests is a complete reinforcement-learning environment. The state is the repository and the processes in the sandbox. An action is a command or an edit, and an observation is its output. The reward arrives once, at the end, from the hidden tests. Training uses two tools from earlier in the chapter.

The token source mask. Each transcript is stored with a mask $m_i$ that is 1 on tokens the policy sampled and 0 on every other token, including everything a tool wrote (§24.1). The loss uses only tokens with $m_i = 1$ . So the thousands of tokens of file contents and test logs carry no policy gradient.
Group baselines. Each rollout's advantage, how much better it did than expected, comes from comparing it with the other rollouts on the same issue, as in GRPO (§18.4). An issue that every rollout solves, or none does, therefore contributes no signal.

The hard part is the data, a working environment for every task in which the tests actually run. Three published answers span the options.

Real issues with real environments. SWE-Gym (Pan et al. 2024) built 2,438 task instances from 11 Python repositories, each with an executable environment and tests. Fine-tuning on agent transcripts in it raised resolve rates by up to 19 points absolute. Adding a verifier trained on the same transcripts reached 32.0% on SWE-bench Verified.
Synthesized bugs. SWE-smith (Yang et al. 2025) turns the problem around. Start from a repository whose tests pass, and break it. A model can rewrite or subtly corrupt a function, a program can change the syntax tree, a past pull request can be undone, or several bugs can be combined. Every change that makes some test fail is kept, and the failing tests become its FAIL_TO_PASS set.
No execution at all. SWE-RL (Wei et al. 2025) runs reinforcement learning on the history of pull requests. The reward is $-1$ for a malformed answer. Otherwise it is the textual similarity between the model's patch and the merged one, measured with Python's difflib.SequenceMatcher. That needs no sandbox and scales to the whole history of open source, and Llama3-SWE-RL-70B reached 41.0% on SWE-bench Verified. The price is that the reward pays for resembling one particular fix. A different correct fix scores low, and a near-copy that fails its tests scores high.

The choice is a trade between realism and scale. Real issues are the most faithful and the most expensive to set up, synthesized bugs scale to tens of thousands of tasks, and similarity rewards need no sandbox but pay for the wrong thing.

🔬 Why synthesized bugs scale, and what frontier models add

All of SWE-smith's tasks from one repository share one environment. So its 50,000 instances from 128 repositories need about 500 times less storage than the 50 to 150 TB that one environment image per task, as SWE-bench builds them, would take. Rejection-sampling fine-tuning keeps only the transcripts that solved their task and trains on those. With 5,016 successful transcripts of Claude 3.7 Sonnet, it took Qwen 2.5 Coder 32B to 40.2% on SWE-bench Verified (Yang et al. 2025).

At frontier scale these pieces are combined with environments synthesized across many domains. Kimi K2 (Kimi Team 2025) was trained with large-scale synthesis of agentic data and joint reinforcement learning. It reports 65.8% on SWE-bench Verified without extended thinking.

The reward is also an attack surface. A policy optimized against tests learns whatever makes them pass, and the cheapest route is often not a fix. The hacks seen in practice are the ones §18.3 catalogued. They are exiting before the assertions run, special-casing the tested inputs, editing or deleting tests, and patching the test framework. The defenses follow from that list.

Keep hidden tests outside the sandbox, and restore any test files before grading.
Flag and inspect patches that touch test infrastructure, as the touches_tests list of Exercise 24.5b does.
Hold out tests that the policy never sees during training.
Measure cheating directly with tasks that are impossible by construction, as §22.5 describes, since any pass on an impossible task is a hack.

An interviewer who asks why a coding agent's reward went up while its patches got worse wants this list, in this order.

A 2026 coding-agent harness

The single shell of mini-SWE-agent is a measuring instrument. It holds the harness fixed so that models can be compared. The coding agents that engineers use every day wrap the same loop in much more machinery. Examples are Claude Code, OpenAI's Codex, and the open-source OpenHands. The authors of CodeAct are among OpenHands' developers, and its default agent acts with code actions inside a Docker container. "Design a coding agent" has become a standard interview question, and the machinery is not arbitrary. Each component answers a problem this chapter has already measured.

The table lists the components and the reason for each. Product details change every few months, and the deeper box below records how the shipping products implemented each component on October 20, 2026. The reasons do not change.

component	what it does	why it is there
repository memory file	a markdown file of build and test commands, conventions, and warnings, loaded at the start of every session	it is part of the prompt $P_0$ that every turn reads (§24.1), so it must be short and stable to stay in the prompt cache (§24.6)
plan or to-do tool	a list of steps with statuses that the agent writes and updates as it works	a plan that is checked and revised (§24.4), and progress the model restates, so compaction cannot lose it (§25.2)
view, search, and string-replace edit	windowed reads with line numbers, search that lists matches, unique-match edits	the agent-computer interface of this section and Exercise 24.5c
a shell in a sandbox	builds, tests, git, and anything else the task needs	code actions need an operating-system boundary (the sandbox part of this section), and running the tests is the detection step of §24.1
read-only subagents	explore the repository in a fresh context and return a summary	context isolation (§25.2, §24.7), and an explorer that cannot write cannot break anything
permission modes and rules	decide which calls run, which ask a person, and which are refused	a program's effects cannot be checked in advance, so approval moves into the harness, the permission engine of §24.6
hooks	deterministic code run at fixed events, able to block a tool call	rules that must hold whatever the model decides, such as refusing edits under `tests/` (the reward-hacking defense above)
compaction	near the context limit, clear old tool outputs, then summarize	the history processor of Exercise 24.5f, treated in general in §25.2
checkpoints	snapshot each file before an edit so the edit can be rewound	retries and search need a way back (§24.4)
background and cloud runs	run a task asynchronously, in its own machine	a long task needs its own sandbox and durable state (§25.4)

🔬 How shipping products implemented each component (October 2026)

Memory file. CLAUDE.md in Claude Code. AGENTS.md is an open format that Codex reads, and that Claude Code reads when a project has no CLAUDE.md. In a monorepo the AGENTS.md nearest the edited file wins.
Plan tool. Claude Code's task tools (TaskCreate, TaskUpdate), which replaced the earlier TodoWrite. They are on by default only for Opus 4.7, Sonnet 4.6, Haiku 4.5, and earlier models. Newer models track multi-step work without a written checklist, so the tools would only take up context.
Viewing and editing. Claude Code's Read and Edit, with search by read-only shell commands such as grep and find that run without approval.
Shell sandbox. Codex's sandbox modes read-only, workspace-write (the default for local work, with the network off), and danger-full-access. They are enforced by Seatbelt on macOS and by bubblewrap plus seccomp on Linux.
Read-only subagents. Claude Code's Explore and Plan subagents, which are denied Write and Edit.
Permissions. Claude Code's modes, among them Manual, Accept edits, Plan, and Auto, with allow, ask, and deny rules on commands. In Auto mode a classifier reviews actions in place of the user. Codex has the approval policies on-request and never.
Hooks. Claude Code's PreToolUse (a hook that exits with code 2 blocks the call), PostToolUse, PreCompact, and Stop.
Compaction. Claude Code clears older tool outputs first and then summarizes. OpenHands' summarizing condenser keeps the first events and the newest ones and summarizes the middle.
Checkpoints. Claude Code's checkpoints cover local files, not remote side effects such as deployments.
Background runs. Claude Code's background subagents, and cloud sessions in Anthropic-managed VMs.

One principle organizes the whole design. Whatever the model reads, it may or may not obey, and only the harness decides what actually happens. Claude Code's documentation says this of its own memory files. It treats them as context rather than as enforced configuration, and it points to a PreToolUse hook for anything that must be blocked.

So conventions go in the memory file. Rules that must hold go in permission rules and hooks, such as never pushing to the main branch, never editing tests during a training episode, or never reading ~/.ssh. The sandbox is the last line of defense when both fail. A memory file is also text that came with the repository, so a hostile one is an injected instruction (§19.3). That is why Claude Code asks before it loads imports that point outside the project.

✏️ What a memory file costs over a session

A 2,500-token memory file sits in the system prompt of a 60-turn session. Without caching it is read 60 times, $60 \times 2{,}500 = 150{,}000$ input tokens. Use the cache prices of §24.6 for most models, a write at 1.25 times the input price and a read at 0.1 times. Then the file is written once and read 59 times, $2{,}500 \times (1.25 + 59 \times 0.1) = 17{,}875$ token-equivalents, 8.4 times less.

Now let the harness put something that changes every turn, such as the current time or the live to-do list, in front of the file. The cache misses at that point on every turn. A harness that still asks for the prompt to be cached writes the file again every turn. The cost becomes $60 \times 2{,}500 \times 1.25 = 187{,}500$ , more than with no cache at all. Stable content goes first, and anything that changes goes at the end.

The same harness, given a budget of hours or days rather than minutes, can run a machine-learning project. Agents that run whole research projects, from proposing an experiment to writing it up, are measured on benchmarks such as MLE-bench. §26.5 reads those benchmarks and the evolutionary coding systems built on the same loop.

Computer and browser use

Most software has no API, no programming interface an agent can call. Booking systems, internal dashboards, spreadsheets, and design tools are built for people, who look at a screen and use a mouse and keyboard. A computer-use agent does the same. Its actions are a user's actions, which are moving the pointer to $(x, y)$ and clicking, typing a string, pressing a key combination, scrolling, and dragging. Its observations are what the screen shows. The hard design question is what form "what the screen shows" should take.

📐 Screen observation formats

Raw HTML or DOM. The page's markup, or the tree of elements the browser builds from it (the document object model), possibly cleaned of scripts and styles. Complete for web pages and useless for desktop applications.
Accessibility tree. The tree that the operating system or browser builds for screen readers. Each node has a role (button, link, text box), a name, states (focused, checked, disabled), and a bounding box. It is usually serialized as one numbered line per visible interactive element.
Set-of-mark screenshot. A screenshot with a numbered mark drawn on each interactive element, so the agent can act by number ("click 12") while still seeing the pixels. Yang et al. 2023 introduced marks on image regions for visual grounding in general, and web and computer agents adopted them.
Raw screenshot with coordinate actions. Pixels only. The agent must say where to click, in pixel coordinates of the image it was shown.

format	what the agent sees	tokens per step	how it points	how it fails
raw HTML	everything in the markup, little of the layout	often tens of thousands or more on real pages	by element id or selector	too long; misses canvases and rendered state
accessibility tree	roles, names, and boxes of interactive elements	hundreds to a few thousand	by element number	apps with poor accessibility data; no visual detail
set-of-mark	the pixels plus numbered elements	an image plus a short legend	by mark number	marks clutter small elements; needs element detection
raw screenshot	the pixels only	one image (1,560 to 2,691 tokens at 1920 by 1080 on Claude)	by pixel coordinates	grounding errors, coordinate scaling bugs

The benchmarks show the trade-offs moving over time. OSWorld (Xie et al. 2024) put agents in full virtual machines with 369 tasks across desktop and web applications. Each task is checked by a script that inspects the final state of the machine. At its release, people solved 72.36% of the tasks. Among the models, a text-only one reading the accessibility tree did best, GPT-4 at 12.24%. GPT-4V managed 5.26% on raw screenshots and 11.77% with set-of-mark. The authors named GUI grounding, turning "click the Save button" into the right coordinates, as a main cause of failure.

Grounding, and the coordinate bug

Grounding is the step from an intended element to an action the computer accepts. With an accessibility tree it is a lookup, since element 12 has a bounding box and the agent clicks its center. With a raw screenshot the model itself must output pixel coordinates, and two separate things can go wrong. The model can misjudge where the element is, a perception error that training on large amounts of screen data reduces. Or the system can misread the coordinates the model gave, a plain software bug that no training fixes.

The bug comes from downscaling. Providers shrink large images before the model sees them, and the model answers in the coordinates of the image it saw. On Claude's standard resolution tier, the documented rule shrinks a 1920 by 1080 screenshot to 1456 by 819. That is the largest size with the same aspect ratio that fits two limits, 1,568 pixels on the long edge and 1,568 tokens for the image (Claude vision documentation). Scaling only to the edge limit would give 1568 by 882, which is wrong, and the documentation warns about exactly this.

✏️ A click that lands 232 pixels away

A Submit button's center is at $(960, 541)$ on a 1920 by 1080 screen. The model sees the 1456 by 819 image and correctly answers $(728, 410)$ , since $960 \cdot 1456/1920 = 728$ and $541 \cdot 819/1080 = 410.3$ . Clicking at $(728, 410)$ on the screen misses by 232 pixels horizontally and 131 vertically.

The right click maps back by the inverse ratios, $x = 728 \cdot 1920/1456 = 960$ and $y = 410 \cdot 1080/819 = 540.7$ , which rounds to $541$ . Use separate ratios for $x$ and $y$ , since rounding the resized size can change the aspect ratio slightly. Clamp to the screen at the far edges.

The widget below plays this example on screens of four sizes and on both tiers. Switch between using the model's coordinates as given and mapping them back.

So a computer-use agent that "cannot click straight" may have a perfect model and a broken harness. Interviewers like this bug because it separates people who have built such an agent from people who have only read about one.

Interview

Serialize a screen for the model

Implement two functions that sit between a computer and a computer-use model. The first turns an accessibility tree into numbered lines within a character budget. The second maps a click from the model's image back to the screen.

serialize(tree, budget) takes an accessibility tree of nested dicts with keys "role", "name", "bbox" ([x0, y0, x1, y1] in screen pixels), "visible", and "children".

List each node that has an interactive role (button, link, textbox, checkbox, combobox, menuitem, tab, radio), is visible, has only visible ancestors, and has a box of positive width and height.
Number them 1, 2, 3, ... in document order, the order of a depth-first walk that visits a node before its children. Write one line each as [k] role "name".
Keep lines in that order while the text joined by newlines stays within budget characters, and stop at the first line that does not fit. If any were cut, add a last line [+m more], which does not count toward the budget.
Return (text, boxes), where boxes[k] is the center of element k's box as a tuple of two floats, for every listed element including the cut ones.

to_screen(x, y, model_size, screen_size) maps a click from the model's image to the screen. With model_size = (wm, hm) and screen_size = (ws, hs), return (floor(x * ws / wm + 0.5), floor(y * hs / hm + 0.5)), clamped to 0..ws - 1 and 0..hs - 1.

INTERACTIVE = {"button", "link", "textbox", "checkbox", "combobox", "menuitem", "tab", "radio"}

def serialize(tree, budget):
    raise NotImplementedError

def to_screen(x, y, model_size, screen_size):
    raise NotImplementedError

Tests, hints, verified solutions, and AI help, with Pro at $7.99 a month.

What screenshots cost

Images are expensive, and the loop makes them more so. Claude prices an image by patches. After any downscaling, an image of $w_{\text{px}}$ by $h_{\text{px}}$ pixels costs

c_{\text{img}} = \left\lceil \frac{w_{\text{px}}}{28} \right\rceil \left\lceil \frac{h_{\text{px}}}{28} \right\rceil

visual tokens, one per 28 by 28 block of pixels. Claude 4.7 and later models use a high-resolution tier, with limits of 2,576 pixels on the long edge and 4,784 tokens, so they take a 1920 by 1080 screenshot at full size. It costs $69 \times 39 = 2{,}691$ tokens there. On the standard tier it shrinks to 1456 by 819 and costs $52 \times 30 = 1{,}560$ tokens. §27.1 explains the idea of cutting an image into patches, and §31.5 covers how open vision-language models turn images into tokens.

The cost of one screenshot is only the start, because every turn reads again the screenshots it keeps. Suppose an agent takes one screenshot per turn for $T$ turns, and turn $t$ reads the last $\min(t, k)$ of them, for some $k \le T$ . With $k = 3$ and $T = 5$ the turns read 1, 2, 3, 3, and 3 screenshots, 12 in all.

∂ Proof

For $t \le k$ turn $t$ reads $t$ screenshots, which sum to $k(k+1)/2$ . For $t > k$ each of the $T - k$ remaining turns reads $k$ , which adds $k(T - k)$ . The total is $k(k+1)/2 + kT - k^2 = kT - k(k-1)/2$ . With $k = T$ this is $T^2 - T(T-1)/2 = T(T+1)/2$ . Check with $k = 3$ and $T = 5$ , which gives $15 - 3 = 12$ , as counted above. ∎

In plain words, keeping every screenshot makes the cost grow with the square of the number of turns. That is the same growth as the transcript's token count in §24.1. Keeping the last $k$ makes it grow linearly, about $k$ screenshots per turn.

This is the screen version of the coding result above, where keeping only the last five observations beat keeping them all. §25.2 treats context trimming in general. The cell checks the sizes and totals.

Python · cell 8

The cell confirms both image sizes and both totals. The widget plots how many observations each turn reads, so the shaded area is the episode's total. Press Play to watch an episode grow, then vary the format, the number of turns, and how many observations are kept.

Here is the patch arithmetic in PyTorch, the way a vision encoder cuts an image into tokens. Pad the image on the bottom and right to multiples of 28. Unfold it into non-overlapping 28 by 28 blocks, and project each block's $3 \cdot 28 \cdot 28 = 2{,}352$ numbers to one embedding. How any particular provider's encoder embeds the patches is not public, but the count is the one the formula gives.

🔥 In PyTorchfor a machine with PyTorch

Native agents, many rollouts, and where the numbers stand

The first computer-use systems wrapped a general model in prompts and added a separate model for grounding. A native agent is one model trained end to end to see screens and act on them, which removes the hand-off between a planning model and a separate grounding model.

The biggest jump came from choosing among attempts rather than from a better model. A single computer-use rollout is brittle, since small errors compound over a long episode, and different rollouts fail in different places. That is the same coverage gap as on the SWE-bench leaderboard. Agent S3 with Behavior Judge (Gonzalez-Pumariega et al. 2025) runs several rollouts and turns each into a short narrative of what the agent did and what changed on the screen. A judge model, a language model asked to compare outputs (§22.4), reads the narratives and picks one. The system reached 72.6% on OSWorld, above the 72.36% human figure.

Computer use inherits every threat of the sandbox part and adds one. The agent reads web pages, so every page is a possible prompt injection (§19.3). An agent with a logged-in browser holds exactly the private data and the outbound channel that make the lethal trifecta dangerous. So run it in a virtual machine with no saved credentials. Require a person's approval for purchases, messages, and deletions. And evaluate it for dangerous autonomous and cyber capabilities before deployment (§19.4).

Where it breaks

Benchmarks drift away from the work. Leaked fixes, weak or overly specific tests, contamination, and websites that change after a benchmark is written all move scores without moving ability (§22.5).
Tests that do not test the fix. A patch that passes FAIL_TO_PASS can still be wrong if those tests are narrow, and reinforcement learning against such tests pays for wrong code.
Agents that edit the grader. Test edits, framework patches, and early exits, unless the tests live outside the agent's reach.
Sandbox escapes and leaks. Restrictions inside the same process, path checks by prefix, secrets in the environment, and unrestricted outbound network access.
Grounding. Clicks that miss because the model misjudged the element, or because the system forgot to rescale its coordinates.
Context. Screenshots and file contents dominate the context unless old observations are dropped or condensed, and every turn reads again what is kept.
Rules the harness does not enforce. An instruction in a memory file or a prompt is a request the model may ignore, and an injected file can countermand it. Anything that must hold belongs in permissions, hooks, and the sandbox.

🧭 What to remember

A code action lets one action compose many tool calls. Tool calling needs at least $L_{\text{cp}} + 1$ turns for a chain of $L_{\text{cp}}$ dependent calls, while a program needs two when the dependencies are computable, and only what it prints enters the context. JSON calls remain better for single calls and for actions that must be checked or approved before they run.
Model-written code needs a sandbox, defined by an isolation boundary, resource limits, a filesystem scope, a network policy, and minimal credentials. Containers share the host kernel, while gVisor and microVMs such as Firecracker (under 125 ms to start, under 5 MB of overhead) isolate more strongly at little cost. Restrictions inside the same process and string path checks are not security.
SWE-bench grades by execution. A patch resolves an instance if and only if every FAIL_TO_PASS test passes and every PASS_TO_PASS test still passes, with missing tests counted as failures. Cost per resolved issue is cost per instance divided by the resolve rate.
The agent-computer interface matters. Short file windows, summarized search, linted unique-match edits, and trimmed history beat more context, and the principles outlived the specific numbers as results rose from 1.96% to about 77%. Production harnesses add a repository memory file, a plan tool, read-only subagents, permissions and hooks that enforce what instructions only request, and compaction.
More samples raise coverage, pass@ $n$ , and a verifier decides how much of it best-of- $n$ keeps. Coding agents are trained with reinforcement learning on masked transcripts with group baselines, in environments built from real issues (SWE-Gym), synthesized bugs (SWE-smith), or patch similarity (SWE-RL). The tests are an attack surface, so keep them out of reach and check the diffs.
Computer-use agents observe HTML, accessibility trees, set-of-mark screenshots, or raw pixels. Grounding and coordinate rescaling cause many failures. A screenshot costs $\lceil w/28\rceil\lceil h/28\rceil$ tokens, and an episode's screenshot cost grows quadratically unless old ones are dropped. Selection across several rollouts carried OSWorld past the human figure.
Next, §24.6 standardizes how agents connect to tools and to each other with MCP and A2A, and puts a price on every step with caching, parallel calls, retries, and budgets.

Wang et al. (2024). Executable Code Actions Elicit Better LLM Agents. CodeAct, the M³ToolEval comparison of code, JSON, and text actions, and the training data for code agents.
Jimenez et al. (2023). SWE-bench: Can Language Models Resolve Real-World GitHub Issues?. How 90,000 pull requests became 2,294 graded tasks, and the FAIL_TO_PASS and PASS_TO_PASS design that most coding benchmarks copy; current results are on the SWE-bench leaderboards.
Yang et al. (2024). SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering. The agent-computer interface, its commands, and the ablations that showed interface design matters.
Xia et al. (2024). Agentless: Demystifying LLM-based Software Engineering Agents. The fixed localize, repair, and validate pipeline that matched the agents of 2024, and a careful look at flawed benchmark tasks.
Yang et al. (2025). SWE-smith: Scaling Data for Software Engineering Agents. Synthesizing training environments by breaking working repositories; compare Pan et al. (2024), Training Software Engineering Agents and Verifiers with SWE-Gym, and Wei et al. (2025), SWE-RL.
Xie et al. (2024). OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments. Real virtual machines, execution-based checks, and the comparison of screenshots, accessibility trees, and set-of-mark; read with Gonzalez-Pumariega et al. (2025), Scaling Agents for Computer Use.
Agache et al. (2020). Firecracker: Lightweight Virtualization for Serverless Applications. The microVM design behind fast, strongly isolated sandboxes.
Anthropic (2025). Writing effective tools for AI agents. Practical interface design for agents, from response formats and truncation to evaluation-driven tool improvement; read with the Claude Code documentation's How Claude Code works for a whole production harness.

Made for larger screens

Code agents and computer use

Code as the action space

Sandboxes

Stay inside the sandbox

Software engineering agents

How SWE-bench is built and graded

Grade a SWE-bench run

The agent-computer interface

A history processor that keeps calls and results paired

A history processor

What still holds in 2026

The editor tool

A small coding agent, end to end

A fixed pipeline, and the 2026 numbers

Sampling many patches and choosing one

Test-time scaling with a verifier

Training coding agents

A 2026 coding-agent harness

Computer and browser use

Grounding, and the coordinate bug

Serialize a screen for the model

What screenshots cost

Native agents, many rollouts, and where the numbers stand

Where it breaks

Read more

Watch next