An agent is a language model that acts. It reads a situation, calls a tool or runs a command, looks at the result, and repeats until the job is done. Evaluating an agent means running it on test tasks inside a test environment, usually several times per task, and grading what it did. This section shows how to do that so that the number means what you think it means. It matters because agent scores now decide launches and back research claims, and they are easier to get wrong than any other score in this chapter.

Three published results show how an agent score can mislead. The first is about reliability. In June 2024 a team at the company Sierra released τ-bench, a test in which an agent serves simulated customers of an online store and of an airline (Yao et al. 2024). The agent reads a policy document, talks with a customer played by another language model, and calls tools that read and change a database. The best agent they tested was gpt-4o with function calling, meaning the model writes structured tool calls that the surrounding code executes. It completed 61.2% of the store tasks on a single try. Then the authors ran every task eight times, and fewer than 25% of the tasks were completed on all eight tries. A real customer meets your agent many times. Which of the two numbers belongs on the slide?

The second result is about grading. SWE-bench asks an agent to fix a real GitHub issue in a Python repository, and grades the fix by running tests the agent never sees (Jimenez et al. 2023). Aleithan et al. (2024) read every fix that one agent, the SWE-agent program driving GPT-4, had gotten right. In 32.67% of them the solution was already written in the issue or its comments. Another 31.08% passed only because the tests were too weak to notice a wrong fix. Without those tasks the agent's success rate fell from 12.47% to 3.97%, a drop of more than two thirds. The third result is the shortest. On τ-bench's airline tasks, an agent that does nothing at all passes 38% of them (Zhu et al. 2025).

None of these failures is bad luck in which tasks were drawn, the sampling error of §22.2. None is test data that leaked into training, the contamination of §22.3. §22.1 split the error of any evaluation into five parts, validity, sampling, leakage, grader, and environment. The SWE-bench audit and the do-nothing agent are failures of the last two. The grader accepted wrong fixes, and the benchmark was built so that an empty answer passes. The τ-bench gap is something new. A model answering one question has a single score, but an agent that is run many times has a reliability, and every run has a price. Does the agent succeed every time, or only on average? And what did each success cost?

This section builds an agent evaluation piece by piece. It starts with the words for the parts (tasks, trials, environments, graders), then shows how to grade an outcome and a path, and how to keep the environment from moving the score. Then it derives the two numbers every agent report needs and most get wrong, a reliability metric called $\text{pass}^k$ and the trade-off between cost and accuracy. It ends with what each major agent benchmark measures and how each one breaks.

What an agent evaluation measures

Stated precisely, an agent is a model in a loop. It observes an environment and chooses an action, such as a tool call, a shell command, or a click. It receives the result and repeats, until it decides to stop or runs out of budget. How to build such a loop is the subject of Chapter 24, from the agent loop and function calling to code agents and computer use. Here we need only the loop itself and the words for testing it.

Start with one test case. A customer of an online store asks the agent to cancel order W1 and refund 54.04 dollars. The store's database starts with W1 marked pending. Run the agent on this request three times. Each run leaves a full record of what the agent said and did, and leaves the database in some final state.

• In the first run the agent cancels W1 and refunds the money.
• In the second it tells the customer "done" but never calls the cancel tool.
• In the third it cancels W1 and also cancels another customer's order.

A grader looks at each run and returns pass or fail. Only the first run should pass. Every agent evaluation is many runs like these, and the box names their parts.

These terms follow Anthropic's engineering guide to agent evaluations (Anthropic 2026). The scaffold is the one people forget. When you evaluate "an agent", you evaluate a model and a scaffold together, under one budget, in one version of one environment.

Now the score itself. §22.1 treated an evaluation as an estimator, an average of grader scores over test items that stands in for the quality we care about. For agents the items are tasks, and each task is run several times. Say our cancellation task passed 1 of its 3 trials and a second task passed 3 of 3. The score is the average of the two tasks' success rates, $(1/3 + 1)/2 = 0.67$.

In general, write the benchmark as $J$ tasks indexed by $j$, and let $p_j$ be the probability that one trial of task $j$ succeeds. This probability belongs to the whole protocol, meaning the model, the scaffold, the tools, the environment version, the budget, and the grader. Change any of them and $p_j$ changes. The benchmark's single-trial success rate is the average $\bar p = \frac{1}{J}\sum_j p_j$. Run $n_j$ trials of task $j$ and count $c_j$ successes. The usual estimate is

In plain words, take each task's success rate over its trials, then average over tasks.

Everything that goes wrong in this section goes wrong in one of three places. The grader can disagree with the truth, so a trial "passes" though the customer was not helped. The environment can make $p_j$ measure something other than the agent's skill, for example through a sandbox that crashes or a task whose answer is visible. And the single number $\bar p$ can be the wrong summary, because it hides how reliable the agent is from trial to trial and what each success cost.

Teams run agent evaluations for two purposes that pull in opposite directions, and they keep them apart.

The two are read differently. A capability eval is read through its mean and the interval around it. A regression eval is read through its failures, and it needs the reliability metric of this section, because "passed once" is not the same as "still works". §22.6 builds regression suites and release gates for your own product on top of what we derive here.

Grading the outcome and grading the path

A grader can look at two things, the outcome and the transcript. Most agent benchmarks grade the outcome. The outcome is what the user cares about, and there are usually many valid ways to reach it. Four kinds of outcome grader are common.

• Hidden tests. SWE-bench applies the agent's patch to the repository and runs two sets of tests the agent never sees (Jimenez et al. 2023). The FAIL_TO_PASS tests failed before the real fix and must pass now. The PASS_TO_PASS tests passed before and must still pass, which catches a patch that fixes the issue by breaking something else.
• State comparison. τ-bench writes down, for each task, the one database that the policy and the customer's wishes allow, and compares the final database with it. Its reward is a product $r = r_{\text{action}} \times r_{\text{output}}$. Here $r_{\text{action}}$ is 1 if the final database equals the goal database, and $r_{\text{output}}$ is 1 if the agent's messages contain every required piece of information, such as a refund amount (Yao et al. 2024).
• Execution scripts. OSWorld gives each of its 369 computer tasks its own script, which inspects files, application settings, and screen state after the trial (Xie et al. 2024).
• Answer match. GAIA (Mialon et al. 2023) and BrowseComp (Wei et al. 2025) ask questions with short, checkable answers and compare the agent's answer with a reference.

State comparison needs a precise statement, because the obvious ways to code it are wrong. Think of the database as a set of records. Each record is a set of named fields with values, such as order W1 with status "pending" and refund 0. A task's goal is a patch, the list of fields the task must set and the values they must take. For the cancellation task the patch sets W1's status to "cancelled" and its refund to 54.04.

In plain words, the agent must make exactly the changes the task asks for, with exactly the right values, and no others. Two choices in the definition do the work. Comparing the whole final state with $s_0 \oplus \Delta^\star$, rather than checking only the fields in the patch, catches side effects. Comparing exact values, rather than checking that something changed, catches wrong values.

Let's test both claims on a toy store with two pending orders and the task "cancel W1 and refund 54.04". We run four agents through three graders of increasing strictness. One agent does the task right, one does nothing, one only claims to have done it, and one also cancels W2. The first grader reads the agent's messages for a claim of success. The second checks only the fields the goal names. The third is the full-state grader of the box.

The grader that reads the agent's claim passes the agent that lied. The grader that checks only the goal's fields passes the agent that cancelled a second customer's order. Only the full-state grader passes exactly the correct agent.

The last line of output is the subtle one. Some tasks have the correct outcome "change nothing". An example is a request the policy forbids, such as changing a non-refundable ticket, where the agent must refuse. On such a task the do-nothing agent is right and should pass. That is not a bug in the grader. It becomes a flaw in the benchmark when such tasks are a large share of it, because then doing nothing earns a high score. Zhu et al. (2025) found that 38% of τ-bench's airline tasks and 6% of its retail tasks are of this kind. An agent that returns empty responses scores 38% on airline, above the 35.2% of the gpt-4o agent in the τ-bench paper.

In an interview, expect to be asked whether 40% on some agent benchmark is good. The first answer is a question back. Compared with what floor?

import copy

def grade_state(initial, final, goal, ignore=(), required_outputs=(), messages=()):
    # Placeholder: trusts that the agent did something.
    return final is not None and final != initial

When the path matters

An outcome grader cannot see how the outcome was reached, and sometimes the path is the point. τ-bench's authors note that a reward of 1 is necessary but not sufficient for a good episode. An agent can issue the right refund without first getting the customer's explicit confirmation, which the policy requires, and the database will look exactly right. Safety rules are the same. Take "never delete a production table". An agent that deletes the table and then restores it from a backup leaves a final state that looks untouched, so only the transcript shows the violation.

Trajectory grading checks the transcript against rules. The cleanest form has three parts, the actions the task requires, an order among them, and actions that are forbidden. Here is a small case. To cancel an order, the agent must verify the customer's identity and look up the order, in either order, and only then cancel. Writing "$a$ before $b$" as a pair, the rules are (verify, cancel) and (look up, cancel). Pairs like these define a partial order. It fixes some orderings and leaves the rest free, so "verify, look up, cancel" and "look up, verify, cancel" both pass while "cancel, verify, look up" fails.

Partial credit comes from the same rules. AgentBoard (Ma et al. 2024) defined a progress rate, the fraction of required subgoals the agent reached. An agent that verifies and finds the order but fails to cancel then scores above one that fails at the first step. Partial credit is what lets a capability eval show progress before agents can finish whole tasks.

The third kind of grader reads the transcript with a language model and scores it against a rubric, such as "the agent explained when the refund will arrive" or "the agent stayed within policy". Everything §22.4 says about model judges holds here. Their verdicts carry biases toward position, length, and their own outputs, and they must be checked against human labels before they are trusted. Two things are new for agents. Agent transcripts are long, so a judge can miss something buried in them. And the judge can itself be an agent that opens files and inspects the workspace. Zhuge et al. (2024) built such a judge for 55 AI development tasks with 365 requirements, arranged so that some requirements depend on others. They found it far more reliable than a judge that only reads, and as reliable as their human evaluation baseline.

Anthropic's guide adds a warning from practice (Anthropic 2026). Graders that demand a specific sequence of tool calls are brittle, because capable agents find valid paths the designer did not foresee. Grade what the agent produced, and keep path checks for what must or must not happen along the way. Exercise 22.5b implements those checks.

def grade_trajectory(actions, required, order=(), forbidden=()):
    # Placeholder: only checks that each required action appears somewhere.
    progress = sum(a in actions for a in required) / len(required) if required else 1.0
    return progress == 1.0, progress

Most real tasks combine graders. An outcome check decides pass or fail, path rules enforce policy and safety, and a rubric judge scores the quality of the conversation. The table compares them.

Environments that hold still

A score belongs to the environment as much as to the agent. So the environment must be the same for every trial and every agent, and it must not leak the answer. The table lists what has gone wrong in practice, sorted by which way it pushes the score.

The defenses are engineering, and they are as much a part of the benchmark as its tasks.

1. One pinned container image per task. A container image is a frozen copy of a small operating system with its software installed. SWE-bench's harness runs each task in its own image, with the repository's dependencies at fixed versions. A test then cannot fail because a library released a new version.
2. A fresh environment per trial. Reset the database, the file system, and the browser profile. Never let one trial see another's files, caches, or history.
3. Self-hosted or recorded worlds. WebArena runs its four websites (a store, a forum, a code host, and a content manager) in its own containers rather than on the live web (Zhou et al. 2023). A task written in 2023 still sees the same page in 2026.
4. A network policy. Allow the hosts a task needs and block the rest, in particular sites that host the benchmark. Scan transcripts for the benchmark's name. An agent that finds the answer key online is the test-time version of the contamination in §22.3.
5. A controlled simulated user. In τ-bench the agent decoded at temperature 0 and the simulated user at temperature 1.0, so trials of a task differ mainly in how the user words things (Yao et al. 2024). Record the user model, its prompt, and its seed with each trial.
6. Stated and enforced budgets. Set a step limit (30 actions in τ-bench), a token limit, and a wall-clock limit, and report them with the score. A larger budget is a different protocol.
7. A reference solution per task. A task that a known-good solution cannot pass is broken. With frontier agents, a pass rate of zero over many trials more often means a broken task than an incapable agent (Anthropic 2026).

Flaky infrastructure and the rerun trap

Some trials die for reasons that have nothing to do with the agent. A container fails to start, the tool server times out, or a rate limit runs out. Say a task has $p_j = 0.6$ and 5% of trials crash this way. If a crash is scored as a failure, the task's measured rate is $0.95 \times 0.6 = 0.57$. The agent lost 3 points it did not deserve to lose.

The fix is to rerun crashed trials, and only crashed trials. With one rerun, a trial succeeds if the first attempt runs and succeeds, or if it crashes and the rerun runs and succeeds. That has probability $0.95 \times 0.6 + 0.05 \times 0.95 \times 0.6 = 0.5985$, within 0.15 points of the truth.

The tempting variant is to rerun every failed trial "in case it was a flake". That measures something else. A task now counts as solved if either of two attempts succeeds, which is the agent's pass@2, the chance that at least one of two tries succeeds. In the example it gives $1 - (1 - 0.57)^2 = 0.815$, 21 points too high. The box states all three cases in general.

Proof. With crashes scored as failures, a trial succeeds when it runs and the agent succeeds, with probability $(1-f)p_j$. With up to $r$ reruns of crashes, the trial succeeds when, for some $i$ from 0 to $r$, the first $i$ attempts crash and attempt $i+1$ runs and succeeds. Summing $f^i(1-f)p_j$ over $i$ gives $(1-f^{r+1})p_j$. Rerunning every failure turns a task into two independent attempts that each succeed with probability $(1-f)p_j$, and the task counts as solved unless both fail. Average each over tasks. $\square$

So the harness must tell a crash from a failure, which is why good harnesses log infrastructure errors separately from agent errors. An interviewer who asks how you handle a flaky sandbox wants to hear exactly this. Rerun crashes, never failures, and report the crash rate. The cell checks all three formulas on 200,000 simulated tasks whose difficulties differ.

A 5% crash rate costs 3 points. Rerunning only the crashes recovers nearly all of them. Rerunning every failure adds 14 points that the agent did not earn on a single attempt. That is fewer than the 21 points of the one-task example, because these simulated tasks differ in difficulty. A second try helps little on a task the agent almost never solves, and not at all on one it always solves. That effect of spread across tasks returns in the reliability part below.

Tasks come in clusters

A standard error says how far a score would move if the benchmark's tasks were drawn again. The usual formula, the spread of the task scores divided by $\sqrt J$, assumes the tasks are independent. Real agent tasks come from the same repository, the same website, or the same template, and tasks in such a cluster tend to succeed or fail together. So they carry less information than the same number of independent tasks.

A tiny case shows how much less. Take four tasks, three from one repository and one from another. If each repository is either easy or hard for the agent as a whole, the four tasks are really two pieces of evidence, and the first piece counts three times. In a sense made precise below, these four tasks are worth 1.6 independent repositories, closer to one than to two.

The fix is the clustered standard error of §0.11, which §14.7 applied to questions that share a reading passage. It adds up the deviations from the mean within each cluster before squaring them. For task scores $x_j$ with mean $\bar x$, grouped into $C$ clusters indexed by $r$,

In plain words, deviations in one cluster that share a sign add up instead of cancelling, and the error bar grows to match. The factor $C/(C-1)$ is a small-sample correction.

Agent benchmarks are extreme cases. Of SWE-bench Verified's 500 tasks, 231 come from Django and 75 from SymPy, and the other ten repositories share the remaining 194. How much information about codebases in general is left? A simple model answers it. Suppose each repository $r$ has its own difficulty shift $u_r$, drawn independently with variance $\sigma_u^2$ and shared by all $J_r$ of its tasks.

Proof. The $u_r$ are independent, so the variance of the sum is the sum of the variances, $\frac{1}{J^2}\sum_r J_r^2\sigma_u^2$. A mean of $m$ independent effects has variance $\sigma_u^2/m$, and matching the two gives $m = J^2/\sum_r J_r^2$. $\square$

For the four-task case, $J_{\text{eff}} = 4^2/(3^2 + 1^2) = 16/10 = 1.6$. For SWE-bench Verified, $\sum_r J_r^2 = 64{,}600$ and $J_{\text{eff}} = 250{,}000/64{,}600 = 3.9$.

The cell simulates this. Each repository gets a random difficulty with a standard deviation of 0.8 on the log-odds scale. That is the scale inside a logistic function, where a shift of 0.8 moves a task that succeeds 80% of the time to about 64% or 90%. Each task also gets its own random difficulty, with a standard deviation of 1.0, around a base log-odds of 1.4, so the mean score is about 75%. Each of 2,000 draws recomputes the score and both standard errors.

Under this model the standard error that assumes independence says 1.9 points, the clustered one about 4.2, and the true spread of the score over fresh draws of repository effects is about 6.4. The clustered estimate is closer but still low. A clustered standard error computed from only twelve clusters, one of them nearly half the data, is itself biased downward.

Which error bar is right depends on the claim, the question of §22.2. For "this agent on these twelve repositories", the repository effects are fixed and the independent-tasks error is the right one. For "this agent on Python repositories in general", the honest error bar is several times wider, and no number of extra Django tasks will narrow it. Interviewers like this point because it turns a vague "the benchmark is biased" into a number.

Reliability, and the metric pass^k

Now back to the τ-bench gap from the opening, 61.2% on one try against under 25% on all eight. A deployed agent meets the same kind of task again and again. A customer who is helped on Monday and harmed on Tuesday does not average the two. Say an agent solves some task 90% of the time and a customer brings it that task eight times. If the tries are independent, the customer is helped every time with probability $0.9^8 = 0.43$. A 90% agent fails this customer more often than not.

The metric that captures this asks for every try to succeed. For one task it is the success probability raised to the number of tries, and for a benchmark it is the average of that over tasks.

Compare it with pass@$k$ from §18.2, the probability that at least one of $k$ trials succeeds, $1 - (1-p_j)^k$ for one task. For the 90% task, pass@$8$ is $1 - 0.1^8$, essentially 1, while $\text{pass}^8$ is 0.43. Both metrics equal $\bar p$ at $k = 1$. As $k$ grows, pass@$k$ rises toward the fraction of tasks the agent can ever solve, and $\text{pass}^k$ falls toward the fraction it always solves.

They answer different product questions. pass@$k$ is the right number when something at deployment can pick the success out of $k$ attempts, such as unit tests for a coding agent. $\text{pass}^k$ is the right number when every attempt reaches a user and every failure costs something, such as a support agent changing a real account. Which of the two you would report for a given product is one of the most common agent questions in interviews.

Estimating pass^k without bias

We never know $p_j$. We run $n$ trials of a task and see $c$ successes. The obvious estimate, $(c/n)^k$, is biased. The fix is the counting argument that §18.2 used for pass@$k$, where the estimate is the fraction of groups of $k$ trials that contain a success. Here, instead of raising the observed rate to a power, count the fraction of groups of $k$ trials that are all successes. With 3 successes in 4 trials and $k = 2$, there are 6 pairs of trials, 3 of them both successes, so the estimate is $3/6 = 0.5$ rather than $(3/4)^2 = 0.5625$.

Proof. Choose a set $S$ of $k$ of the $n$ trials uniformly at random, independently of their outcomes. Given the outcomes, the probability that every trial in $S$ succeeded is the number of $k$-subsets of the $c$ successes over the number of $k$-subsets of all $n$ trials, $\binom{c}{k}/\binom{n}{k}$. Taking expectations over the outcomes, by the law of total probability,

The trials are i.i.d., independent and identically distributed. Because $S$ is chosen without looking at the outcomes, the $k$ trials in $S$ are themselves $k$ i.i.d. trials, and they all succeed with probability $p^k$. $\square$

Writing out the binomial coefficients gives the product form, $\binom{c}{k}/\binom{n}{k} = \frac{c!\,(n-k)!}{(c-k)!\,n!} = \prod_{i=0}^{k-1}\frac{c-i}{n-i}$. The product never forms a large number, so it is safe for any $n$. The estimator needs $n \ge k$. With $n = k$ it is the plain indicator "all $n$ trials passed", which is unbiased but as noisy as a single coin flip. With $n \gt k$ it averages that indicator over all $k$-subsets, which can only lower the variance.

Which way is the plug-in $(c/n)^k$ biased? Jensen's inequality says that for a convex function (one that curves upward), the average of the function is at least the function of the average. The function $u \mapsto u^k$ is convex on $[0, 1]$, so $\mathbb{E}[(c/n)^k] \ge (\mathbb{E}[c/n])^k = p^k$, with equality only for $k = 1$ or $p \in \lbrace 0, 1\rbrace$. The plug-in overstates reliability. That is the opposite direction from pass@$k$, whose plug-in $1 - (1 - c/n)^k$ §18.2 showed to be biased low. The two plug-ins err in opposite directions, so no rule of thumb corrects both. The safe habit is the unbiased estimator, and an interviewer who asks for it will usually ask for the direction of the plug-in's bias next.

import numpy as np

def pass_hat_k(n, c, k):
    raise NotImplementedError

def benchmark_pass_hat_k(trials, k):
    raise NotImplementedError

Why tasks that differ keep pass^k high

A tiny case shows what the τ-bench gap is telling us. Two benchmarks each have two tasks and the same mean success rate of 0.5.

Back to τ-bench. If every retail task had the same success probability 0.612, eight trials would all succeed with probability $0.612^8 = 0.020$. The paper reports a $\text{pass}^8$ below 25% but near it, about ten times higher. The gap is not noise. It is information about how difficulty is spread over the tasks, as in benchmark B. That spread is called heterogeneity, and it always pushes the two metrics apart. In words, spread across tasks keeps $\text{pass}^k$ above what the mean alone predicts, and keeps pass@$k$ below it.

Proof. Let $P$ be the success probability of a task chosen uniformly at random, so $\text{pass}^k = \mathbb{E}[P^k]$ and $\bar p = \mathbb{E}[P]$. The function $u^k$ is convex on $[0, 1]$, strictly for $k \ge 2$, so Jensen's inequality gives $\mathbb{E}[P^k] \ge (\mathbb{E}P)^k$, with equality exactly when $P$ is constant. Since $0 \le P \le 1$, $P^k \le P$, so $\mathbb{E}[P^k] \le \mathbb{E}[P]$, with equality exactly when $P \in \lbrace 0, 1\rbrace$. For pass@$k$, $(1-u)^k$ is also convex, so $\mathbb{E}[(1-P)^k] \ge (1 - \mathbb{E}P)^k$, and subtracting from 1 reverses the inequality. $\square$

In plain words, a benchmark whose tasks are each nearly always or nearly never solved keeps its $\text{pass}^k$ near $\bar p$, because the easy tasks keep succeeding. It gains little pass@$k$ from retries, because the hard tasks keep failing. A benchmark of uniformly middling tasks shows the opposite. The single number $\bar p$ cannot tell these apart, and $\text{pass}^k$ at a few values of $k$ can.

To put numbers on this we need a model of how the $p_j$ are spread. The standard choice is the Beta distribution, a family of distributions on $[0, 1]$ with two positive parameters $a$ and $b$ that can be bell-shaped, flat, or U-shaped. Its density is proportional to $p^{a-1}(1-p)^{b-1}$, its mean is $a/(a+b)$, and its variance is $ab/\big((a+b)^2(a+b+1)\big)$ (here $a$ and $b$ are local to this section). The sum $s = a + b$ is a concentration. Large $s$ puts every task near the same $p$, and small $s$ piles the tasks up near 0 and 1, like benchmark B.

The moments of a Beta have a closed form. It uses the Gamma function $\Gamma$, which extends the factorial to real numbers. It satisfies $\Gamma(m+1) = m!$ for whole numbers $m$ and $\Gamma(x+1) = x\,\Gamma(x)$ for every $x \gt 0$. The Beta function $\mathrm{B}(a, b) = \Gamma(a)\Gamma(b)/\Gamma(a+b)$ is the constant that makes the Beta density integrate to one, so

The third step holds because the integral is the normalizing constant of a $\operatorname{Beta}(a+k, b)$ density. The last step applies $\Gamma(x+1) = x\,\Gamma(x)$ $k$ times in the numerator and in the denominator. The same computation with $1 - p \sim \operatorname{Beta}(b, a)$ gives $\text{pass@}k = 1 - \prod_{i=0}^{k-1}\frac{b+i}{a+b+i}$.

The widget below draws pass@$k$ and $\text{pass}^k$ against $k$ for any mean and concentration. The solid curves come from tasks spread as in the histogram beside them, and the dashed curves from tasks that all sit at the mean. Before you slide, keep the mean at 0.612, set the concentration $a + b$ to 100, and predict what happens as you lower it to 0.3. Which way does $\text{pass}^8$ move, which way does pass@$8$ move, and how does the histogram change shape? Then raise the mean to 0.9 and ask whether a 90% agent is reliable at $k = 32$.

The cell computes the same curves with betaln, the logarithm of the Beta function, which never overflows. It prints $\text{pass}^8$ and pass@$8$ at four concentrations and plots both curves for $a + b = 2$.

The printed rows show the trade directly. As the concentration falls from 200 to 0.5, $\text{pass}^8$ climbs from 0.021 to 0.405 while pass@$8$ falls from 0.999 to 0.801. Spread across tasks moves the two metrics in opposite directions around the same $\text{pass}^1$.

The widget also answers the question about a 90% agent. At a mean of 0.9 and concentration 2, $\text{pass}^{32}$ is 0.53, and with every task at 0.9 it is only 0.034. So a 90% agent serves a user who brings the same task 32 times without a failure only about half the time when its tasks are spread, and almost never when they are alike.

Predicting pass^k beyond the trials you ran

The unbiased estimator needs at least $k$ trials per task. Agent trials are expensive, since a long coding task can take many minutes and several dollars of tokens per trial. A team that ran four trials per task may still want to know $\text{pass}^{16}$, because a user will meet the task sixteen times. That needs a model of how $p_j$ is spread across tasks, and the Beta model gives one.

Here is the idea in words. Each task has its own unknown $p_j$, drawn from a Beta. Given $p_j$, the task's success count out of $n$ trials is binomial. Averaging the binomial over the Beta gives the distribution of counts we actually observe, called the beta-binomial,

Fit $a$ and $b$ by maximum likelihood over all tasks, that is, choose the values under which the observed counts are most probable. Then plug them into $\mathrm{B}(a+k, b)/\mathrm{B}(a,b)$ to predict $\text{pass}^k$ for any $k$.

A closed-form fit also exists. It is the method of moments, which picks the parameters whose moments equal the observed ones, and the unbiased $\text{pass}^k$ estimates are exactly estimates of the moments of $p$. Write $m_1 = \widehat{\text{pass}^1}$ and $m_2 = \widehat{\text{pass}^2}$, unbiased estimates of $\mathbb{E}[p]$ and $\mathbb{E}[p^2]$. For a Beta with $s = a + b$, $m_1 = a/s$ and $m_2 = m_1\frac{a+1}{s+1}$. Substituting $a = m_1 s$ into the second gives $m_2(s+1) = m_1(m_1 s + 1)$, so

In plain words, the excess of $\text{pass}^2$ over $(\text{pass}^1)^2$ is the variance of $p$ across tasks, and that variance fixes the concentration. This fit needs only two trials per task.

The cell plants a Beta(1.224, 0.776) population, runs four trials per task, fits by the method of moments, and predicts $\text{pass}^{16}$.

With four trials on each of 115 tasks, the size of τ-retail, the moment fit predicts $\text{pass}^{16} = 0.142$ against a truth of 0.121. With 2,000 tasks it predicts 0.127. The guess that ignores spread, $(\text{pass}^1)^{16}$, says 0.0004, off by a factor of 300. Maximum likelihood uses the whole distribution of counts rather than two moments, and it is the method of Exercise 22.5d.

import numpy as np
from scipy.optimize import minimize
from scipy.special import betaln

def fit_beta_binomial(c, n):
    raise NotImplementedError

def pass_hat_curve(a, b, ks):

One assumption has carried all of this, that the trials of a task are independent. They are only if each trial's randomness is fresh. Seed the simulated user identically for every trial of a task and run the agent at temperature 0, and the trials become near copies of each other. Then $\text{pass}^k$ stays close to $\text{pass}^1$ for every $k$, and the metric measures little. Give each trial its own recorded seed instead. Every trial can then be replayed exactly, and no two trials are the same draw, so reproducibility and independence hold together. Interviewers like to ask how to make an environment reproducible without making its trials identical, and this is the answer.

Cost is part of the result

An agent's score is bought with tokens, tool calls, and time, and agents differ in price by orders of magnitude. The cost of one attempt is roughly the input tokens times their price plus the output tokens times theirs, plus tool and sandbox costs. Output tokens cost more per token than input tokens. The model reads all the input tokens in one parallel pass, but it writes output one token per pass, and §20.1 shows why that makes each output token dearer to serve.

Agents have one more twist. Each step resends the whole transcript so far as input, so the total input tokens of an attempt grow with the square of the number of steps.

Leaderboards that rank by accuracy alone hide this. Kapoor et al. (2024) re-ran the agents at the top of the HumanEval leaderboard, a set of 164 Python programming problems graded by unit tests, next to simple baselines. One baseline retried GPT-4 up to five times at temperature 0 while its code failed the example tests given with the problem. Another did the same while raising the temperature from 0 to 0.5 across retries, which they called warming. Warming was not significantly less accurate than the best agent design. The most elaborate design, a tree search over the model's possible actions (Zhou et al. 2023), cost over 50 times more than warming. Agents of similar accuracy differed in cost by almost two orders of magnitude, and none of the papers reported cost as a headline number.

The fix is to report the two numbers together and to call one agent better than another only if it is at least as good on both. Write $\omega_i$ for agent $i$'s mean dollar cost per task and $\text{acc}_i$ for its success rate.

For example, an agent at 70% for $6 dominates one at 68% for $15. The front is the menu of sensible choices, since any agent off it is beaten on both counts by something on it. Two further facts sharpen the comparison.

The first is that you can mix agents. Run a cheap agent on some tasks and an expensive one on the rest, chosen by a coin flip, and you get any cost and accuracy on the straight line between them. Mixing $0.03 at 55% and $1.50 at 66% half and half, for instance, gives $0.765 at 60.5%. The upper convex hull of a set of points is the boundary you get by stretching a string over them from above, so it is made of straight segments between some of the points.

The proof is linearity of expectation, applied to cost and to success separately. The Holistic Agent Leaderboard, HAL, plots its frontiers this way, as convex hulls that include the origin (Kapoor et al. 2025). A router can do better than any mixture. It sends each task to the cheap or the expensive agent based on the task itself, or escalates only when the cheap agent fails a check, so it uses information a coin flip does not. Kapoor et al.'s escalation baseline is a router of this kind. It starts with a cheap model and moves to more expensive ones when the example tests fail.

The second fact settles most arguments about whether an expensive agent is worth it. When a checker can recognize success, the cheap agent can try again until it succeeds. A verifier here means such a checker, for example unit tests, that is available when the agent is deployed. Say a cheap agent costs 1 cent per try and solves a task half the time. Allowed 3 tries, it solves the task with probability $1 - 0.5^3 = 0.875$. It stops at the first success, so it uses $1 + 0.5 + 0.25 = 1.75$ tries on average, 1.75 cents. The box states the general case.

Proof. The task is solved unless all $k$ attempts fail, which has probability $(1-p_j)^k$. The number of attempts $N = \min(G_j, k)$ is a nonnegative integer, so $\mathbb{E}[N] = \sum_{i \ge 0}\mathbb{P}(N \gt i)$. For $i \lt k$, $N \gt i$ exactly when the first $i$ attempts all fail, which has probability $(1-p_j)^i$. Summing the geometric series gives the formula. $\square$

In plain words, when a checker exists, the honest baseline for an expensive agent is the cheap agent retried at the same spend. The cheap agent usually spends less than its budget, because it stops at the first success. The heterogeneity result above says when retries disappoint. Tasks with $p_j$ near 0 stay unsolved however often you retry, so the expensive agent must beat the average of $1 - (1-p_j)^k$, never $1 - (1-\bar p)^k$.

The cell finds the front of four agents (only the ensemble is dominated) and shows the retried small agent climbing from 42% to 82% at 10 tries while spending under 4 cents. Its gains slow after that, from 82% at 10 tries to 91% at 30, because the tasks it almost never solves stay unsolved.

The widget starts from these five agents, and you can drag any of them to a new cost and accuracy. The budget slider sets the spend per task, and the concentration sets how spread out the small agent's per-task success is, where a low value means many tasks are nearly hopeless for it. Watch whether the dotted curve of the retried small agent passes above the agents you can afford. The no-verifier switch takes retries away, so only single attempts compete.

At real scale the same lessons hold. HAL ran 21,730 rollouts, each one full agent run on a task, of 9 models on 9 benchmarks for about $40,000 (Kapoor et al. 2025). In 21 of 36 pairings of a model, a scaffold, and a benchmark, higher reasoning effort gave equal or lower accuracy. On the web benchmark Online Mind2Web, the SeeAct scaffold with GPT-5 Medium cost $171 in total and the Browser-Use scaffold with Claude Sonnet 4 cost $1,577, nine times more, for a two-point difference in accuracy. Claude models did better with Browser-Use and OpenAI models with SeeAct. The scaffold is part of the entry, and cost must sit beside accuracy for the number to be usable. An interviewer who hears "our agent is 5 points better" will ask at what cost, and against what retry baseline.

import numpy as np

def pareto_front(acc, cost):
    raise NotImplementedError

def budget_matched_accuracy(p, cost_per_try, budget):
    raise NotImplementedError

The agent benchmarks, and how each one breaks

Every widely used agent benchmark combines a source of tasks, an environment, a grader, and a metric, and each has a documented weakness. The status column is as of October 2026, from the benchmarks' own leaderboards and Epoch AI's Benchmarking Hub (Epoch AI 2026).

Two patterns stand out. A benchmark gives way to a successor once top agents reach its human baseline, as OSWorld did to OSWorld 2, or once its flaws come to dominate its error, as SWE-bench Verified did to SWE-Bench Pro. And an entry on a leaderboard is a model in a scaffold, so the Terminal-Bench leader is a pair, not a model. Agents that do machine learning research, the subject of MLE-bench, are treated with research agents.

A different summary of agent capability is the time horizon of the research group METR. It is the length of task, measured by how long a skilled human needs for it, that an agent completes with 50% probability (Kwa et al. 2025). §19.4 derived it as a logistic regression of success on log task length and fit its doubling trend. Its intervals are wide. As of October 2026 the longest measured horizon in METR's Time Horizon 1.1 data, which Epoch's hub also lists, is about 1,045 minutes, or 17 hours, for an early preview of Claude Mythos. Its 95% interval runs from 509 to 3,304 minutes, a factor of six and a half. There are three reasons for the width. Few tasks are long enough to pin the curve where it crosses one half. Tasks come in correlated families. And the estimate extrapolates the fitted curve beyond most of the data. METR itself calls measurements above 16 hours unreliable with its current tasks, so this one is a lower-confidence reading than the rest of the series, which §24.1 reads in full. How horizons stretch for agents that work for days is taken up with long-running agents.

Where agent evaluations go wrong, and what labs do

The last group of failures comes from the agent itself. A capable agent finds shortcuts that the benchmark's designers did not foresee, and each one needs its own defense.

• Looking up the answer. HAL's log analysis caught agents searching Hugging Face for the benchmark instead of solving the task (Kapoor et al. 2025). Block the benchmark's hosts, scan transcripts for its name, and treat a hit as contamination at test time (§22.3).
• Gaming the grader. An agent that can see the tests can delete them or special-case them. ImpossibleBench builds tasks whose specification contradicts their tests, so any pass is a cheat, and found GPT-5 passing 76% of one such SWE-bench variant (Zhong et al. 2025). §18.3 covers the same hacks when they appear during RL training. Hide tests from the agent, make test files read-only, and measure cheating with impossible tasks.
• Being smarter than the task. Claude Opus 4.5 solved a τ²-bench flight-booking task through a loophole in the policy that left the user better off, and "failed" as graded. METR found tasks that told agents to reach a score threshold but graded them on exceeding it, which penalized the agents that followed instructions (both from Anthropic 2026). Read failing transcripts before trusting a failure.
• Unsafe success. HAL found agents completing flight purchases with the wrong credit card, a mistake that would be costly for a real customer. An outcome grader that checks only the booking misses it. Add forbidden-action checks for anything irreversible.
• Untrusted content in the environment. Web pages, files, and tool results can carry instructions, so an agent benchmark is also an attack surface. §19.3 lays out the structure of that problem. Agent evaluations should include injected content and report a separate score for resisting it.
• Knowing it is being tested. Models increasingly recognize evaluation settings, and behavior under test can differ from behavior in deployment (§19.6). Realistic environments and tasks drawn from real use shrink the gap without closing it.

In PyTorch

At scale the estimators run over thousands of tasks at once, and the beta-binomial fit is a two-parameter optimization that autograd handles directly. The block below computes unbiased $\text{pass}^k$ and pass@$k$ for every task at once. It works with lgamma, the logarithm of the Gamma function, so the binomial coefficients never overflow. It then fits the beta-binomial on $(\log a, \log b)$ with L-BFGS, an optimizer that builds an approximation of the curvature from recent gradients, and compares the predicted $\text{pass}^{16}$ with the truth.

Read more

• Yao et al. (2024). τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains. State-based grading with a simulated user, and the paper that introduced $\text{pass}^k$ with its unbiased estimator.
• Kapoor et al. (2024). AI Agents That Matter. Cost-controlled agent evaluation, Pareto frontiers, simple retry baselines that match elaborate agents, and holdout sets for agents.
• Kapoor et al. (2025). Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation. 21,730 rollouts across models, scaffolds, and benchmarks, convex-hull frontiers, and what reading the logs revealed.
• Zhu et al. (2025). Establishing Best Practices for Building Rigorous Agentic Benchmarks. The Agentic Benchmark Checklist and the audit of ten benchmarks, including τ-bench's do-nothing floor.
• Aleithan et al. (2024). SWE-Bench+: Enhanced Coding Benchmark for LLMs. The manual audit of "successful" SWE-bench patches that found leaked solutions and weak tests.
• Jimenez et al. (2023). SWE-bench: Can Language Models Resolve Real-World GitHub Issues?. The repository-level benchmark and its test-based grading, read with the audits above.
• Anthropic (2026). Demystifying evals for AI agents. The practitioner's vocabulary of tasks, trials, and graders, capability and regression evals, and grading bugs found by reading transcripts.
• Xie et al. (2024). OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments. Execution-based evaluation scripts in real virtual machines, and the human baseline later agents reached.

Watch next

▶

▶

▶