How to measure a model honestly. What a benchmark actually measures, how much its number would move on a redraw, whether the model has seen the test, how far to trust a judge, how to score an agent that acts, how to build the evals that decide what you ship, and how millions of human votes become a leaderboard, all derived and coded from scratch.

Chapter 22 · Section 22.1 · 90 min

Why evaluation is hard

This chapter is about measuring a model honestly. The book so far has built models, trained them, aligned them, and made them fast, and every step was judged by a number. Fine-tuned models were scored on held-out tasks with error bars (§14.7), reasoning models by pass@k (§18.2), and safety by refusal and attack rates (§19.4). Those numbers now decide what labs train and ship, and they fail in ways no amount of compute fixes. The chapter treats an evaluation as a measurement with its own errors. It builds the statistics, the leak detectors, the model judges, the agent harnesses, and the human arenas that keep the measurement honest, using the confidence intervals and paired tests of §0.11, recapped where they are needed.

This section is about why a benchmark score can be wrong, and how to tell the kinds of error apart. A benchmark is a fixed set of test questions, called items, with a rule for grading the answers. Its score is usually the share of items a model gets right. Labs pick checkpoints, ship models, and decide which ideas to scale on gaps of a point or two in such scores, so knowing what a score can and cannot tell you is part of the job.

The one idea to carry away is an asymmetry. More test items shrink only the random noise in a score. Every other kind of error is a bias, and a bias survives any number of items.

Two measurements from the last two years show the problem. The first is a leaderboard, a public ranking of models by their scores on one benchmark. GPQA Diamond is a set of 198 graduate-level multiple-choice science questions with four options each. In October 2026 its top five looked like this, in independent runs by Epoch AI (2026).

model	GPQA Diamond accuracy
Gemini 4 Argon (high)	96.4%
GPT-6 Astra (max)	95.8%
Claude Sonnet 5.5 (max)	95.6%
GPT-6.1 Sol (max)	95.4%
Gemini 3.8 Flash (high)	95.4%

The five models are 1.0 point apart, and on 198 questions one point is about two questions. Which one is best at graduate-level science? By the end of this section you will be able to show that the benchmark cannot say.

The second is a randomized trial. In the first half of 2025 the research group METR had 16 experienced open-source developers work on 246 real tasks in large repositories they knew well (Becker et al. 2025). Each task was randomly assigned to allow or forbid AI tools. Before starting, the developers forecast that AI would cut their time per task by 24%. Afterward they believed it had cut it by 20%. Measured, tasks with AI took 19% longer.

Over the same months, scores on SWE-bench Verified rose steeply. That benchmark has 500 real GitHub issues, and a fix counts as correct when hidden unit tests pass. In Epoch AI's runs, which use 484 of the issues, GPT-4o (November 2024) solved 31.0%, Claude 3.7 Sonnet (February 2025) 61.0%, and Claude Opus 4 (May 2025) 70.7%.

Both stories are failures of measurement, of different kinds. In the first, a number is too noisy, and too close to the best score the test allows, to rank the models. In the second, a number rose fast and did not track the thing people cared about. This section answers five questions, in order. What does an evaluation estimate, and how does its error split into parts? What does it mean for a benchmark to measure the right thing? How much does reporting the best of many tries inflate a score, and how far can an adversary push a public leaderboard? Why do benchmarks stop improving below 100%? And how large is the measured gap between benchmark scores and real work?

Five ways a score can be wrong

A thermometer reads a temperature. A thermostat reads it and then acts to change it. Benchmarks are built to be read like thermometers and are used like thermostats. Labs choose training mixtures, checkpoints, and prompts by what raises the number. Researchers decide which ideas deserve attention by it, and buyers choose models by it. Each of those uses pushes on the number, and a number that is pushed on drifts away from the thing it was built to read.

This drift is Goodhart's law, named for the economist Charles Goodhart. In the short form of the anthropologist Marilyn Strathern, when a measure becomes a target, it stops being a good measure (Strathern 1997). §16.5 met it as reward hacking, where a policy trained against a learned reward model found answers that the reward model scored highly and humans disliked. On a public benchmark the optimizer is a whole research community, and the reward model is the test set.

The pressure acts through five ways a reported score can be wrong. To see all five at once, picture a 20-question quiz meant to tell whether a new hire can do a job. The quiz covers a syllabus that is not quite the job. Three of its questions leaked from last year's exam. A teacher who likes long answers grades it. And it is taken on a calculator that sometimes freezes. In the table, $n$ is the number of items in a benchmark and $K$ is the number of answers sampled from the model on each item.

error term	in the quiz	in a language model evaluation	where the chapter treats it
validity	the syllabus is not the job	the benchmark's items, grading, or protocol differ from real use	this section, and §22.6
sampling	20 questions is a small sample of the syllabus	finite items $n$ and samples $K$ , prompt and seed variation	§22.2
leakage	three questions were seen before	test items or their answers were in the training data	§22.3
grader	the teacher rewards length	an answer key with errors, or a biased model judge	§22.4 and §22.7
environment	the calculator freezes	flaky sandboxes, broken test harnesses, scaffold choices	§22.5

Only one row, sampling, shrinks when you add questions. A 2,000-question quiz on the wrong syllabus, graded by the same teacher, still measures the wrong thing. It measures it more precisely, and that is all. The next part makes this exact.

An evaluation is an estimator

An estimator is a rule that turns random data into a guess at a fixed unknown number. The fraction of heads in 100 coin flips is an estimator of the coin's chance of heads. An evaluation is the same kind of object. The unknown number is how good the model is at the job you care about. The data are the model's answers to a sample of test items, scored by a grader.

Here is a tiny case with numbers. A model will answer customer-support tickets, and on real tickets it gives a good answer 60% of the time. You cannot measure that directly, so you run a benchmark of 10 FAQ-style questions, and the grader marks 8 of them right. The estimate is 0.8 and the target is 0.6. Part of the 20-point gap is luck, since 10 questions is very few. Part of it is that FAQ questions are easier than real tickets. Only the first part would shrink if you asked 1,000 FAQ questions instead of 10.

To make that split precise we need names for the pieces. The protocol is everything between an item and its score. That means the prompt template and any example answers shown in it, the decoding settings, any tools or agent scaffold, the rule that extracts the answer, and the rule that averages the scores. The grader is whatever scores a response, such as exact match against an answer key, unit tests, a model judge, or a human.

📐 An evaluation as an estimator

Fix a model and a protocol. Let $\mathcal{D}_{\text{use}}$ be the distribution of inputs $x$ the model will meet in real use, and let $g^\star(x, y) \in [0, 1]$ be the true quality of a response $y$ to $x$ . The target is the expected true quality under use,

\mu^\star = \mathbb{E}_{x \sim \mathcal{D}_{\text{use}}}\,\mathbb{E}_{y \sim \text{model}(\cdot \mid x)}\big[g^\star(x, y)\big],

with responses produced the way they will be produced in deployment. An evaluation draws $n$ items $x_1, \dots, x_n$ from a benchmark distribution $\mathcal{D}_{\text{bench}}$ , samples $K$ responses $y_{i1}, \dots, y_{iK}$ to each under its protocol, scores each response with a grader $g$ , and reports the estimate

\hat\mu = \frac{1}{n}\sum_{i=1}^{n} \frac{1}{K}\sum_{k=1}^{K} g(x_i, y_{ik}).

In plain words, the number you want is how good the model is where it will be used, judged by what quality really means. The number you compute is an average of a grader's verdicts on a sample of benchmark items. Every way a benchmark result misleads is a way these two numbers differ.

To split the difference we need one number in between. Let $\mu_{\text{bench}} = \mathbb{E}_{x \sim \mathcal{D}_{\text{bench}}}\mathbb{E}_{y}\big[g^\star(x, y)\big]$ be the true quality on fresh benchmark items the model has never seen, with the protocol run exactly as intended and graded by true quality. In the support example it is the model's real success rate on FAQ questions, whatever the grader says. Write $\mathbb{E}[\hat\mu]$ for the average of the estimate over everything random in it. That covers the draw of items, the sampled responses, and any randomness in the grader and the environment.

∴ The three-way decomposition of evaluation error

For any evaluation,

\hat\mu - \mu^\star \;=\; \underbrace{(\mu_{\text{bench}} - \mu^\star)}_{\text{validity bias } b_{\text{val}}} \;+\; \underbrace{(\mathbb{E}[\hat\mu] - \mu_{\text{bench}})}_{\text{measurement bias } b_{\text{meas}}} \;+\; \underbrace{(\hat\mu - \mathbb{E}[\hat\mu])}_{\text{sampling noise}},

and the mean squared error is

\mathbb{E}\big[(\hat\mu - \mu^\star)^2\big] = (b_{\text{val}} + b_{\text{meas}})^2 + \operatorname{Var}(\hat\mu).

The validity bias compares the benchmark and its protocol with real use. The measurement bias collects leakage (the model behaves differently on items it has seen), grader error ( $g \ne g^\star$ ), and environment error (the protocol was not executed as intended). The sampling noise has mean zero. For $n$ fresh items with $K$ samples each, its variance is $(\sigma_b^2 + \sigma_w^2/K)/n$ .

Why it holds. The first identity is exact, because the middle terms cancel when you add the three brackets. For the mean squared error, write $b = b_{\text{val}} + b_{\text{meas}}$ , a constant, and $e = \hat\mu - \mathbb{E}[\hat\mu]$ , which has mean zero. Then $\mathbb{E}[(b + e)^2] = b^2 + 2b\,\mathbb{E}[e] + \mathbb{E}[e^2] = b^2 + \operatorname{Var}(\hat\mu)$ .

The variance formula comes from §14.7, which derived the variance of an average over items and samples. Each item $i$ has its own true pass rate $p_i$ , the chance that one sampled answer to it passes. The between-item variance $\sigma_b^2$ is how much the $p_i$ differ from item to item. The within-item variance $\sigma_w^2$ is the average spread of one graded sample around its own item's pass rate. For a pass or fail grade it is the average of $p_i(1 - p_i)$ . More samples per item shrink only the second term, while more items shrink both. The square root of $\operatorname{Var}(\hat\mu)$ is the standard error, the typical distance between the score and its own average over redraws (§0.11).

In plain words, the squared error of a reported score is a squared bias, which no amount of data removes, plus a variance that falls like $1/n$ . Setting the two equal gives the crossover size,

n^\ast = \frac{\sigma_b^2 + \sigma_w^2/K}{b^2}.

Past $n^\ast$ items the bias is most of the error, and more items buy almost nothing. Jacobs and Wallach (2021) make the same split in the language of measurement theory, where reliability roughly means low variance and validity roughly means low bias.

✏️ An error budget by hand

A benchmark has $n = 500$ items with one sample each, and the model's true accuracy is about $p = 0.7$ . For one pass or fail sample per item, $\sigma_b^2 + \sigma_w^2 = p(1-p) = 0.21$ , so the standard error is $\sqrt{0.21/500} = 2.05$ points. Suppose the grader is lenient by 2 points. The root mean squared error is $\sqrt{2^2 + 2.05^2} = 2.86$ points, and bias and sampling contribute about equally.

Grow the benchmark tenfold to $n = 5{,}000$ . The standard error falls to 0.65 points, but the root mean squared error only falls to $\sqrt{4 + 0.42} = 2.10$ points, and 90% of the squared error is now the grader's bias. The crossover is $n^\ast = 0.21/0.02^2 = 525$ items. Past a few hundred items, fix the grader before you buy more items.

A small simulation shows all five terms at once. A model handles three kinds of request, lookups, coding, and long analyses, with true pass rates 0.9, 0.7, and 0.4. Users mostly ask for long analyses. The benchmark was built from what was easy to collect, which is mostly lookups. On top of that, 15% of the items leaked into training, the grader passes 20% of wrong answers, and 3% of runs crash and score zero.

Python · cell 1

Two lessons follow. A bigger benchmark makes a wrong number more precise, not more right. And biases can cancel, so a score that passes a sanity check can still be wrong in three ways at once. In an interview, when someone proposes more eval data as the fix for a misleading number, the right first question is which term dominates the error.

The widget uses the model of the worked example, with $p = 0.7$ and one sample per item. Drag the end of each bias bar to set that bias in points, and grow the benchmark with the slider or with Play. The plot below the bars shows the sampling error falling with $n$ against the flat dashed line of the total bias.

What construct validity asks

Every benchmark has a name, and the name makes a claim. GPQA claims to test graduate-level scientific reasoning, and HumanEval, a set of 164 short Python function problems with unit tests, claims to test coding. Whether the scores back the name is the question of validity. It is the hardest of the five terms to measure, because it compares the benchmark with real use, which you cannot sample directly.

Psychology and education faced this problem a century before machine learning. They measure constructs, properties you cannot observe directly, such as reading ability or anxiety. They do it through an operationalization, a concrete procedure you can run, such as a reading test. A bathroom scale is a good operationalization of weight and a poor one of fitness. The same reading supports one claim and not the other. The theory of when a measurement can be trusted is called measurement theory, and Jacobs and Wallach (2021) brought it to computer scientists.

📐 Construct validity

A construct is the unobservable property a measurement claims to capture, such as "coding ability" or "graduate-level scientific reasoning". Construct validity is the degree to which a measurement's scores support the claims made about that construct. It is never a yes or no property. It is argued from several kinds of evidence, and Jacobs and Wallach list seven.

Face validity. The scores look plausible on inspection. This is a sniff test, necessary and never sufficient.
Content validity. The items cover the whole construct, and nothing else.
Convergent validity. The scores correlate with other, already validated measures of the same construct.
Discriminant validity. The scores do not correlate with measures of different constructs any more than those constructs are themselves related.
Predictive validity. The scores agree with an accepted outside measure of the construct taken later, as a hiring test should predict later performance in the job.
Hypothesis validity. The scores support known hypotheses about the construct, for example that experts score higher than novices.
Consequential validity. The effects of using the scores, and of the decisions they drive, are acceptable.

Psychometrics calls agreement with an accepted outside measure criterion validity. It is predictive when the outside measure comes later and concurrent when it is taken at the same time.

In plain words, the question is never "is this benchmark valid?" but "does this score justify this particular claim?". The same 198 questions can support "the model answers these multiple-choice science questions well" and fail to support "the model reasons like a PhD scientist".

The field's most famous benchmarks have been read as more than they are. ImageNet and GLUE were treated as measures of general visual and general language understanding, while each is a particular collection of items, labels, and formats (Raji et al. 2021). A later review had 29 expert reviewers examine 445 language model benchmarks from the main machine learning and language conferences (Bean et al. 2025). Of the benchmarks that defined the phenomenon they measure, 47.8% used a contested definition. Across all 445, 12.3% drew their items only from datasets that happened to be easy to get, a practice called convenience sampling, and another 27.0% did so in part. Only 16.0% used uncertainty estimates or statistical tests to compare models.

The review closes with eight recommendations that read like an outline of this chapter, from defining the phenomenon and sampling items to represent it, to preparing for contamination and comparing models with statistics.

✏️ Testing a claim about GPQA Diamond

The claim is "Model X scores 96% on GPQA Diamond, so it has PhD-level scientific reasoning." Take the kinds of validity one at a time.

Content. Domain experts in biology, physics, and chemistry wrote GPQA's questions. Diamond keeps only those that both expert validators answered correctly (or that the second validator missed and then clearly explained the mistake) and that at most one of three skilled non-experts with web access answered correctly (Rein et al. 2023). That is a narrow slice of science, answering one self-contained question with four options. It leaves out posing questions, designing experiments, long derivations, and noticing that a question is ill-posed.
Format. Four options give a guessing floor of 25%, and they allow elimination strategies that need no derivation.
Criterion. No published study links GPQA accuracy to a model's usefulness to working scientists. The predictive claim has no evidence for or against it.
Convergent and discriminant. GPQA scores climb together with scores on many other hard benchmarks as models improve, which is convergent evidence for something. Nobody has shown discriminant evidence that the something is reasoning rather than knowledge or overall scale.
Reliability. With $n = 198$ the standard error at 96.4% is 1.3 points, as computed later in this section.

The claim the data support is narrower and still impressive. Under this protocol the model answers about 96% of these 198 expert-written questions correctly, give or take 2.6 points (two standard errors).

An interviewer who asks "is benchmark X a good measure of Y?" wants this kind of answer. Name the claim, then go through the kinds of evidence for and against it.

Checking that a benchmark measures something other than capability

Discriminant validity has a famous failure in safety research. To see it we need one number for a model's general capability. It comes from a striking regularity. Collect the scores of many models on many benchmarks, and one direction accounts for most of the differences between models. For 77 open base models from 21 families, the first principal component of standard benchmark scores explains nearly 80% of the variance, and the top three explain about 97% (Ruan et al. 2024).

A quick recap of the tool. Principal component analysis (PCA) finds the direction in which a cloud of points is most spread out. When the data are standardized, that direction is the top eigenvector of the correlation matrix. Each point's score is its coordinate along that direction, the dot product of its standardized row with the eigenvector. §1.6 used PCA to look inside embeddings, and §0.10 derived it from the spectral theorem.

✏️ A capabilities score from two benchmarks, by hand

Two benchmarks have correlation $r = 0.8$ across models. Their correlation matrix is $\begin{pmatrix} 1 & 0.8 \\ 0.8 & 1 \end{pmatrix}$ . Its eigenvalues are $1 + r = 1.8$ and $1 - r = 0.2$ , with eigenvectors $(1, 1)/\sqrt 2$ and $(1, -1)/\sqrt 2$ . The first component is the plain average direction, and it explains $1.8/2 = 90\%$ of the variance. A model one standard deviation above average on both benchmarks has a capabilities score of $(1 + 1)/\sqrt 2 = 1.41$ . A model above average on one and equally far below on the other scores 0. With more benchmarks the arithmetic is the same, done by an eigensolver.

📐 Capabilities score

Let $\mathbf{S} \in \R^{N \times J}$ hold the scores of $N$ models on $J$ benchmarks. Standardize each column across models, $Z_{ij} = (S_{ij} - \bar S_j)/\sigma_j$ , where $\bar S_j$ and $\sigma_j$ are the mean and the standard deviation (divisor $N$ ) of column $j$ . The matrix $\mathbf{C} = \mathbf{Z}^\top\mathbf{Z}/N \in \R^{J \times J}$ is then the correlation matrix of the benchmarks. Let $\lambda_1 \ge \dots \ge \lambda_J$ be its eigenvalues and $\mathbf{v}_1$ a unit eigenvector for $\lambda_1$ , with its sign chosen so that its entries sum to a positive number. The capabilities score of model $i$ is its first principal component score $(\mathbf{Z}\mathbf{v}_1)_i$ . The variance share of the first component is $\lambda_1/\sum_j \lambda_j = \lambda_1/J$ , since the trace of a correlation matrix is $J$ . A benchmark's capabilities correlation is the correlation, across models, between its scores and a capabilities score computed from benchmarks that do not include it.

In plain words, put every benchmark on the same scale, find the one direction along which the models differ most, and call a model's position along it its capability. Standardizing matters because benchmarks are reported in different units and spreads. Without it, the benchmark with the widest spread would decide the direction. The variance share says how much of the differences between models that one direction explains. A large share is convergent evidence that the benchmarks measure something in common, though not that the something is what any of their names claim.

Ren et al. (2024) applied this to safety. They built the capabilities score from standard capability benchmarks, among them MMLU (multiple-choice academic knowledge), MATH and GSM8K (math word problems), and BBH (hard reasoning puzzles). The first component explained 78.3% of the variance for 27 base models and 70.8% for 26 chat models. They then correlated each safety benchmark with it, with every safety score oriented so that higher means safer.

Many safety benchmarks tracked capability closely. For chat models, TruthfulQA MC1, a multiple-choice test of whether a model repeats common misconceptions, had a capabilities correlation of 81.2%. The ETHICS benchmark of moral knowledge had 82.2%. A larger model scores higher on them, so an improvement on such a benchmark cannot be credited to safety work.

Others moved the other way. WMDP is a multiple-choice test of hazardous knowledge in biosecurity, chemistry, and cybersecurity, scored so that higher means less of that knowledge. Its three subsets had correlations of $-81\%$ to $-88\%$ for the same chat models, plausibly because more capable models know more of everything, including what is dangerous. Ren et al. called the resulting confusion safetywashing, capability gains presented as safety progress. A benchmark meant to measure something distinct from general capability has to show that it is distinct, and the capabilities correlation is the test.

Here is the construction on simulated data. Forty models have a hidden capability. Ten benchmarks track it with different strengths and are reported on different scales, the last on a very wide one. A "safety" benchmark has a loading $\beta$ on capability, the correlation it would have with the true capability, and we vary it.

Python · cell 2

The first component explains 65.2% of the variance and correlates 0.969 with the true capability, so it recovers the hidden variable well from ten noisy benchmarks. Without standardizing, the one benchmark on a wide scale takes 90% of the direction's squared weight instead of 13%. The "capabilities score" then becomes little more than that one benchmark.

A safety benchmark with loading $\beta$ has a correlation with the score of about $0.97\beta$ , its loading times the score's own correlation with capability, plus a little chance correlation. A loading of 0.9 shows up as 0.88, and a loading of 0 shows up as 0.17. That last number is pure chance. With 40 models, a correlation whose true value is zero has a standard deviation of about $1/\sqrt{N-1} = 0.16$ . So read a capabilities correlation against an interval, because small differences between benchmarks mean nothing.

Core

The capabilities correlation of a benchmark

Implement capability_correlation(S) for a score matrix S of shape (N, J), with one row per model and one column per benchmark. It returns a tuple (scores, share, r).

scores is the length- $N$ capabilities score of the definition. Use all $J$ benchmarks, standardize each with the divisor $N$ (S.std(0)), and orient $\mathbf{v}_1$ so that its entries sum to a positive number.
share is the variance share $\lambda_1/\sum_j\lambda_j$ .
r is a length- $J$ array. Entry $j$ is the Pearson correlation between column $j$ and the capabilities score computed from the other $J - 1$ columns, oriented the same way. That way no benchmark is compared with a score it helped build.

Ren et al. used Spearman's rank correlation, which is the Pearson correlation of the ranks. Pearson keeps the exercise short, and swapping in ranks changes one line.

import numpy as np

def capability_correlation(S):
    raise NotImplementedError

Tests, hints, verified solutions, and AI help, with Pro at $7.99 a month.

🔬 When the model knows it is being tested

One more validity threat grows with capability. A model that can tell when it is being tested can behave differently in tests. Lynch et al. (2025) placed Claude Opus 4 in fictional corporate scenarios where blackmailing an employee served its goal. When its reasoning stated that the scenario was real, it chose blackmail 55.1% of the time. When its reasoning stated that it was in an evaluation, it did so 6.5% of the time. §19.6 covers these results and the related behavior of deliberately underperforming on tests.

The capacity behind this has been measured directly. Needham et al. (2025) built 1,000 transcripts from 61 datasets, some from evaluations and some from real deployment, and asked models which was which. They scored the answers by the area under the ROC curve (AUC, §0.11). Here it is the chance that a random evaluation transcript is rated more likely to be an evaluation than a random deployment transcript, and chance level is 0.5. Gemini 2.5 Pro reached 0.83, and the authors themselves, who knew the datasets, reached 0.92. Frontier models did better on agentic transcripts, 0.88 to 0.95, than on chat transcripts, 0.71 to 0.77, and so did the authors, 0.996 against 0.81.

When the protocol itself changes the behavior, $\mu_{\text{bench}}$ describes a different model from the one users meet. The defense is to make the protocol look like use, with realistic tasks and environments and, wherever possible, items and transcripts drawn from deployment itself.

Goodhart's law on a shared test set

Goodhart's law comes in several forms, and two of them matter here (§16.5, §19.1). The regressional form says that when you pick whatever scores highest on a noisy measure, you also pick whatever had the luckiest noise. The extremal form says that the measure and the truth part ways exactly where you push hardest.

A public benchmark is a measure that a whole community pushes on. A submission is one model, checkpoint, prompt, or seed scored on it. Every paper that reports the best of its runs, and every lab that keeps the highest-scoring submission, is selecting on the benchmark's noise. §0.11 called the many quiet choices behind such selection the garden of forking paths. Here we measure how much it inflates a leaderboard. Start with the smallest case, by hand.

✏️ Two tries are already biased

Two submissions have the same true accuracy $p = 0.7$ and are scored on the same $n = 1{,}000$ items, with independent noise. Each observed score is the truth plus a normal error with standard error $\mathrm{SE} = \sqrt{0.7 \cdot 0.3/1000} = 1.45$ points. You report the better of the two.

For standard normals $Z_1, Z_2$ , $\max(Z_1, Z_2) = \tfrac12(Z_1 + Z_2) + \tfrac12|Z_1 - Z_2|$ . The first term has mean zero. The difference $Z_1 - Z_2$ is normal with variance 2. The mean of $|X|$ for a normal $X$ with mean zero and standard deviation $s$ is $s\sqrt{2/\pi}$ , so $\mathbb{E}|Z_1 - Z_2| = \sqrt 2 \cdot \sqrt{2/\pi} = 2/\sqrt\pi$ . Hence $\mathbb{E}[\max(Z_1, Z_2)] = 1/\sqrt\pi = 0.564$ .

The reported score overstates the truth by $0.564 \times 1.45 = 0.82$ points on average, from nothing but trying twice.

Now the general case. Take $m$ submissions that all have the same true accuracy $p$ on the benchmark's distribution, scored on the same $n$ items. By the normal approximation to the binomial, each observed score is approximately $\hat S_j \approx p + \mathrm{SE}\cdot Z_j$ , with $\mathrm{SE} = \sqrt{p(1-p)/n}$ and $Z_j$ standard normal. The leaderboard reports $\max_j \hat S_j$ , so the expected reported score is

\mathbb{E}\Big[\max_{j \le m} \hat S_j\Big] = p + \mathrm{SE}\cdot\mathbb{E}\Big[\max_{j \le m} Z_j\Big].

The submissions are equally good, so all of the excess is selection. §16.1 computed the expected maximum of $m$ independent standard normals as an integral against the density of the maximum, $\mathbb{E}[\max_j Z_j] = \int z\,m\,\varphi(z)\,\Phi(z)^{m-1}\,dz$ . Here $\varphi$ and $\Phi$ are the standard normal density and cumulative distribution function. What we need now is how it grows with $m$ , and what correlation between submissions does to it.

∴ Selection bias of the best of m

Let $Z_1, \dots, Z_m$ be standard normal, with any joint distribution. Then

\mathbb{E}\Big[\max_{j \le m} Z_j\Big] \;\le\; \sqrt{2\ln m}.

If the $Z_j$ are independent, the bound is tight to first order, $\mathbb{E}[\max_j Z_j]/\sqrt{2\ln m} \to 1$ as $m \to \infty$ . If they are equicorrelated, $Z_j = \sqrt{\rho}\,U + \sqrt{1-\rho}\,V_j$ with $U, V_1, \dots, V_m$ independent standard normals and $0 \le \rho \le 1$ , then $\mathbb{E}[\max_j Z_j] = \sqrt{1-\rho}\;\mathbb{E}[\max_j V_j]$ . So the best of $m$ equally good submissions overstates their accuracy, in expectation, by

\mathrm{SE}\cdot\sqrt{1-\rho}\;\mathbb{E}\Big[\max_{j\le m} V_j\Big] \;\le\; \sqrt{\frac{p(1-p)}{n}}\,\sqrt{1-\rho}\,\sqrt{2\ln m}.

∂ Proof of the bound, its tightness, and the correlated case

The bound. Fix $t > 0$ . The exponential is convex, so Jensen's inequality gives $e^{t\,\mathbb{E}[\max_j Z_j]} \le \mathbb{E}\big[e^{t \max_j Z_j}\big]$ . The exponential is also increasing, so $e^{t\max_j Z_j} = \max_j e^{tZ_j} \le \sum_j e^{tZ_j}$ , since a maximum of positive numbers is at most their sum. The moment generating function of a standard normal is $\mathbb{E}[e^{tZ}] = e^{t^2/2}$ . Together,

e^{t\,\mathbb{E}[\max_j Z_j]} \le \sum_{j=1}^m \mathbb{E}\big[e^{tZ_j}\big] = m\,e^{t^2/2} \quad\Longrightarrow\quad \mathbb{E}\Big[\max_j Z_j\Big] \le \frac{\ln m}{t} + \frac{t}{2}.

The right side is smallest where its derivative $-\ln m/t^2 + 1/2$ vanishes, at $t = \sqrt{2\ln m}$ , where it equals $\sqrt{\ln m/2} + \sqrt{\ln m/2} = \sqrt{2\ln m}$ . Nothing used independence, only that each $Z_j$ is standard normal.

Tightness for independent normals. Fix $\delta \in (0, 1)$ and let $t_m = (1-\delta)\sqrt{2\ln m}$ . Write $A$ for the event $\max_j Z_j \le t_m$ . By independence and $1 - x \le e^{-x}$ ,

\mathbb{P}(A) = \Phi(t_m)^m = \big(1 - \bar\Phi(t_m)\big)^m \le \exp\!\big(-m\,\bar\Phi(t_m)\big),

with $\bar\Phi = 1 - \Phi$ . The standard Gaussian tail bound $\bar\Phi(t) \ge \frac{t}{1+t^2}\varphi(t)$ gives $m\,\bar\Phi(t_m) \ge \frac{t_m}{1+t_m^2}\cdot\frac{m^{1-(1-\delta)^2}}{\sqrt{2\pi}}$ , since $m\,\varphi(t_m) = m\,e^{-(1-\delta)^2\ln m}/\sqrt{2\pi}$ . The factor $m^{1-(1-\delta)^2}$ grows like a power of $m$ , while $t_m/(1+t_m^2)$ shrinks only like $1/\sqrt{\ln m}$ . So $m\,\bar\Phi(t_m) \to \infty$ and $\mathbb{P}(A) \to 0$ .

Now split the expectation on $A$ . Off $A$ the maximum exceeds $t_m$ . On $A$ it is at least $Z_1 \ge -|Z_1|$ , and by Cauchy-Schwarz $\mathbb{E}[|Z_1|\mathbb{1}_A] \le \sqrt{\mathbb{E}[Z_1^2]\,\mathbb{P}(A)} = \sqrt{\mathbb{P}(A)}$ . Hence

\mathbb{E}\Big[\max_j Z_j\Big] \ge t_m\big(1 - \mathbb{P}(A)\big) - \sqrt{\mathbb{P}(A)}.

Dividing by $\sqrt{2\ln m}$ and letting $m \to \infty$ gives a limit inferior of at least $1 - \delta$ . With the upper bound and $\delta$ arbitrary, the ratio tends to 1.

The correlated case. $\max_j\big(\sqrt\rho\,U + \sqrt{1-\rho}\,V_j\big) = \sqrt\rho\,U + \sqrt{1-\rho}\,\max_j V_j$ , because the shared term is the same for every $j$ . Taking expectations and using $\mathbb{E}[U] = 0$ gives $\mathbb{E}[\max_j Z_j] = \sqrt{1-\rho}\,\mathbb{E}[\max_j V_j]$ . Each $Z_j$ is standard normal, and any two have correlation $\rho$ .

In plain words, reporting the best of $m$ runs adds about $\sqrt{2\ln m}$ standard errors of pure luck. That amount grows slowly and never stops growing. Correlated submissions add less, because they share their luck. The bound is loose at practical sizes. For $m = 100$ it says 3.03 standard errors where the truth is 2.51. In the decomposition of the previous part, this is a measurement bias created by the reporting rule. Each submission's own score is unbiased for $p$ , yet the reported maximum is not.

What is $\rho$ in practice? It is the correlation between the scoring errors of two submissions, and it comes from items they find hard or easy together. Suppose every submission has the same per-item pass rates $p_i$ , with between-item variance $\sigma_b^2$ , and their outcomes on an item are independent given $p_i$ . Over a fresh draw of the $n$ items, two scores then have covariance $\sigma_b^2/n$ and variance $p(1-p)/n$ each, so $\rho = \sigma_b^2/\big(p(1-p)\big)$ . Variants of one model, such as seeds, prompts, and nearby checkpoints, also make the same mistakes on the same items, which pushes $\rho$ toward 1. Mania et al. (2019) measured this similarity among ImageNet models and showed that it provably limits the overfitting that comes from reusing a test set.

The cell simulates 400 leaderboards of 1,000 equally good submissions. It compares the average best score with the exact integral and with the bound.

Python · cell 3

On many leaderboards a 4-point gain is a headline, and here it comes from nothing but a thousand tries. This is why a careful team reports how many runs preceded a number. It is also why an interviewer who hears "we tried a few prompts and took the best" will ask how much of the gain is selection.

Core

The selection bias of a leaderboard

Implement two functions.

best_of_m_inflation(m, n, p, rho=0.0) returns the inflation, the expected amount by which the best of $m$ equally good submissions overstates their common accuracy $p$ . All of them are scored on the same $n$ items, and their scoring errors have correlation rho.
debiased_best(observed_best, m, n, rho=0.0) returns the accuracy $p$ for which $p$ plus its inflation equals observed_best.

The setup provides expected_max_normal(m), which computes $\mathbb{E}[\max_{j \le m} Z_j]$ by the integral of §16.1.

import numpy as np
from scipy import optimize

def best_of_m_inflation(m, n, p, rho=0.0):
    raise NotImplementedError

def debiased_best(observed_best, m, n, rho=0.0):
    raise NotImplementedError

Tests, hints, verified solutions, and AI help, with Pro at $7.99 a month.

The widget runs the same experiment with $p = 0.7$ and $n = 1{,}000$ . Each dot is one submission's public score, the board reports the best, and the board's winner is also scored on 10,000 private items it was never selected on. Press Play to add submissions one at a time. The plot below the dots follows the board, in points above the truth, along the dashed curve of the expected inflation, while the winner's private score stays near zero.

How large is this effect at real scale? Smaller than the arithmetic suggests, for reasons the arithmetic also explains. Kaggle is a platform for machine learning competitions. Each competition scores submissions on a public split of the test set while it runs, and on a private split at the end, so the gap between the two directly measures overfitting to the public split. Across 120 Kaggle competitions, Roelofs et al. (2019) found little evidence of substantial overfitting.

Recht et al. (2019) rebuilt the test sets of CIFAR-10 and ImageNet by following the original collection procedures. Accuracy dropped by 3 to 15 points on CIFAR-10 and 11 to 14 points on ImageNet. Yet gains on the old test sets carried over to larger gains on the new ones, so the ranking of models largely held. They attributed the drops to slightly harder images, a shift between $\mathcal{D}_{\text{bench}}$ and the new test distribution, rather than to repeated reuse of the old test set.

Two things keep real overfitting small. Submissions are highly correlated, so $\sqrt{1-\rho}$ is small. And most participants are not adversaries. The second assumption is the one to worry about, because the next result shows how far an adversary can go.

How far an adversary can push a leaderboard

Blum and Hardt (2015) asked what an analyst can do with nothing but a leaderboard that reports accurate scores. The answer is unsettling. Knowing nothing about the labels, the analyst can climb the board steadily. The recipe has three steps. Submit random guesses. Keep the ones the board scored at one half or above, since each of those tends to agree with the hidden labels a little more than chance. Then submit their majority vote, which adds up those small agreements. We write $k$ for the number of these probing submissions, following the paper.

∴ The boosting attack

Let the hidden public labels be $\mathbf{y} \in \{0, 1\}^n$ , and let the leaderboard report the public accuracy of any submitted $\mathbf{u} \in \{0, 1\}^n$ to within $n^{-1/2}$ . The boosting attack draws $k$ independent uniformly random vectors $\mathbf{u}_1, \dots, \mathbf{u}_k$ , submits each, keeps those whose reported accuracy is at least $\tfrac12$ , and submits their coordinatewise majority $\mathbf{u}^\ast$ . With probability at least $2/3$ ,

\text{public accuracy of } \mathbf{u}^\ast \;\ge\; \frac12 + \Omega\!\left(\sqrt{\frac{k}{n}}\right),

while on any labels the leaderboard did not use, $\mathbf{u}^\ast$ is a uniformly random guess (Blum and Hardt 2015, Theorem 5.1).

∂ Why the attack works, and its size

Here is a heuristic calculation of the gain, with normal approximations, which the simulation below confirms. A random vector agrees with $\mathbf{y}$ on each coordinate with probability $\tfrac12$ , so its public accuracy is $a_j \approx \tfrac12 + Z_j/(2\sqrt n)$ with $Z_j$ standard normal. The attack keeps the vectors with $Z_j > 0$ , about $s = k/2$ of them, and $\mathbb{E}[Z \mid Z > 0] = \sqrt{2/\pi}$ . So a kept vector agrees with $\mathbf{y}$ on an average coordinate with probability

\tfrac12 + \Delta, \qquad \Delta = \frac{\sqrt{2/\pi}}{2\sqrt n} = \frac{1}{\sqrt{2\pi n}} .

On one coordinate, the $s$ kept vectors are close to independent votes, each right with probability $\tfrac12 + \Delta$ . They are close to independent because each vector's selection depends on that coordinate through only one of $n$ terms. The count of right votes is approximately normal with mean $s(\tfrac12 + \Delta)$ and variance $s/4$ , so the majority is right with probability

\Phi\!\left(\frac{s\Delta}{\sqrt{s}/2}\right) = \Phi\big(2\Delta\sqrt s\big) = \Phi\!\left(\frac{2}{\sqrt{2\pi n}}\sqrt{\frac{k}{2}}\right) = \Phi\!\left(\sqrt{\frac{k}{\pi n}}\right) \approx \frac12 + \varphi(0)\sqrt{\frac{k}{\pi n}} = \frac12 + \frac{1}{\pi}\sqrt{\frac{k}{2n}}.

The last step is the first-order expansion of $\Phi$ at 0, valid while the gain is small. That is about $\tfrac12 + 0.225\sqrt{k/n}$ . Each kept vector carries a sliver of information about $\mathbf{y}$ , of size $1/\sqrt n$ , and the majority adds the slivers coherently while their noise averages out.

In plain words, a leaderboard that reports exact scores leaks a sliver of information about its labels with every submission, and $k$ slivers add up to about $\sqrt{k/n}$ of fake accuracy. With 1,000 submissions against 2,000 public labels, the attack scores about 65.5% while knowing nothing. On labels the board never used it scores 50%, so the whole gain is overfitting.

Interview

The boosting attack

A classic interview question asks why a public leaderboard cannot be trusted, and the strongest answer is to break one. Implement boosting_attack(oracle, n, k, rng). Here oracle(u) returns the accuracy of a 0/1 vector u of length n on a hidden public subset of the items.

Draw all $k$ random vectors at once with rng.integers(0, 2, (k, n)).
Query the oracle on each and keep those scoring at least 0.5.
Return their coordinatewise majority as an integer 0/1 vector, with ties broken toward 1.

The setup provides make_leaderboard(n, rng), which returns hidden labels, the public mask, and an oracle, for your own experiments.

import numpy as np

def boosting_attack(oracle, n, k, rng):
    raise NotImplementedError

Tests, hints, verified solutions, and AI help, with Pro at $7.99 a month.

The Ladder, a leaderboard that leaks less

The attack works because every exact score leaks information, so the fix is to release less. A competition needs to know which submission is best so far, not the score of every submission. The board can therefore stay silent unless a submission makes a clear improvement. Blum and Hardt (2015) call this the Ladder.

Hiding the small gains is the whole design. Of eight submissions the board moved six times, and of the other two the analyst learned only that neither was a clear improvement. A board that moves rarely has a short history, and a short history is what the next guarantee rests on.

∴ The Ladder's guarantee, and the limit no leaderboard can beat

Define the leaderboard error after $k$ submissions as $\max_{t \le k}\big|R_t - \max_{i \le t} a_i\big|$ , where $a_i$ is the true accuracy of submission $i$ on the data distribution. Take any analyst who chooses each submission adaptively from everything released so far. The Ladder with a step $\eta$ of order $n^{-1/3}\log^{1/3}(kn)$ has, with high probability,

\text{leaderboard error} = O\!\left(\frac{\log^{1/3}(kn)}{n^{1/3}}\right).

A board that reports every score with fine precision suffers leaderboard error $\Omega(\sqrt{k/n})$ under the boosting attack. And no mechanism of any kind can guarantee error below $\Omega(\sqrt{\log k / n})$ , even for submissions fixed in advance (Blum and Hardt 2015, Theorems 3.1 and 3.3, Corollary 5.2).

∂ Why the Ladder's error grows only like a logarithm of k

The argument is a compression argument. Whatever the analyst does, the entire history of the board is determined by the steps at which it moved and the values it moved to. Accuracies lie in $[0, 1]$ and each move raises the board by more than $\eta$ , so there are at most about $1/\eta$ moves. Each goes to one of about $1/\eta$ values, at one of $k$ positions. So the whole history can be written down with about $B = (1/\eta + 2)\log(4k/\eta)$ bits, which is the paper's count. Fix the analyst's random seed. An adaptive analyst is then a deterministic function of the history, so across all possible histories it can submit at most $2^B$ distinct classifiers.

Now recall Hoeffding's inequality from §15.1. The mean of $n$ independent variables in $[0, 1]$ falls more than $\gamma$ below its expectation with probability at most $e^{-2\gamma^2 n}$ . The same bound holds for rising more than $\gamma$ above it, so a deviation of more than $\gamma$ either way has probability at most $2e^{-2\gamma^2 n}$ . A fixed classifier's public accuracy is the mean of $n$ independent 0/1 outcomes, so it is within $\gamma$ of its true accuracy except with that probability. (We write $\gamma$ for the deviation because $\varepsilon$ is reserved in this section for the label-error rate.) A union bound, which adds up the failure probabilities of all $2^B$ possible classifiers, gives a total failure probability of at most $2\cdot 2^B e^{-2\gamma^2 n}$ . That is small once $\gamma \approx \sqrt{B/n}$ .

Balancing $\gamma \approx \sqrt{1/(\eta n)}$ against the rounding error $\eta$ gives $\eta \approx n^{-1/3}$ , up to the logarithm, and an error of the same order. The board that shows every score has no such bound. Its history takes about $\log n$ bits per submission, so $B$ grows linearly in $k$ .

In plain words, the Ladder caps how much an adaptive analyst can learn from the board, and what cannot be learned cannot be overfit. Its error depends on the number of submissions $k$ only through a logarithm, while the error of a board that shows every score grows like $\sqrt k$ . The guarantee is of order $n^{-1/3}$ , larger than the $\sqrt{\log k/n}$ floor that no mechanism can beat, and whether any mechanism can close that gap was left open by the paper. Against the boosting attack, the simulation below shows the Ladder's board at roughly the best-of- $k$ noise of the previous part.

Research

The Ladder

Implement ladder(scores, eta). It takes the public accuracies of the submissions in the order they arrive and returns the array of released values $R_1, \dots, R_k$ of the Ladder, starting from $R_0 = 0$ . For the eight-submission sequence worked above, [0.513, 0.548, 0.561, 0.587, 0.50, 0.595, 0.603, 0.622] with $\eta = 0.02$ , the answer is [0.52, 0.54, 0.56, 0.58, 0.58, 0.58, 0.60, 0.62].

The tests check that sequence, then run the boosting attack against your board. The attacker votes with only the submissions that moved the board. The tests check that the board ends below the best-of- $k$ noise plus two steps.

import numpy as np

def ladder(scores, eta):
    return np.array(scores, dtype=float)      # placeholder: the plain leaderboard shows every score

Tests, hints, verified solutions, and AI help, with Pro at $7.99 a month.

Now the attack and the Ladder side by side, on 2,000 public and 2,000 private labels, averaged over 20 boards. Against the Ladder the attacker does its best, voting with the submissions that moved the board, since only those carry information.

Python · cell 4

✏️ The attack against two leaderboards

Against the board that shows every score, the attack reaches 0.516, 0.553, and 0.654 public accuracy for $k = 10$ , 100, and 1,000. That matches the heuristic $\Phi\big(\sqrt{k/(\pi n)}\big)$ , which gives 0.516, 0.550, and 0.655. Its private accuracy stays at 0.50 throughout.

Against the Ladder the board ends at 0.513, 0.525, and 0.532. That is not the attack working. It is close to the best of $k$ random public scores, $\tfrac12 + \tfrac{1}{2\sqrt{2000}}\,\mathbb{E}[\max_{j \le 1000} Z_j] \approx 0.536$ for $k = 1{,}000$ , held a little lower by the Ladder's steps. It is the unavoidable selection bias of the previous part.

The widget runs the attack on one board with 2,000 public and 2,000 private labels and a Ladder step of $\eta = 0.01$ . After each of $k$ random submissions, the pink line is what the board shows for the attack's vote and the green line is the vote's accuracy on the private labels. Against the Ladder, the pink line is the Ladder's released value.

The practical lesson holds for any internal leaderboard. Release as little as the decision needs, because every released number is a little information an optimizer can use.

🔬 The Ladder in a real competition, and the reusable holdout

In practice the paper uses a version with no step to tune. A submission moves the board only if it beats the current best by more than $\hat\sigma/\sqrt n$ , where $\hat\sigma$ is the sample standard deviation of the per-item difference in loss. That is the threshold of a paired $t$ -test. On the 1,785 successfully processed submissions from 200 teams in Kaggle's Photo Quality Prediction competition, it produced nearly the same final rankings as Kaggle's own board.

The Ladder protects one number, the best score. Dwork et al. (2015) generalized the idea to any sequence of adaptively chosen questions asked of a held-out set. Their reusable holdout, called Thresholdout, answers each question from the training data unless the held-out set disagrees by more than a noisy threshold. Only then does it reveal a noisy held-out answer. It provably allows many more adaptive questions than plain reuse of the held-out set.

Why benchmarks saturate below 100%

A benchmark that cannot separate the models you care about has stopped working, even if every number on it is correct. This happens in two ways. Top scores can approach the best score the benchmark allows, which is below 100% whenever some answer keys are wrong. Or top scores can get too close to each other for the benchmark's size to resolve.

The example shows two effects at once. The perfect model loses 6 points, so wrong keys put a cap on the score. The 80% model loses only 4.4, so the true 20-point gap between the two models shows up as 18.4. Wrong keys also compress the scale below the cap. The definition names the cap.

The compression is the subtler effect, because it moves every score, not only the top one. The next result gives the exact formula under a simple model of how keys go wrong, and inverts it to recover a model's true accuracy.

∴ The label-error ceiling

Consider multiple-choice items with $K_{\text{opt}}$ options. Suppose a fraction $\varepsilon$ of items has a wrong key, chosen uniformly among the $K_{\text{opt}} - 1$ wrong options. Suppose also that whether an item is mislabeled is independent of whether the model answers it correctly, and that when the model is wrong its choice is independent of the wrong key. Then a model with true accuracy $a$ has expected observed accuracy

\mathbb{E}[\text{observed}] = a(1 - \varepsilon) + (1 - a)\,\frac{\varepsilon}{K_{\text{opt}} - 1}.

A perfect model scores $1 - \varepsilon$ , and a model that is always wrong scores $\varepsilon/(K_{\text{opt}} - 1)$ . For $\varepsilon \lt (K_{\text{opt}} - 1)/K_{\text{opt}}$ , the true accuracy is recovered as

a = \frac{\text{observed} - \varepsilon/(K_{\text{opt}} - 1)}{1 - \varepsilon - \varepsilon/(K_{\text{opt}} - 1)}.

∂ Proof of the label-error formula

Condition on whether the key is right and whether the model is right. These two events are independent by assumption, so the four cases have probabilities $(1-\varepsilon)a$ , $(1-\varepsilon)(1-a)$ , $\varepsilon a$ , and $\varepsilon(1-a)$ .

If the key is right, the model matches it exactly when it is right, so the first case contributes $(1-\varepsilon)a$ and the second nothing. If the key is wrong and the model is right, the model chose the true answer and the key is another option, so they never match. If both are wrong, the model chose one of the $K_{\text{opt}} - 1$ wrong options and the key is a uniformly random one of the same $K_{\text{opt}} - 1$ , independently, so they match with probability $1/(K_{\text{opt}} - 1)$ .

Adding the cases gives the formula. It is linear in $a$ with slope $1 - \varepsilon - \varepsilon/(K_{\text{opt}} - 1) = 1 - \varepsilon K_{\text{opt}}/(K_{\text{opt}} - 1)$ , which is positive under the stated condition, so it can be inverted. $\square$

In plain words, label errors cap the score and also squeeze the scale below the cap. Every difference between two models shrinks by the factor $1 - \varepsilon K_{\text{opt}}/(K_{\text{opt}} - 1)$ , so a noisy key makes models look closer together than they are. In the hand example the factor is $1 - 0.06 \cdot 4/3 = 0.92$ , so a true 10-point gap shows up as 9.2.

The assumptions matter, and MMLU shows why. MMLU is a multiple-choice benchmark of academic knowledge, with four options per question and 14,042 test questions in 57 subjects. Gema et al. (2024) re-annotated 5,700 of its questions across all 57 subjects, a set they call MMLU-Redux. They estimated that 6.49% of the full benchmark's questions contain errors, and 57% of the sampled Virology questions did. Only part of that is a wrong key. Their estimates by type are 1.42% wrong ground truth, 1.54% multiple correct answers, 0.62% no correct answer, 2.47% unclear questions, and 0.44% unclear options. On an item with two correct answers, a perfect model may well match the key.

The cell checks the formula by simulation and inverts it for MMLU. Its last line recomputes the ceiling with an assumed chance, for each error type, that a perfect model still matches the key.

Python · cell 5

✏️ What does 88% on MMLU mean?

Take $\varepsilon = 0.0649$ and four options. Under the uniform-wrong-key model the ceiling is $1 - 0.0649 = 93.5\%$ . An observed 88% implies a true accuracy of $(0.88 - 0.0649/3)/(1 - 0.0649 - 0.0649/3) = 0.8584/0.9135 = 94.0\%$ on the correctly keyed questions. The simulation agrees with the forward formula, 0.8806 against 0.8803.

The model simplifies Gema et al.'s categories. Suppose a perfect model matches the key half the time on items with multiple correct answers, unclear questions, or unclear options, and a quarter of the time on items with no correct answer. Then the ceiling rises to 95.9%. Either way, the best MMLU score in Epoch AI's compilation, 88.1% for GPT-4o, sits within a few points of the ceiling, and some of the remaining points cannot be scored at all.

Label errors are not only a problem of old benchmarks. Humanity's Last Exam (HLE) was built in 2025 to be hard for the best models. It has 2,500 questions written by experts, 76% with a short exact answer and the rest multiple choice (Phan et al. 2025). Two audits found that experts dispute the key on about 15% to 29% of its questions, depending on the subset and on how strict the check is.

For a short-answer item, a wrong answer essentially never matches a wrong key, which is the limit $K_{\text{opt}} \to \infty$ of the theorem. The floor $\varepsilon/(K_{\text{opt}} - 1)$ then vanishes and the expected score is $a(1 - \varepsilon)$ . A key-error rate of 29% caps a subset near 71%, and 15.4% puts the ceiling of the whole benchmark near 85%. A perfect expert would score in those ranges, and a model that matched every key would be matching the errors too. So the best score of 54.8% in October 2026 still leaves real headroom, but a future 80% on HLE may already be close to saturation.

Core

What true accuracy does a score imply on a noisy benchmark?

Implement ceiling(eps) and implied_accuracy(observed, eps, K_opt, n) under the model of the theorem.

ceiling(eps) returns the score of a perfect model.
implied_accuracy inverts the formula and clips its answer to $[0, 1]$ , since sampling noise can push a score slightly past either end.
implied_accuracy raises ValueError when the observed score exceeds the ceiling by more than three standard errors, that is, when $\text{observed} > 1 - \varepsilon + 3\sqrt{\text{observed}(1 - \text{observed})/n}$ . A score that high means the error model or the reported $\varepsilon$ is wrong.

import numpy as np

def ceiling(eps):
    return 1.0                                  # placeholder: a perfect model would score 100%

def implied_accuracy(observed, eps, K_opt, n):
    return observed                             # placeholder: takes the score at face value

Tests, hints, verified solutions, and AI help, with Pro at $7.99 a month.

When a benchmark is too small to rank the leaders

Saturation also comes from the other direction, a benchmark too small for the gaps it is asked to resolve. §9.6 showed one face of this at the bottom of the scale. A model with a true accuracy of $10^{-4}$ scores exactly zero on a 200-question test about 98% of the time, since $(1 - 10^{-4})^{200} \approx 0.98$ . A metric that only counts exact successes then turns smooth progress into an apparent sudden jump. At the top of the scale the same arithmetic stops the board from ranking models at all.

✏️ Is the 96.4% model best?

GPQA Diamond has $n = 198$ questions. At $p = 0.964$ the standard error of one model's score is $\sqrt{0.964 \cdot 0.036/198} = 1.32$ points. (Epoch reports 1.2, because its scores average repeated runs, 96.40% being 190.875 of 198 questions. §22.2 explains which of the two error bars a claim needs.) The five leaders span 1.0 point, under one standard error of a single score.

For two models near 96%, the unpaired standard error of their difference is about $\sqrt{2 \cdot 0.96 \cdot 0.04/198} = 1.97$ points, so the 1-point gap between first and fifth is half a standard error. To resolve a 1-point gap at the 5% level with this unpaired comparison you would need $n \ge 2 \cdot 0.96 \cdot 0.04 \cdot (1.96/0.01)^2 \approx 2{,}950$ questions, fifteen times Diamond's size. A paired comparison on the same questions needs fewer, as §22.2 shows, but the order of the top five is not established.

The label ceiling is unknown too. Diamond kept only questions on which experts agreed, so its error rate should be below the full set's. But at 96% the remaining seven or so missed questions are few enough that a handful of debatable keys could account for them.

📌 Benchmarks lose resolution fast

MMLU (Hendrycks et al. 2020) was built to be hard for the models of its day. By late 2024 the best scores in Epoch AI's compilation reached 88.1%, within a few points of the ceiling its 6.49% error rate implies (Gema et al. 2024).
GPQA (Rein et al. 2023, 448 main-set questions) launched with PhD experts at about 65%, skilled non-experts with web access at 34%, and its strongest GPT-4 baseline at 39%. By October 2026 the best score on its 198-question Diamond subset was 96.4% (Epoch AI 2026).
FrontierMath (Glazer et al. 2024), a set of new, unpublished math problems written by mathematicians, launched in November 2024 with the best models solving under 2% of them. By October 2026 the best models scored 93.7% on tiers 1 to 3, and one model scored 100% on the second version of tier 4, the research-level problems (Epoch AI 2026). Epoch's newest set, 68 problems posed or studied by Erdős that were open as of August 2026, had at most 3 solved by any model.
Humanity's Last Exam, with 2,500 questions, was still unsaturated at 54.8% for the best model in October 2026 (Epoch AI 2026), but its ceiling is uncertain. Audits put it near 71% to 82% on the text-only chemistry and biology questions and near 85% overall (FutureHouse 2025, Phan et al. 2025).
Across 3,765 benchmarks in computer vision and natural language processing, a large fraction quickly trended toward near-saturation (Ott et al. 2022).

Core

Is this benchmark saturated?

Implement saturation_report(acc, n, eps, human=None, alpha=0.05) for a list of model accuracies on one benchmark of $n$ items. It returns a dict with three keys. Write $\text{top}$ for the highest accuracy and $\mathrm{SE}_j = \sqrt{\text{acc}_j(1 - \text{acc}_j)/n}$ for model $j$ 's standard error.

"ceiling" is the effective ceiling $c = \min(1 - \varepsilon, h)$ , or $1 - \varepsilon$ when human is None.
"saturated" says whether the top score is within noise of the ceiling by a one-sided test, that is, whether $\text{top} \ge c - z_{1-\alpha}\,\mathrm{SE}_{\text{top}}$ .
"n_tied" is a rough resolution count. It is the number of models, the leader included, whose gap to the leader is below $z_{1-\alpha/2}\sqrt{\mathrm{SE}_{\text{top}}^2 + \mathrm{SE}_j^2}$ , each model using its own standard error.

Real comparisons should be paired and corrected for comparing many models at once, which §22.2 does.

import numpy as np
from scipy import stats

def saturation_report(acc, n, eps, human=None, alpha=0.05):
    raise NotImplementedError

Tests, hints, verified solutions, and AI help, with Pro at $7.99 a month.

How do you tell saturation from a real plateau, where models have really stopped improving? Read the remaining errors. If expert review finds that most items the best models still miss have wrong or debatable keys, the benchmark is saturated, and further gains are noise. If the remaining items are valid and hard, the plateau is real and the benchmark still has something to say. MMLU-Redux is exactly this exercise done at scale, and its error types are the reason the MMLU ceiling above is a range rather than a number.

The gap between benchmark scores and real use

Everything so far concerned a benchmark's internal health. The validity bias $b_{\text{val}}$ asks a harder question. Does a perfectly measured benchmark score predict the thing you will use the model for? That is a question of criterion validity, and the only way to answer it is to measure the criterion itself.

How can both the benchmark curve and the trial be right? List the differences between the two measurements, and each one is a validity gap.

Different items. SWE-bench tasks are self-contained issues with tests. The trial's tasks were the developers' own backlog, in repositories with very high quality bars, much unwritten context, and code review.
Different graders. A benchmark task passes when hidden tests pass. A real change passes when it meets the project's standards. The developers accepted fewer than 44% of the AI's generations and spent about 9% of their time reviewing and cleaning AI output.
Different populations. The trial's developers were experts on their own code, which is exactly where help is hardest to give. The authors list high familiarity with the repository among the factors likely to contribute to the slowdown.
Different estimands. The estimand is the quantity a study sets out to estimate. A benchmark estimates whether a model can finish a task alone. The trial estimated the time a human takes with the model's help, which includes prompting, waiting, and checking.

The authors are careful about scope. Their result is about early-2025 tools, for one population, in one setting, and it does not show that AI cannot speed up software work in general. Its lesson for evaluation is sharper than its lesson for productivity. People, experts included, badly misjudge usefulness from experience and from benchmark curves, and only a measurement of the criterion settles the question.

GDPval (Patwardhan et al. 2025) is one response, a benchmark built from the work itself. Its 1,320 tasks come from the real deliverables of professionals in 44 occupations across the 9 sectors that contribute most to US GDP. The professionals average 14 years of experience, and a task takes one of them about 7 hours on average. Graders from the same occupations compare a model's deliverable with a human expert's without knowing which is which. The metric is a win rate against the expert, so its ceiling is set by the human baseline, and the baseline can be replaced by a stronger one as models catch up. On the public 220-task gold subset, 47.6% of Claude Opus 4.1's deliverables were judged as good as or better than the expert's at release. GDPval has its own validity limits, which its authors name. Its tasks are precisely specified and done in one shot, while real work involves finding out what is wanted and iterating.

What labs do about it

No single practice fixes all five error terms, so evaluations that hold up combine several. The table lists the main practices by the term each attacks.

practice	error term it attacks	what it costs	what it misses
private held-out test sets, used rarely	Goodhart pressure, leakage	building and guarding the set	validity, since it shares the public set's distribution
living benchmarks with fresh, dated items (LiveBench, White et al. 2024)	leakage, saturation	continuous item writing	comparability across versions
adversarial collection, humans writing items a target model fails (Dynabench, Kiela et al. 2021)	saturation	annotators, and items tied to one model	coverage of ordinary use
contamination-resistant construction and detection (§22.3)	leakage	engineering	leaks it cannot see
reporting the protocol and intervals (§22.2)	sampling, protocol	almost nothing	bias of any kind
fixing the primary metric in advance, counting every attempt, or a Ladder	selection across runs	discipline	every bias that does not come from selection
validated graders, expert re-labeling (§22.4)	grader	expert time	validity
task-specific evals from real traces (§22.6)	validity	building one per product	comparison with other labs
measuring use, with A/B tests and randomized trials	validity	time, users, and ethics review	anything before deployment

The last row is the only one that measures $\mu^\star$ directly, and the only one you cannot run before launch. That is why the rest of the chapter exists. Each later section makes one of the cheaper rows rigorous.

Notice also what none of them does. Every row treats the model as a black box judged by its outputs. Asking what a model computes inside, instead of only what it outputs, is the program of interpretability, which complements behavioral evaluation rather than replacing it. And putting all these pieces together into a system that gates releases, monitors deployed models, and escalates safety concerns is the job of designing an evaluation and safety system, a standard design interview at frontier labs.

Measuring your own selection bias in PyTorch

The best-of- $m$ formula needs $\rho$ and assumes equally good submissions. In practice you have something better, the per-item results of every submission, and you can estimate your own leaderboard's selection bias directly.

Split the public items in half at random.
Pick the best submission on one half and score it on the other half. The gap is the selection bias at $n/2$ items.
The bias scales like the standard error, so divide by $\sqrt 2$ to convert it to $n$ items. Average over many random splits.

The block checks this against the true public-minus-private gap and against the formula. It uses $m = 50$ submissions whose items have pass rates $p_i \sim \operatorname{Beta}(1.4, 0.6)$ . For a Beta distribution with parameters $a$ and $b$ and mean $p = a/(a+b)$ , $\operatorname{Var}(p_i) = p(1-p)/(a + b + 1)$ . So $\rho = \sigma_b^2/(p(1-p)) = 1/(1.4 + 0.6 + 1) = 1/3$ .

🔥 In PyTorchfor a machine with PyTorch

All three agree at about 2.6 points (2.64, 2.62, and 2.66 in this run). The split-half estimate needs no model of the submissions. It still works when they differ in true quality, where only the rescaling by $\sqrt 2$ is approximate. Run it before you report the best of many runs, and subtract what it finds. Better still, report the chosen submission's score on items you never selected on.

🧭 What to remember

An evaluation is an estimator. Its error is validity bias plus measurement bias (leakage, grader, environment) plus sampling noise, and its mean squared error is the squared total bias plus a variance that falls like $1/n$ . Past $n^\ast = (\sigma_b^2 + \sigma_w^2/K)/b^2$ items, more items buy almost nothing.
Construct validity asks whether a score supports a particular claim, argued from content, criterion, convergent, and discriminant evidence. State the claim the data support, under the protocol used. A benchmark meant to measure something other than capability must show a low correlation with the capabilities score, the first principal component of standardized benchmark scores computed without it.
Reporting the best of $m$ equally good submissions inflates the score by $\mathrm{SE}\sqrt{1-\rho}\,\mathbb{E}[\max_j Z_j] \le \mathrm{SE}\sqrt{1-\rho}\sqrt{2\ln m}$ , about 3.6 points for $m = 100$ at $n = 1{,}000$ and $p = 0.7$ . Estimate it by split-half selection and report held-out scores.
An adversary with exact scores can reach $\tfrac12 + \Omega(\sqrt{k/n})$ public accuracy with random guesses. The Ladder releases a new score only on a clear improvement, so its error grows with $k$ only through a logarithm, and against the attack its board sits near the unavoidable best-of- $k$ noise.
Label errors cap a score at $1 - \varepsilon$ and shrink every gap. MMLU sits within a few points of its ceiling, audits put Humanity's Last Exam's ceiling near 85% rather than 100%, and GPQA Diamond's top five lie within one standard error of each other.
Benchmark curves and usefulness can diverge, as the METR trial showed. Only measuring the criterion settles it.
Next, §22.2 takes on the sampling term, with the benchmark families, the standard error derived for a fixed and a resampled test set, paired and many-model comparisons, and the protocol choices that make two "MMLU" numbers different measurements.

Jacobs and Wallach (2021). Measurement and Fairness. Measurement theory for computer scientists, with construct reliability and the seven aspects of construct validity defined carefully.
Raji et al. (2021). AI and the Everything in the Whole Wide World Benchmark and Bean et al. (2025), Measuring what Matters: Construct Validity in Large Language Model Benchmarks. Why "general" benchmarks overclaim, and a review of 445 language model benchmarks with eight recommendations.
Blum and Hardt (2015). The Ladder: A Reliable Leaderboard for Machine Learning Competitions and Dwork et al. (2015), Generalization in Adaptive Data Analysis and Holdout Reuse. The boosting attack, the Ladder and its compression proof, and the reusable holdout with the theory of adaptive data analysis behind it.
Ren et al. (2024). Safetywashing: Do AI Safety Benchmarks Actually Measure Safety Progress? and Needham et al. (2025), Large Language Models Often Know When They Are Being Evaluated. The capabilities score and the capabilities correlations of safety benchmarks, and a direct measurement of evaluation awareness.
Recht et al. (2019). Do ImageNet Classifiers Generalize to ImageNet? and Mania et al. (2019), Model Similarity Mitigates Test Set Overuse. New test sets, falling numbers, stable rankings, and why similar models overfit less.
Roelofs et al. (2019). A Meta-Analysis of Overfitting in Machine Learning. Public against private scores in 120 Kaggle competitions.
Gema et al. (2024). Are We Done with MMLU? and FutureHouse (2025), About 30% of Humanity's Last Exam chemistry/biology answers are likely wrong. MMLU-Redux with the 6.49% error estimate and its taxonomy of label errors, and the literature audit of HLE's keys.
Becker et al. (2025). Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity and Patwardhan et al. (2025), GDPval: Evaluating AI Model Performance on Real-World Economically Valuable Tasks. Measuring the criterion, in a randomized trial and in a benchmark built from real work.

Made for larger screens

Why evaluation is hard

Five ways a score can be wrong

An evaluation is an estimator

What construct validity asks

Checking that a benchmark measures something other than capability

The capabilities correlation of a benchmark

Goodhart's law on a shared test set

The selection bias of a leaderboard

How far an adversary can push a leaderboard

The boosting attack

The Ladder, a leaderboard that leaks less

The Ladder

Why benchmarks saturate below 100%

What true accuracy does a score imply on a noisy benchmark?

When a benchmark is too small to rank the leaders

Is this benchmark saturated?

The gap between benchmark scores and real use

What labs do about it

Measuring your own selection bias in PyTorch

Read more

Watch next