A benchmark score is a measurement, and every measurement has an error. This section is about how large that error is, where it comes from, and how to report a score so that the error shows. It matters because labs pick models, gate releases, and choose which training recipe to scale on gaps of one or two points. Many of those gaps are smaller than the noise.

Two stories show the two kinds of error. The first is about how a model is asked. MMLU is a multiple-choice exam of 14,042 questions in 57 school and university subjects. In June 2023 the Hugging Face Open LLM Leaderboard, a public ranking of open models, put LLaMA 65B at 48.8% on it. The model's authors had reported 63.4% (Touvron et al. 2023). The leaderboard team then ran the same model on the same questions with three public implementations of MMLU and got 63.6%, 63.7%, and 48.8% (Hugging Face 2023). The model had not changed. Only the way it was asked and scored had.

The second story is about size. AIME, an American high-school math competition on which reasoning models are now scored, sets 30 problems a year, so one problem is worth 3.3 points. If one model card reports 87% on AIME 2025 and a rival reports 80%, the gap is two problems. We will see that one exam of 30 problems can almost never tell two such models apart.

§22.1 treated an evaluation as an estimator, a number computed from a sample that stands in for a quantity we care about, and split its error into parts with different causes. This section measures two of those parts. The sampling error is how much a score would move if we graded a fresh draw of questions, or a fresh draw of the model's sampled answers. The protocol error is how much it would move under a different but equally reasonable way of asking and scoring.

Here is the plan. We first say exactly what a score measures and survey the benchmarks frontier labs report in October 2026. Then we derive the error bar of a score for the two different claims people make with it, and compare many models on one benchmark honestly. Then we take apart multiple-choice scoring and prompt sensitivity, measure the noise that training itself adds, and end with item response theory, which asks how few questions a measurement needs.

A benchmark score is a protocol plus a sample

A benchmark is a fixed set of test questions, called items, with a rule for grading answers. To score a model, you take $n$ items and let the model answer each one $K$ times. A grader marks every answer with 1 for right and 0 for wrong, or a number in between for partial credit. Then you average.

Take three items and two answers each. The grades are (1, 1), (1, 0), and (0, 0). The item averages are 1, 0.5, and 0, and the score is their mean, 0.5.

In symbols, write $x_1, \dots, x_n$ for the items, drawn from the benchmark's pool of questions $\mathcal{D}_{\text{bench}}$. Write $y_{i1}, \dots, y_{iK}$ for the model's $K$ answers to item $x_i$, and $g(x_i, y_{ik})$ for the grade of one answer. The reported score is

In plain words, grade every sampled answer, average the grades of each item, then average over the items. Everything between the item and the grade is part of the measurement. That includes how the question is shown, how the answer is pulled out of the model's text, and how it is checked.

The MMLU story is this definition at work. All three implementations used the same questions and the same five few-shot examples. They differed a little in the wording of the prompt and a lot in how they read the model's answer, items 3, 5, and 6 of the list.

So a benchmark number carries two kinds of error, and they need different tools.

In practice, the first question to ask of any reported score is "under which protocol?". An interviewer who shows you two different numbers for one model expects that question back.

The benchmark families labs report

Frontier model cards report a dozen or so benchmarks, chosen to cover what users ask of a model. They fall into families, each with a format and a grader. Every family goes through the same life cycle. A benchmark is built to be hard, and models climb it within a year or two. Then it saturates, meaning the best scores sit so close to the highest score the benchmark allows that it can no longer separate models. That highest score is below 100%, because some answer keys are wrong and a perfect model cannot match them (§22.1). After saturation, labs stop reporting the benchmark.

The status column below comes mostly from Epoch AI, a research group whose benchmarking hub (retrieved 20 October 2026) runs public benchmarks on new models with one fixed protocol per benchmark and also collects results published elsewhere. Where Epoch has no run, the status comes from the labs' own reports.

Here is the top of Epoch's October 2026 data for the benchmarks where sampling error matters most. FrontierMath is a set of unpublished, research-level math problems with answers a script can check, written by mathematicians. The last column is the weight of one item in points, $100/n$ for $n$ items.

The table teaches three things. First, the top five on GPQA Diamond lie within one point, two questions out of 198, and Epoch's own standard error is larger than the gaps. Second, FrontierMath tiers 1 to 3 were built in late 2024 to stay out of reach for years, and they are above 90% two years later. The frontier has moved to open research problems, where a score is a count of a few successes. Third, a small benchmark's resolution is set by its size before anything else, which is where the statistics begin.

One more detail is worth noticing. Epoch's methodology notes say that it grades most models 16 times on every GPQA Diamond question and averages the grades. Its standard error for the leader, 1.19 points, is smaller than the 1.32 points a single pass would give, $\sqrt{0.964 \cdot 0.036/198}$. The next two parts of the section explain why.

The error bar for one score

The standard error of a score is the standard deviation it would show if we repeated the whole evaluation many times. Start with the simplest case. A model answers each of $n$ independent items once, and each answer is right with probability $p$. The accuracy $\hat p$ is then the mean of $n$ coin flips, so its standard error is $\sqrt{p(1-p)/n}$ (§0.11).

Try it on AIME. A model solves 24 of 30 problems, so $\hat p = 0.8$ and the standard error is $\sqrt{0.8 \cdot 0.2/30} = 0.073$, or 7.3 points. A 95% interval reaches about two standard errors on each side, so it runs from roughly 66% to 94%. For small $n$, or for scores near 0 or 1, use the Wilson interval instead. It keeps every candidate accuracy $p$ that a normal test, using the standard error at that $p$ rather than at $\hat p$, would not reject. So it stays inside 0 and 1 and keeps close to its promised 95% coverage near the ends (§0.11). Here it runs from 63% to 90%.

Comparing two models needs one more idea. When models A and B answer the same items, compare them item by item. Write $a_i$ and $b_i$ for their grades on item $i$ and $\Delta_i = b_i - a_i$ for the difference. The mean of the $\Delta_i$ is the gap in accuracy. Its standard error is $s_\Delta/\sqrt n$, where $s_\Delta$ is the standard deviation of the $\Delta_i$. This paired standard error is smaller than the unpaired one, because both models find the same items easy or hard, and each difference cancels that shared difficulty.

Three more tools from §0.11 appear below, so here they are in a sentence or two each.

• McNemar's test compares two models using only the items where they disagree, the $n_{10}$ that only A gets right and the $n_{01}$ that only B gets right. If the models were equally good, each disagreement would go either way like a fair coin, so the test asks how lopsided the split is.
• Holm's method adjusts the p-values of many tests run together. It keeps the chance of even one false claim of a difference across all of them, the family-wise error rate, below the chosen level $\alpha$.
• Power is the chance that a test detects a real gap. Detecting a true gap $\Delta$ with 80% power at $\alpha = 0.05$ takes about $n = (2.80\,s_\Delta/\Delta)^2$ items, where $2.80 = 1.96 + 0.84$.

In practice, report a score with its interval and a comparison with its paired interval. An interviewer who asks for "the error bar on 24 of 30" wants the 7 points, and the remark that one year of AIME cannot rank close models.

Many samples per question, and two error bars

Reasoning models are usually scored with several sampled answers per question, often $K = 8$ or $K = 64$, and the score is the average grade. That changes two things. The $K$ answers to one question are not $K$ independent items. And the right error bar now depends on which claim you are making.

Start with the two claims. Ask "what is this model's score on AIME 2025?" and the 30 problems are fixed. Nobody will ever set AIME 2025 again, so the only randomness left is in the model's sampled answers. Ask "how good is this model at competition math?" and AIME 2025 is a sample of 30 from all the problems that could have been set. A different year would have given a different score.

The exact quantity a claim is about is called its estimand. These two claims have different estimands, and so they have different error bars. To write them down, give each item $i$ a true pass rate $p_i$, the chance that one sampled answer to it is graded right. Write $\bar g_i = \frac1K\sum_k g(x_i, y_{ik})$ for the average of its $K$ grades, so that $\hat\mu = \frac1n\sum_i \bar g_i$.

The two new symbols have plain meanings, the same as in §14.7, which derived the population formula. The between-item variance $\sigma_b^2$ is how much the true pass rates differ from item to item. The within-item variance $\sigma_w^2$ is the average spread of one sampled grade around its own item's pass rate. The two add up to the variance of one graded answer, $\sigma_b^2 + \sigma_w^2 = \bar p(1 - \bar p)$ with $\bar p = \mathbb{E}[p_i]$ (§0.11). That follows in one line from $\mathbb{E}[p_i(1-p_i)] = \bar p - \mathbb{E}[p_i^2]$ and $\mathbb{E}[p_i^2] = \sigma_b^2 + \bar p^2$.

In plain words, the variance of one graded answer, $\bar p(1 - \bar p)$, has a part that comes from which question was drawn and a part that comes from which answer was sampled. With $K = 1$ the population standard error is the familiar $\sqrt{\bar p(1-\bar p)/n}$. With $K > 1$ only the sampling part shrinks. The fixed-question error bar counts only the dice the model rolls when it samples. The population error bar adds the dice the benchmark's authors rolled when they picked the questions. More samples per question remove the first kind of noise and do nothing about the second.

Which error bar does a claim need? "Model A scored 87% on AIME 2025" is a claim about these 30 problems, so the fixed-question error bar is the honest one, and it can be small. "Model A is better at competition math than model B" is a claim about the population of problems. It needs the population error bar, or better the paired one, which also treats questions as sampled. Miller (2024) frames evaluations in these terms. The benchmark is a sample from a larger super-population of possible questions, and the population version is the right one for most claims.

Greedy decoding, which always picks the most likely next token, makes the difference stark. Every $p_i$ is then 0 or 1, so every $p_i(1-p_i)$ is zero and the fixed-question variance is exactly zero. A deterministic model's AIME score has no fixed-question error bar at all, and it still says little about the model's skill, because the population error bar is as large as ever. (Even greedy outputs are not perfectly reproducible on GPUs, as §14.7 noted, so "exactly zero" holds only in principle.)

Both variances can be estimated from the $n \times K$ matrix of grades. For the population one, take the sample standard deviation of the item means $\bar g_i$, with the $n - 1$ denominator, and divide by $\sqrt n$. Each $\bar g_i$ has variance exactly $\sigma_b^2 + \sigma_w^2/K$, so the squared result is an unbiased estimate of the population variance. For the fixed-question one, let $s_i^2$ be the sample variance of item $i$'s $K$ grades, computed with the $K - 1$ denominator. It is an unbiased estimate of $p_i(1-p_i)$, so $\frac{1}{n^2}\sum_i s_i^2/K$ is an unbiased estimate of the fixed-question variance. It needs $K \ge 2$. What you must never do is treat all $nK$ grades as independent items. That divides by $\sqrt{nK}$ and shrinks the error bar by up to a factor of $\sqrt K$.

Now Epoch's GPQA numbers make sense. With $K = 16$ answers per question, the population standard error is $\sqrt{(\sigma_b^2 + \sigma_w^2/16)/198}$, which is below the single-pass $\sqrt{\bar p(1 - \bar p)/198} = 1.32$ points. Suppose Epoch's 1.19 points is this population error bar. Substituting $\sigma_w^2 = \bar p(1 - \bar p) - \sigma_b^2$ and solving gives $\sigma_b^2 = \big(16 \cdot 198 \cdot 0.0119^2 - 0.964 \cdot 0.036\big)/15 = 0.0276$. A single answer's variance is $0.964 \cdot 0.036 = 0.0347$, so about 80 percent of it comes from which question was drawn. Grading every question a thousand times instead of 16 would lower the error bar only to its floor, $\sqrt{0.0276/198} = 1.18$ points.

The cell below checks the theorem by simulation. It draws 30 problems with pass rates from a Beta distribution that puts most of its mass near 0 and 1, the shape a strong reasoning model shows on contest problems. Then it compares both formulas with the actual spread of the score, once redrawing only the samples (same problems) and once redrawing problems and samples together.

Both formulas match their simulations. The fixed-question error bar halves every time $K$ quadruples, from 5.7 points at one sample to 0.7 at 64. The population error bar falls from 8.6 to 6.3 points and stops near its floor of 6.2 points. Sixty-four samples per problem make the score a precise measurement of how this model does on this exam. They leave it an imprecise measurement of how good the model is at math. Only more problems shrink the second, which is why evaluation suites that care about math skill pool several years of AIME, or move to larger sets such as MATH-500 or FrontierMath.

import numpy as np

def se_fixed_questions(S):
    raise NotImplementedError

def se_population(S):
    raise NotImplementedError

Comparing many models on one benchmark

A leaderboard is many comparisons at once, and two habits make most of them wrong. The first is reading overlapping error bars as "no difference" and separated ones as "a difference". The second is forgetting that every model answered the same items, so the right comparison is paired. Here is one pair by hand.

Why overlapping intervals are the wrong test

Many leaderboards draw a 95% interval around each model and invite you to check for overlap. That rule is far stricter than it looks. Take two independent scores with standard errors of 1 point each, 3 points apart. Each interval reaches 1.96 points on either side, so they overlap, since $3 < 3.92$. Yet the gap has standard error $\sqrt 2 = 1.41$ points, so it is $3/1.41 = 2.1$ standard errors from zero, and a correct test calls it significant at the 5% level.

In plain words, two intervals can overlap even when the difference is clearly real. On a shared benchmark, where models get the same easy questions right and the same hard ones wrong, their scores are positively correlated, and the intervals overlap far more often than the models are really tied. The overlap rule throws away most of the evidence. An interviewer who shows you a leaderboard with error bars is often checking whether you know this.

The models tied with the leader

With $m$ models, the question a leaderboard reader usually cares about is "which models are statistically tied with the leader?". That takes $m - 1$ paired tests, the leader against each other model. Because they are run together, they need a correction for multiple tests. Holm's step-down method (§0.11) is the default. It sorts the p-values from smallest to largest and multiplies the smallest by the number of tests, the next by one fewer, and so on. Then it replaces each result by the largest one so far, so the adjusted p-values never decrease down the sorted list. It controls the same family-wise error rate as the Bonferroni correction, which multiplies every p-value by the number of tests, and it is never less powerful.

Two cautions belong with this definition. First, the leader was chosen because it scored highest, and the highest of several noisy scores is biased upward, the selection effect that §22.1 measures. Its gaps to the others are therefore inflated, the tests reject a little too easily, and the tied set errs on the small side. A bootstrap over items that picks the leader again in every resample, or the rank intervals of §22.7, accounts for the selection. Second, "tied" means the data cannot separate the models, not that the models are equal.

Here are six models on a shared benchmark of 1,000 items. Item difficulty is spread widely, so the models agree on most items and pairing has a lot to remove.

The interval-overlap rule calls models 1 to 4 tied with the leader. Paired tests with Holm's correction separate models 2 and 4, at gaps of 4.0 and 5.5 points, and leave models 1 and 3 in the top tier. Every Wilson interval here reaches about 3 points on each side, while the paired standard errors are about 1.3 points, because two models on this benchmark agree on most of the 1,000 items.

The widget lets you set six scores, a benchmark size, and the discordance, the share of items on which two models disagree. Each model gets two verdicts against the leader, one from the paired test with Holm's correction and one from the rule that the two intervals must not overlap. The shaded band, just left of the leader's score, marks the gaps too small for the paired test to call real.

import numpy as np
from math import erfc, sqrt

def tied_with_leader(R, alpha=0.05):
    R = np.asarray(R, dtype=float)
    acc = R.mean(axis=1)
    return {"leader": int(np.argmax(acc)), "se": None, "p_holm": None, "tied": [int(np.argmax(acc))]}

Multiple choice, cloze or letters

Most knowledge and reasoning benchmarks are multiple choice, and the MMLU story showed that there are several ways to read a model's answer to one. You can let the model write and parse what it wrote. You can score each option's text by how likely the model finds it. Or you can score the option letters. Generation comes first here, because it is how chat and reasoning models are scored.

Answer extraction for models that write

Chat and reasoning models are rarely scored by letter probabilities. They write an answer, often after a long chain of thought, and a parser pulls out the choice. The parser is part of the grader $g$, and a careless one is a large source of protocol error.

Each naive rule has a failure. Taking the first letter that appears picks up "Option A looks tempting" in the reasoning. Matching letters without regard to case turns the article "a" into option A. Accepting any letter scores "E" on a four-option question. And a model that hedges with "A or B" gets credit whenever the parser happens to read the right half. Harnesses differ in exactly these rules, which is one reason two generative implementations of the same benchmark can disagree. Checking a final numeric or symbolic answer in math is the subject of §18.3.

import re

def extract_choice(text, letters="ABCD"):
    m = re.search(r"[A-D]", text)          # first capital letter that could be an option
    return m.group(0) if m else None

Cloze or letters, for models scored by likelihood

A base model can answer without generating anything. We can score each option by how likely the model finds it as a continuation of the question and pick the best, as §6.6 did. That section found that the details matter. The raw sum of the option's token log-probabilities favors short options. Dividing by the number of tokens or characters favors long ones. A third rule subtracts each option's log-probability in a neutral context, which asks how much the question raised its likelihood. Scoring the option text is only one of two ways to pose a multiple-choice question, and the choice between the two ways moves scores more than any of these normalizations does.

The two formulations pose different questions, and models learn to answer the second one late in training. Gu et al. (2024) built a documented standard for evaluating base models on this observation, called OLMES.

OLMES therefore evaluates every model both ways and reports the better of the two. The practical rule is to use CF to track small models and early checkpoints, and MCF for models that have learned the format. Madaan et al. measured how steadily a score rises with monotonicity, the rank correlation of the score with the training step. It was 0.09 for lettered MMLU and 0.95 for cloze.

The cell simulates a family of six model sizes. Each model has a belief about which option is right, and the belief sharpens with size. In CF that belief competes with how likely each option string is anyway, because of its length and wording, an effect §6.6 called surface form competition. In MCF the belief passes through a format skill that small models lack, and a fixed preference for some letters is added.

The crossover appears on its own. The two smallest models answer A every time in the lettered format, so they score exactly the share of items whose key happens to be A, 25.9% here. CF already sees their knowledge, at 32% and 41%. From size 3 on, MCF wins, reaching 99% while CF tops out at 92%, held back by the options' surface likelihood. And size 3 still answers A 44% of the time on a key where A is right 26% of the time.

Letter-position bias, and how to remove it

That last number is real at scale. Zheng et al. (2023) sampled 1,000 MMLU questions whose right answers were exactly 250 each of A, B, C, and D. Llama-30B chose A, B, C, and D 34.6%, 27.3%, 22.3%, and 15.8% of the time. GPT-3.5-turbo chose them 22.5%, 25.6%, 32.3%, and 19.6% of the time. Across 20 models they traced the bias mostly to the label tokens themselves, a prior the model puts on "A" before reading the options. Moving the right answer to a different letter changes the score, so the score measures the key's letter distribution as well as the model.

Their fix, called PriDe for prior debiasing, estimates that prior from a small calibration set and divides it out of every prediction. The trick for reading off the prior is to show each calibration item in every cyclic order of its options. The options rotate, so each one visits each letter once. Averaged over the rotations, the content of the options is the same at every letter, so whatever preference is left is the prior. The theorem makes this exact under a simple model of how the letter probabilities arise.

In plain words, rotate the options through every letter, average the model's log-probabilities letter by letter, and what survives the averaging is the model's taste for letters. Then subtract that taste, in log space, from every later prediction.

The cell runs PriDe on the size-3 model from the last cell, the one that over-answers A. It shows 5% of the items in all four orders to estimate the prior, then debiases the rest.

The estimate recovers the planted prior to three decimals, because this toy follows the factored model exactly. Debiasing raises accuracy from 74% to 88%, and the share of A answers falls to the key's 26%. Real models are only roughly factored, and the gains are smaller. Zheng et al. measured this on MMLU with zero-shot prompts, averaged over their 20 models. PriDe with the prior estimated on 5% of the questions costs 1.15 times as much as plain scoring. It raised accuracy by 1.2 points and evened out recall across letters. Showing every question in all four of its cyclic orders and averaging, at 4 times the cost, raised accuracy by 4.9 points. The simplest diagnostic costs nothing. Print the model's distribution of answers over letters next to the key's.

import numpy as np

def letter_prior(L_cal):
    K = np.shape(L_cal)[-1]
    return np.full(K, 1.0 / K)

def debias_predict(logits, prior):
    return np.argmax(np.asarray(logits), axis=1)     # placeholder, no debiasing

Prompt sensitivity

The prompt is part of the protocol, and small changes to it move scores a lot. §6.8 showed how fragile few-shot prompts are. Reordering four fixed examples moved GPT-3 2.7B on a sentiment task from 54% to 93% (Zhao et al. 2021). Formatting changes that keep the meaning, such as a different separator or casing, moved LLaMA-2-13B by up to 76 points (Sclar et al. 2024). The best format for one model was a poor guide to the best for another. Contextual calibration, which divides out the answer probabilities the model gives to a content-free input such as "N/A", removed much of the order effect. Benchmarks inherit all of this, and it does not go away with scale.

The statistics are the two-estimand story again. A single prompt template is one draw from the set of reasonable templates. A score under one template is a fixed-template estimate, and its error bar ignores the template. If the claim is about the model in general, treat the template as random. Score each model under $P$ templates and report the mean and the spread across them. Suppose each template shifts the score by a random amount with variance $\sigma_{\text{tmpl}}^2$, independent of the item effects. Then the mean over $P$ templates carries an extra variance of $\sigma_{\text{tmpl}}^2/P$, a term that no number of items or samples can shrink.

And because the best template differs between models, a comparison under one template can flip under another. The widget shows three simulated models under 27 prompt formats, every combination of 3 separators, 3 casings, and 3 pairs of field names, such as Question and Answer or Q and A. Lettered means the multiple-choice formulation, small and large set the models' size, and Play steps through the formats one by one.

The defensible practice follows. Fix and publish one protocol per benchmark, as OLMES does, so that numbers are comparable at all. Report results under several templates, or the spread across them, when the claim is about the model rather than the protocol. And never compare a number from one evaluation harness with a number from another. An interviewer who asks why two papers disagree on the same benchmark wants the protocol named first.

Noise from training itself, seeds and checkpoints

Every error bar so far treats the model as fixed and the questions or samples as random. A pretraining team uses benchmarks differently. It trains two small models that differ in one choice, say the data mixture, and asks which choice to scale up. Such a pair of runs is called an ablation. Now the training run is random too. A different initialization seed or data order gives a different model. Even consecutive checkpoints of one run differ, because the last few hundred optimizer steps wobble the weights. A difference between two ablations means something only if it is larger than this training noise.

Heineman et al. (2025) turned this into a property of a benchmark. A benchmark is useful for small-scale decisions if it spreads models that really differ far apart, which they call signal, and keeps one model's score steady from checkpoint to checkpoint, which keeps the noise low.

Across the benchmarks Heineman et al. studied, signal alone and noise alone barely predicted decision accuracy, while their ratio did, with a correlation of $R = 0.791$ ($R^2 = 0.626$). Their fix rests on why accuracy is noisy, an argument §9.6 made about sudden jumps in scaling curves. Accuracy thresholds a continuous quantity, the margin by which the right answer beats the others. A model that wins an item by a hair scores the same as one that wins by a mile, and a small wobble of the weights flips the items that sit near the threshold. A continuous metric keeps the size of the margin and changes smoothly. One such metric is bits per byte of the correct answer (§6.6), the model's negative log-probability of the right answer per byte of its text, measured in bits.

The cell builds 25 small models that differ slightly in skill and gives each a fixed per-item margin. It adds a checkpoint-to-checkpoint wobble to every margin and scores the last five checkpoints two ways, with accuracy and with the mean negative log-probability of the correct answer.

The continuous loss has more than twice the SNR of accuracy on the same items and the same wobble, 48.3 against 21.8. It orders 96% of the model pairs as their noise-free scores would, against 91% for accuracy. Averaging five checkpoints lifts both, to 98% and 94%. The model is a toy, but the mechanism is the one Heineman et al. measured at scale. In practice, use continuous metrics for decisions between small models, and save accuracy for reporting what a finished model can do. An interviewer who hears "the ablation won by half a point" will ask what the noise was.

import numpy as np

def relative_dispersion(scores):
    raise NotImplementedError

def relative_std(checkpoints):
    raise NotImplementedError

How few questions can we get away with

Running every benchmark on every checkpoint is expensive, so it is natural to ask how many items a measurement really needs. The answer depends on whether some items carry more information than others. A model from psychometrics, the science of designing tests for people, makes that precise. It is called item response theory (IRT).

The idea is that each model has one number, its ability, and each item has two, its difficulty and how sharply it separates weak models from strong ones. The chance of a right answer rises with ability along an S-shaped curve centered at the item's difficulty.

The 2PL is logistic regression (§0.4) in which the logit $z_{\ell i} = \alpha_i(\vartheta_\ell - \beta_i)$ multiplies unknowns together. It is fitted by maximizing the Bernoulli log-likelihood of all the grades with gradient ascent. Every gradient is a weighted sum of residuals, the grade minus its predicted probability.

Once the item parameters are fitted on a population of models, a new model needs only a few well-chosen items. Fit its ability from those, and predict its full-benchmark accuracy as $\frac1n\sum_i\sigma(\alpha_i(\hat\vartheta - \beta_i))$. That prediction uses what the population revealed about every item the new model did not answer.

That is the idea behind tinyBenchmarks (Polo et al. 2024). They fitted the item parameters on hundreds of models from public leaderboards. Then they estimated a new model's accuracy on all of MMLU, about 14,000 questions, from 100 curated items, to within about 2% on average.

The counterpoint came from the setting where cheap evaluation matters most. Madaan et al. (2024) applied the same 100-item sets to their ten 7B seed runs. On ARC-Challenge the seed standard deviation rose from 0.80 points on the full set to 1.80 on the 100 items. It was 1.86 with IRT++, the fuller estimator that blends the IRT prediction with the observed mean of the answered items. On HellaSwag it rose from 0.21 to 2.06 (2.42 with IRT++). And the plain IRT estimate put ARC-Challenge at 46.2% against a full-set score of 39.7%, 6.5 points too high. A small set can match the average of a population of finished models and still be too noisy to separate two checkpoints. IRT also assumes the new model behaves like the population it was fitted on, which is least true of the new models you most want to measure.

import numpy as np

def fit_2pl(R, n_iter=400, lr=0.05, lam=0.01):
    raise NotImplementedError

def ability_from_anchors(r, alpha, beta, n_iter=50):
    raise NotImplementedError

The same fit in PyTorch, and a real harness

At scale, a 2PL model is fitted the way any logistic model is, with autograd and an optimizer over the whole response matrix. Writing $\alpha = e^{a}$ keeps discriminations positive without clipping. The block fits 300 simulated models on 150 items and checks how well it recovers the planted parameters.

Difficulties and abilities come back almost exactly, and discriminations noticeably less well, with correlations of 0.98, 0.98, and 0.90. Each item's slope is estimated from how 300 models fall around its difficulty. With a few dozen models the slopes are mostly noise, which is one reason IRT estimates from small populations of models disappoint.

The other real-world tool is an evaluation harness, a program that turns a model and a task name into a score under a fixed protocol. EleutherAI's lm-evaluation-harness writes the protocol in a task file. The three commands below differ only in formulation and chat template, and that is enough to give three different numbers called "MMLU". The harness's maintainers recommend sharing the exact task configuration and the logged samples with every reported number, so that anyone can see which of these was run (Biderman et al. 2024).

# The README form is lm_eval; recent harness versions also accept "lm-eval run" with the same flags.
# Lettered (MCF): log-probability of " A", " B", " C", " D" after "Answer:"
lm_eval --model hf --model_args pretrained=meta-llama/Llama-3.1-8B \
    --tasks mmlu --num_fewshot 5 --output_path results/mcf --log_samples

# Cloze (CF): log-probability of each option's text after "Question: ... Answer:"
lm_eval --model hf --model_args pretrained=meta-llama/Llama-3.1-8B \
    --tasks mmlu_continuation --num_fewshot 5 --output_path results/cf --log_samples

# Generative, in the chat template: the model writes, and its first line is matched against the letter
lm_eval --model hf --model_args pretrained=meta-llama/Llama-3.1-8B-Instruct \
    --tasks mmlu_generative --num_fewshot 5 --apply_chat_template \
    --output_path results/gen --log_samples

Choosing what to report

The error bars of this section rest on assumptions that real benchmarks break. Items that come in groups, such as several questions on one passage or several tasks from one repository, are correlated. They need the clustered standard errors of §14.7, which treat each group as one unit. A judge as grader adds its own bias and noise, which §22.4 measures. Agents add flaky environments and scaffolds (§22.5). Generative models of images and speech need metrics of their own. Image and video generation uses the Fréchet inception distance (FID), which compares feature statistics of generated and real images. Speech recognition uses word error rate, with careful normalization of the text before counting errors. And every interval here surrounds the wrong number if the model saw the test during training, the leakage term the next section takes on.

Read more

• Liang et al. (2022). Holistic Evaluation of Language Models. HELM, the framework of scenarios crossed with metrics, run under one standardized protocol for every model.
• Petrov et al. (2025). Proof or Bluff? Evaluating LLMs on 2025 USA Math Olympiad. Expert graders score full proofs, and the gap to final-answer benchmarks appears.
• Miller (2024). Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations. The super-population view of a benchmark, with formulas for clustered, paired, and resampled evaluations and for planning sample sizes.
• Madaan et al. (2024). Quantifying Variance in Evaluation Benchmarks. Ten 7B seed runs measure how much benchmarks move with nothing but the seed, and why cloze formats and continuous metrics help.
• Heineman et al. (2025). Signal and Noise: A Framework for Reducing Uncertainty in Language Model Evaluation. Defines signal, noise, and SNR across 30 benchmarks and 375 models, and shows that bits per byte and checkpoint averaging improve decisions.
• Gu et al. (2024). OLMES: A Standard for Language Model Evaluations. A fully documented evaluation protocol, with the evidence for cloze versus lettered formulations and for each normalization choice.
• Zheng et al. (2023). Large Language Models Are Not Robust Multiple Choice Selectors. Measures letter-position bias across 20 models and introduces PriDe.
• Sclar et al. (2024). Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting. FormatSpread, and the 76-point spread from formatting alone.
• Polo et al. (2024). tinyBenchmarks: evaluating LLMs with fewer examples. Item response theory for estimating full-benchmark scores from about 100 items.
• Biderman et al. (2024). Lessons from the Trenches on Reproducible Evaluation of Language Models. The lm-evaluation-harness maintainers on why evaluation numbers fail to reproduce and what to report.

Watch next

▶

▶