A fraud model reports 99.5% accuracy on a test set where 0.5% of transactions are fraud. That sounds excellent until you notice that a model that never says "fraud" also scores 99.5%. A second team replaces it with a detector that catches 90% of the fraud, and its accuracy goes down. Meanwhile, across the hall, a new language model scores 71.2% on a 500-question benchmark against the old model's 70.4%, and someone wants to announce the improvement.

Both groups are about to fool themselves, in the two ways this section is about. The fraud team is judging with the wrong number. The language-model team has no idea how much its number would move if the test set were drawn again. Measuring a model honestly takes both a metric that matches the question and an error bar that says how far to trust it, and the statistics behind the error bar decide whether one model really beats another.

This section builds the metrics first (the confusion matrix, precision and recall, ROC and precision-recall curves, and calibration), then the statistics (confidence intervals, paired comparisons, hypothesis tests, multiple comparisons, and power), and it ends with the ways each is misused in papers and model evaluations. Later chapters use all of it, for retrieval metrics in §7.5, evaluating fine-tuned models in §14.7, reward model accuracy in §16.3, and safety evaluations in §19.4.

The confusion matrix

A binary classifier makes four kinds of decisions, and the confusion matrix counts each. A true positive (TP) is a positive example predicted positive, a false positive (FP) a negative predicted positive, a false negative (FN) a positive predicted negative, and a true negative (TN) a negative predicted negative. With rows for the truth and columns for the prediction, the detector above gives

Every common metric is a ratio of these four counts, and the useful ones divide by a row or a column rather than by everything.

• Recall, also called the true positive rate (TPR) or sensitivity, is $\text{TP}/(\text{TP} + \text{FN})$, the fraction of real positives the model catches. Here $434/485 = 0.895$.
• Precision, or positive predictive value, is $\text{TP}/(\text{TP} + \text{FP})$, the fraction of alarms that are real. Here $434/1431 = 0.303$.
• The false positive rate (FPR) is $\text{FP}/(\text{FP} + \text{TN})$, the fraction of negatives that raise an alarm. Here 0.010. Its complement, $\text{TN}/(\text{TN} + \text{FP})$, is the specificity.
• Accuracy is $(\text{TP} + \text{TN})/N$, which here is 0.9895, worse than the 0.9951 of never predicting fraud.

Accuracy is dominated by whichever class is common, so on imbalanced data it mostly measures the prevalence. The detector catches nine frauds in ten, yet loses on accuracy because its 997 false alarms outnumber the 485 frauds in the whole set. And notice what precision did. The detector has a false positive rate of only 1%, but seven of every ten of its alarms are false.

Precision depends on the base rate

That last fact is Bayes rule from §0.4, the same arithmetic as the 99%-accurate disease test. Write $\pi = \mathbb{P}(y = 1)$ for the prevalence of positives. Precision is the probability that an example is positive given an alarm, so

In plain words, among all alarms, the true ones come from the positives that are caught ($\text{TPR}\cdot\pi$) and the false ones from the negatives that slip through ($\text{FPR}\cdot(1-\pi)$). Recall and FPR are properties of the classifier alone, measured within one true class. Precision mixes in the prevalence. The same detector, unchanged, is excellent at one base rate and nearly useless at another.

At a prevalence of 0.05%, which is realistic for many abuse and safety problems, every true alarm comes with 22 false ones. This is why the safety classifiers of §19.7 need false positive rates far below 1%, and why a precision number means nothing without the base rate of the set it was measured on.

F1, and averaging over classes

When one number must summarize both errors, the usual choice is the F1 score, the harmonic mean of precision $P$ and recall $R$,

The second form follows by substituting the definitions and clearing denominators. In plain words, F1 is high only when both precision and recall are high, because a harmonic mean is pulled toward the smaller value. A model with $P = 1$ and $R = 0.1$ gets $F_1 = 0.18$, not the 0.55 an ordinary average would give. The detector above has $F_1 = 868/(868 + 997 + 51) = 0.453$. The more general $F_\beta = (1 + \beta^2)PR/(\beta^2 P + R)$ weights recall $\beta$ times as much as precision ($F_2$ is common when misses are costlier than false alarms). Notice that F1 never looks at TN, which is why it suits problems where the negatives are a vast, uninteresting background.

With $K$ classes, compute precision, recall, and F1 for each class $c$ by treating "$c$" as positive and everything else as negative, then average. Macro-averaging takes the plain mean over classes, so every class counts equally however rare. Micro-averaging adds up TP, FP, and FN over all classes first and computes one F1 from the totals. For single-label classification every wrong prediction is one false positive (for the predicted class) and one false negative (for the true class), so micro precision, micro recall, and micro F1 all equal plain accuracy. Report macro F1 when rare classes matter, and say which average you used, because they can differ enormously.

import numpy as np

def confusion_matrix(y_true, y_pred, K):
    raise NotImplementedError

def prf(C):
    raise NotImplementedError

Thresholds, and the curves that sweep them

Most classifiers do not output a decision. They output a score $s$ (a probability, a logit, a distance), and a decision comes from a threshold $t$, predicting positive when $s \ge t$. Every threshold gives a different confusion matrix. Lower it and you catch more positives at the price of more false alarms. To judge a scorer rather than one decision, sweep the threshold through every value and plot what happens.

The ROC curve (receiver operating characteristic, a name from radar engineering) plots TPR against FPR. It starts at $(0, 0)$, where the threshold is above every score and nothing is flagged, and ends at $(1, 1)$, where everything is flagged. A perfect scorer goes straight up to $(0, 1)$, and a random one follows the diagonal. The precision-recall curve plots precision against recall over the same sweep.

The area under the ROC curve, the ROC AUC, has a beautiful meaning. It is the probability that a randomly chosen positive scores higher than a randomly chosen negative,

where $s^+$ and $s^-$ are the scores of a random positive and a random negative, and ties count half. In plain words, AUC measures ranking. It asks how often the model puts a positive above a negative, ignoring any threshold and any calibration. An AUC of 0.5 is chance, and 1 is a perfect ranking.

For the precision-recall curve the standard summary is average precision (AP), the precision at each threshold weighted by how much recall increased there, $\text{AP} = \sum_k (R_k - R_{k-1})P_k$ over the distinct thresholds $k$ from the highest score down. Do not draw straight lines between PR points and integrate under them. Unlike ROC space, precision does not vary linearly between two operating points, and linear interpolation overstates the area (Davis & Goadrich, 2006).

Why precision-recall beats ROC under imbalance

TPR and FPR are each computed within one true class, so neither depends on how many positives there are. The ROC curve, and its AUC, are therefore the same at any prevalence. That sounds like a virtue, and for comparing ranking quality across data sets it is. But when positives are rare, the ROC curve hides exactly what you care about. An FPR of 1% is a point near the left edge of the ROC plot, yet at 0.5% prevalence it means two false alarms for every real one. Precision feels that, and the ROC curve cannot.

The cell draws the same scorer (positives score $\mathcal{N}(2, 1)$, negatives $\mathcal{N}(0, 1)$) at three prevalences. The theoretical ROC AUC is $\Phi(2/\sqrt2) = 0.921$ at every prevalence, where $\Phi$ is the standard normal CDF, since $s^+ - s^-$ is $\mathcal{N}(2, 2)$.

The three ROC curves lie on top of each other, with AUCs between 0.91 and 0.93 (the theoretical 0.921 plus sampling noise from 500 positives). The average precision falls from 0.93 to 0.50 to 0.19, and that collapse is the real story, since at 0.5% prevalence no threshold gets you high precision and decent recall at once. Saito & Rehmsmeier (2015) make this case at length. A random scorer has an expected AP equal to the prevalence, so AP should always be read against that baseline, while a random ROC AUC is always 0.5.

Choosing the threshold

A curve is not a decision. To deploy, you pick one threshold, and the right one depends on what each error costs. If a false positive costs $c_{\text{FP}}$ and a false negative $c_{\text{FN}}$, choose the threshold that minimizes $c_{\text{FP}}\cdot\text{FP} + c_{\text{FN}}\cdot\text{FN}$ on a validation set. Many applications state the constraint directly instead, as recall at a fixed false positive rate, or precision at a fixed recall. Security work, for example, judges membership-inference attacks by their true positive rate at a false positive rate of 0.1% or lower, because an attack that is right on average but cannot confidently identify anyone is no threat (Carlini et al., 2022).

Choose thresholds on validation data, never on the test set. Tuning a threshold to the test set is a quiet form of training on it, and the reported precision and recall will not hold up.

import numpy as np
from scipy.stats import rankdata

def roc_auc(y, s):
    raise NotImplementedError

def average_precision(y, s):
    raise NotImplementedError

Calibration

A weather forecaster who says "70% chance of rain" is calibrated if it rains on about 70% of the days they say it. A classifier's probabilities can be judged the same way. Formally, a model that outputs a confidence $\hat p$ for its prediction is calibrated if

In plain words, among all the predictions made with confidence 0.8, 80% should be right. Calibration is separate from accuracy. A model can rank perfectly and still be overconfident, and a model that always predicts the base rate is perfectly calibrated and useless. It matters whenever a probability is used as a probability, to decide when to abstain, to combine with costs, to route a hard question to a human, or to tell a user how sure the model is.

To check it, sort predictions into $M$ bins by confidence (here $M$ is the number of bins). For each bin $b$, compare the average confidence $\operatorname{conf}(b)$ with the fraction correct $\operatorname{acc}(b)$. Plotting accuracy against confidence gives a reliability diagram, which for a calibrated model lies on the diagonal. The expected calibration error summarizes it as the bin-size-weighted gap,

where $n_b$ is the number of predictions in bin $b$ (Naeini et al., 2015; Guo et al., 2017). For multi-class models the usual convention uses the top-label confidence, the largest softmax probability, and asks whether the predicted class is correct.

Guo et al. (2017) found that modern deep networks are often overconfident, much more so than older, smaller ones, and that a one-parameter fix works remarkably well. Temperature scaling divides the logits by a single scalar $\tau$, fitted on a validation set by minimizing the negative log-likelihood. Dividing all logits by the same positive number never changes their order, so the predictions and the accuracy stay exactly the same, and only the confidences move. (This is the same $\tau$ as the sampling temperature of §0.4, used here to repair probabilities rather than to sample.)

The cell builds a 10-class model whose logits are 2.5 times too large, so it is badly overconfident, and fixes it.

Temperature scaling recovers $\tau \approx 2.53$, close to the true 2.5, drops the ECE from 0.24 to about 0.02, and improves the log-likelihood and the Brier score, while the accuracy stays at 0.544 to the last digit. The cell below draws the two reliability diagrams.

Calibration is not discrimination

"Are the confidences any good" hides two separate questions. Calibration asks whether confidences match frequencies. Discrimination asks whether higher confidence goes with being right, so that the confidence can tell the model's right answers from its wrong ones. Discrimination is a ranking question, so it is measured by the AUROC of the confidence used as a score for correctness, the probability that a randomly chosen right answer gets a higher confidence than a randomly chosen wrong one. That is exactly the ROC AUC of the previous section, with "correct" as the positive class. A model can be calibrated without discriminating, and discriminate well while being badly calibrated.

Temperature scaling cuts the ECE more than tenfold and leaves the AUROC almost where it was (it moves slightly, because dividing every logit vector by the same $\tau$ can reorder the top probabilities of two different examples). The constant-confidence model is perfectly calibrated and has an AUROC of exactly 0.5, no better than chance at telling right from wrong. Report both numbers. Calibration says whether a confidence of 0.8 can be read as 80%, discrimination says whether the confidence is worth reading at all, and a model that is to abstain when unsure needs both.

What ECE does not tell you

ECE is useful and easy to misread. Three cautions.

• It can be gamed. A model that outputs its overall accuracy as the confidence for every prediction has an ECE of exactly zero and conveys nothing about which predictions are right. ECE should always be reported next to accuracy or a proper score.
• It is a biased estimate, and it depends on the bins. Within each bin, the accuracy is measured from a finite sample, and its sampling noise never cancels inside the absolute value, so even a perfectly calibrated model gets a positive ECE. The bias grows with the number of bins and shrinks with the number of predictions, roughly like $\sqrt{M/n}$. Too few bins have the opposite problem, averaging real miscalibration away, and for a model whose confidences vary continuously the binned estimate can understate the true calibration error (Kumar, Liang & Ma, 2019). Equal-mass bins, which put the same number of predictions in each, avoid nearly empty bins when confidences pile up near 1, and Nixon et al. (2019) recommend them. Always state the number and kind of bins, and compare models only at the same number of predictions.
• It sees only the top label. A model can be calibrated on its top choice and badly miscalibrated on the rest of the distribution.

A proper scoring rule avoids the gaming problem. It is a score whose expectation is optimized only by reporting the true probability. The Brier score $(\hat p - y)^2$ is one. If $y \sim \text{Bernoulli}(p)$, its expectation for a report $q$ is $\mathbb{E}[(q - y)^2] = q^2 - 2qp + p = (q - p)^2 + p(1 - p)$, minimized exactly at $q = p$. The log loss (the negative log-likelihood) is another. Both reward calibration and sharpness together, which is why calibration studies report NLL or Brier alongside ECE.

The true calibration error of this model is zero. With 500 predictions and 50 bins the estimate says 0.09 to 0.10, worse than many published "miscalibrated" models, and it falls only as the evaluation grows. Equal-mass bins do not remove the bias, which comes from finite samples, but they keep every bin populated. So a reported ECE is only meaningful next to its number of predictions and bins, and small differences in ECE between models are usually noise.

For language models these questions are live. Kadavath et al. (2022) found that large pretrained models are well calibrated on multiple-choice questions when the question is posed in the right format, and the GPT-4 technical report (OpenAI, 2023) showed calibration on a multiple-choice benchmark getting markedly worse after post-training. Calibration and abstention for language models are the subject of §19.8.

import numpy as np

def ece(conf, correct, n_bins=15):
    raise NotImplementedError

How precise is a measured number

Now the second problem from the opening. A model answers 500 benchmark questions and gets 352 right, $\hat p = 0.704$. If we drew another 500 questions of the same kind, we would get a somewhat different number. How different?

Each question is a Bernoulli trial with success probability $p$, the model's true accuracy on questions like these. So the number correct $k$ is binomial, and by §0.4 the estimate $\hat p = k/n$ has standard error $\sqrt{p(1-p)/n}$. By the central limit theorem, $\hat p$ is approximately $\mathcal{N}(p, p(1-p)/n)$, so with probability about 95% it falls within $1.96$ standard errors of $p$. Turning that around gives the normal (or Wald) interval,

For 352 of 500, that is $0.704 \pm 0.040$. The second team's 70.4% and 71.2% are 0.8 points apart, with error bars of four points each. Nothing has been shown yet.

What does "95% confidence interval" mean? It is a statement about the procedure, not about this one interval. If you repeated the whole experiment many times and built an interval each time, 95% of those intervals would contain the true $p$. This one interval either contains it or does not. Saying "there is a 95% probability that $p$ is in $[0.664, 0.744]$" is the common misreading, and it is only justified in a Bayesian analysis with a prior.

When the normal interval fails, and the Wilson interval

The Wald interval has two well-known failures. With $k = 0$ (no harmful outputs in 50 red-team prompts, say) it gives $[0, 0]$, claiming certainty from 50 trials. And near 0 or 1 its true coverage falls well short of 95% even for large $n$. Brown, Cai & DasGupta (2001) showed its coverage is erratic and often poor, and recommend the Wilson interval instead. Wilson's idea (1927) is to use the standard error at the hypothesized $p$, not at $\hat p$, and collect every $p$ that the data do not reject,

In plain words, the Wilson interval is centered not at $\hat p$ but pulled toward $1/2$, by the equivalent of adding $z^2/2 \approx 2$ successes and 2 failures, and it never leaves $[0, 1]$. For 0 successes in 50 it gives $[0, 0.071]$. A handy companion is the rule of three. With zero events in $n$ trials, the exact one-sided 95% upper bound solves $(1 - p)^n = 0.05$, so $p = 1 - 0.05^{1/n} \approx -\ln(0.05)/n \approx 3/n$, here 0.06. Zero failures in 300 tests still allows a failure rate of 1%.

How well do the two intervals keep their promise? The coverage of an interval method, the probability that it contains the true $p$, can be computed exactly by summing the binomial probabilities of every outcome $k$ whose interval contains $p$.

At $p = 0.5$ with 1,000 trials both are fine. For rare events, which is where safety evaluations live, the Wald interval covers the truth only 87% of the time, while Wilson stays near 95%. The widget lets you watch 100 simulated evaluations at once.

import numpy as np
from scipy.stats import norm

def wilson_interval(k, n, level=0.95):
    raise NotImplementedError

The bootstrap, for numbers without a formula

Accuracy has a formula for its standard error. F1, AUC, ECE, a median latency, or the pass rate of a judge do not, or the formula is messy. The bootstrap (Efron, 1979) estimates the sampling distribution of any statistic by treating the test set as a stand-in for the population. Resample $n$ items from the test set with replacement, recompute the statistic, repeat a few thousand times, and read off the spread. The percentile interval takes the 2.5th and 97.5th percentiles of the resampled values as a 95% interval.

Even with 400 examples, an F1 of 0.64 is uncertain by about nine points either way. The bootstrap has its own rules. Resample the unit that was sampled independently. If each question was asked five times, or several questions share one passage, resample questions or passages, not individual answers, otherwise the interval comes out far too narrow (Miller, 2024, calls the fix clustered standard errors). It is unreliable for statistics driven by a few extreme values, such as a maximum, and for very small test sets. And it measures only the noise of the test sample, not the variation from retraining with another seed, which needs separate runs.

Comparing two models

Back to the second team, with 70.4% against 71.2%. The key fact is that both models answered the same questions, and that changes the arithmetic completely.

Questions differ in difficulty, so two models' answers are correlated. Both get the easy ones right and both miss the hardest. Let $a_i, b_i \in \{0, 1\}$ say whether models A and B got question $i$ right, and let $\Delta_i = b_i - a_i$ be the per-question difference, with mean $\bar\Delta$ over the $n$ questions. Then

In plain words, the covariance term subtracts the shared difficulty. Treating the two accuracies as independent, as if the models had answered different questions, ignores it and overstates the noise. The paired standard error is just the standard deviation of the $\Delta_i$ divided by $\sqrt n$.

There is an even more direct way to see it. Every $\Delta_i$ is $-1$, $0$, or $+1$, and the questions both models got right or both got wrong contribute zeros. Only the discordant questions carry information about which model is better. Let $n_{10}$ count questions A got right and B got wrong, and $n_{01}$ the reverse. If the models are equally good, each discordant question is equally likely to go either way, so conditional on the $n_{10} + n_{01}$ discordant questions, $n_{01} \sim \text{Binomial}(n_{10} + n_{01}, \tfrac12)$. That is McNemar's test (McNemar, 1947), and the exact version computes its p-value from this binomial, with no approximation at all. Dietterich (1998) recommends it for comparing two classifiers on one test set.

The same data give two conclusions depending on the arithmetic. Treated as unpaired, a 3.7-point gain has $z = 1.88$, short of significance. Paired, the standard error shrinks by a quarter, $z = 2.44$, McNemar's exact p-value is 0.018, and the paired bootstrap interval excludes zero. Pairing is free precision, and it is the default way to compare models on a shared benchmark. With language models, where answers are sampled and graded by judges, the same idea extends to several samples per question and to clustered questions, which §14.7 works through.

import numpy as np
from scipy.stats import binom

def paired_bootstrap_ci(a, b, n_boot, rng, level=0.95):
    raise NotImplementedError

def mcnemar_exact(a, b):
    raise NotImplementedError

Hypothesis tests and p-values

McNemar's test is one instance of a general recipe. State a null hypothesis $H_0$, the boring explanation (the models are equally good). Choose a test statistic that tends to be large when $H_0$ is false (the paired $z$, or $n_{01}$). The p-value is the probability, computed assuming $H_0$ is true, of a statistic at least as extreme as the one observed. If it falls below a significance level $\alpha$ chosen in advance (often 0.05), reject $H_0$. Rejecting a true null is a type I error, which happens with probability $\alpha$ by construction. Failing to reject a false one is a type II error.

Tests and intervals are two views of the same calculation. A two-sided test at level $\alpha$ rejects "the difference is zero" exactly when the $1 - \alpha$ confidence interval for the difference excludes zero. The interval says more, because it also shows how big the difference might be, so prefer reporting intervals.

The p-value is the most misread number in science, so it is worth being blunt about what it is not (Wasserstein & Lazar, 2016, give the American Statistical Association's version).

• It is not the probability that the null hypothesis is true. That would be $\mathbb{P}(H_0 \mid \text{data})$, and the p-value is a probability of data assuming $H_0$, the same confusion of conditionals that §0.4 warned about.
• It is not the size or importance of an effect. With a million test items, a 0.1-point improvement nobody would notice is highly significant.
• A large p-value is not evidence of no difference. "Not significant" usually means "too few items to tell", which the section on power makes precise.

A permutation test builds the null distribution directly from the data, with no formula. If the models are equally good, then for each question, which model got the $+1$ and which the $-1$ is arbitrary, so flipping the sign of each $\Delta_i$ at random produces a draw from the null. Repeat many times and see how often the flipped mean is as extreme as the real one. For paired data this is called the paired randomization test (or sign-flip test).

The three tests agree closely, as they should, since they ask the same question of the same discordant questions.

Nothing in the paired randomization test or the paired bootstrap needs the per-item scores to be 0 or 1. They work unchanged for any metric computed per item and averaged, such as nDCG or reciprocal rank per query in retrieval (§7.5), a judge's score per prompt, or a per-document ROUGE. Smucker, Allan & Carterette (2007) compared significance tests for information retrieval and recommend the randomization test for comparing two systems on the same queries. Here are two retrieval systems scored on the same 200 queries.

The paired tests agree with each other and find the 3.5-point gain clearly, while an unpaired test, swamped by how much queries differ from one another, calls it noise.

Multiple comparisons

Evaluate two identical models on 20 benchmarks, and test each at $\alpha = 0.05$. The chance that at least one benchmark shows a "significant" difference is $1 - 0.95^{20} = 0.64$. Run 20 seeds, 20 prompts, or 20 hyperparameter settings and report the best, and the same arithmetic applies whether or not anyone counted the attempts. Gelman and Loken call this the garden of forking paths. It is one of the main reasons published improvements fail to replicate.

Three corrections are standard, for $m$ tests with p-values $p_1, \dots, p_m$.

• Bonferroni rejects only when $p_i \le \alpha/m$. By the union bound, the probability of any false rejection is at most $m \cdot \alpha/m = \alpha$. This controls the family-wise error rate, at a high cost in power.
• Holm's method sorts the p-values, compares the smallest with $\alpha/m$, the next with $\alpha/(m-1)$, and so on, stopping at the first failure. It controls the same error rate and is never less powerful than Bonferroni, so there is no reason to use plain Bonferroni.
• Benjamini-Hochberg (1995) controls something different, the false discovery rate, the expected fraction of rejections that are false. Sort the p-values, find the largest $k$ with $p_{(k)} \le qk/m$, and reject the $k$ smallest. When many effects are real it finds far more of them than Holm, at the price of allowing a controlled share of false ones.

Testing each benchmark unadjusted finds 85% of the real effects but produces at least one false positive in almost half the runs. Holm nearly eliminates false positives but finds only half the real effects. Benjamini-Hochberg sits between, finding 71% of the real effects while its false discovery rate stays near 3%, under the 5% it promises.

Statistical power and sample size

A test's power is the probability that it detects an effect that is really there, $1 - \mathbb{P}(\text{type II error})$. It is the question to ask before running an evaluation, since an underpowered evaluation can only produce noise or exaggeration.

For a paired comparison, suppose the true mean difference is $\Delta$ and the per-item differences have standard deviation $s_\Delta$. The estimate $\bar\Delta$ is approximately $\mathcal{N}(\Delta, s_\Delta^2/n)$, and a two-sided test at level $\alpha$ rejects when $|\bar\Delta| > z_{1-\alpha/2}\,s_\Delta/\sqrt n$, where $z_q = \Phi^{-1}(q)$. Standardizing,

The second term, rejecting in the wrong direction, is negligible once power is meaningful. Setting the first term equal to a target power $1 - \beta$ means $\Delta\sqrt n/s_\Delta - z_{1-\alpha/2} = z_{1-\beta}$, so

In plain words, the number of items grows with the square of the noise-to-effect ratio. Halving the effect you want to detect quadruples the evaluation. For $\alpha = 0.05$ and 80% power, $z_{1-\alpha/2} + z_{1-\beta} = 1.96 + 0.84 = 2.80$, and $2.80^2 \approx 7.85$.

For 0-or-1 grades, $\Delta_i$ is $+1$ with probability $p_{01}$ (only B right), $-1$ with probability $p_{10}$ (only A right), and 0 otherwise, so $s_\Delta^2 = p_{01} + p_{10} - (p_{01} - p_{10})^2$. The discordance rate $p_{01} + p_{10}$ is what matters. Two models near 70% accuracy that disagree on 15% of questions have $s_\Delta \approx 0.39$, while an unpaired comparison of two independent samples has $s^2 = 2 \cdot 0.7 \cdot 0.3 = 0.42$, which is $s \approx 0.65$.

A 2-point improvement needs about 2,900 paired questions to detect reliably, and 1 point needs nearly 12,000. Many popular benchmarks have a few hundred to a thousand questions, so many claimed improvements on them were never detectable. Card et al. (2020) reached the same conclusion for NLP more broadly. They found underpowered comparisons common, and estimated, for example, that machine translation test sets of 2,000 sentences have about 75% power to detect a 1-point BLEU difference.

Low power does worse than miss real effects. It exaggerates the ones it finds. If only significant results get reported, and significance requires an observed gain of about two standard errors, then an underpowered study can only report gains that are much larger than the truth. Gelman & Carlin (2014) call these type M (magnitude) and type S (sign) errors.

With a true gain of one point and 1,000 questions, only 13% of evaluations reach significance, and those that do report a gain three times too large on average. This is why surprising results from small evaluations so often shrink when someone reruns them.

import numpy as np
from scipy.stats import norm

def paired_sd(p10, p01):
    raise NotImplementedError

def power(delta, sd, n, alpha=0.05):
    raise NotImplementedError

In PyTorch

At scale, metrics run on the GPU over millions of predictions, and calibration is fitted on held-out logits. Here is temperature scaling the way Guo et al. implemented it, with L-BFGS on the validation NLL, along with ECE and a rank-based AUC in plain PyTorch.

The ece here uses the identity $n_b\,|\operatorname{acc}(b) - \operatorname{conf}(b)| = |\sum_{i \in b}(\text{conf}_i - \text{correct}_i)|$, so two bincount calls compute every bin at once with no Python loop.

How each is misused

Two failures in the table are about the data, not the statistics, and no interval can fix them. A test set that leaked into training (contamination) makes every number optimistic, which the evaluation chapters treat as a problem of its own. And comparing your number with a number from a paper that used another prompt, another answer extractor, or another evaluation harness compares two different measurements.

Read more

• Evan Miller (2024). Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations. Standard errors, clustering, paired differences, and power analysis, written for language model evaluations.
• Guo, Pleiss, Sun & Weinberger (2017). On Calibration of Modern Neural Networks. Why deep networks are overconfident, reliability diagrams, ECE, and temperature scaling.
• Davis & Goadrich (2006). The relationship between Precision-Recall and ROC curves. Dominance in ROC and PR space, and why PR curves must not be linearly interpolated.
• Saito & Rehmsmeier (2015). The Precision-Recall Plot Is More Informative than the ROC Plot When Evaluating Binary Classifiers on Imbalanced Datasets. The imbalance argument, with simulations and real examples.
• Brown, Cai & DasGupta (2001). Interval Estimation for a Binomial Proportion. Why the Wald interval fails, and why Wilson and its relatives should replace it.
• Dietterich (1998). Approximate Statistical Tests for Comparing Supervised Classification Learning Algorithms. McNemar's test and the pitfalls of resampled t-tests for comparing classifiers.
• Card et al. (2020). With Little Power Comes Great Responsibility. How often NLP comparisons are underpowered, with tools for power analysis.
• Benjamini & Hochberg (1995). Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing. The false discovery rate and the step-up procedure.

Watch next

▶

▶

▶