This section is about the tests you build for your own product, the ones that decide whether a change ships. A public benchmark is written by someone else and shared by everyone, so its items and its grading are fixed. For your product you choose both. That control makes a trustworthy measurement possible, and it is also how teams fool themselves. What follows turns it into a method, with the statistics behind each step.

Two terms come first. An item is one test input together with a way to grade the response to it, such as a reference answer, a code check, or a judge's rubric. A product's eval suite is its collection of items. Its only job is to predict what your users will see.

A second failure sits under the first. Shankar et al. (2024) watched practitioners build automatic graders for their own LLM pipelines. People could not write down their grading criteria until they had graded some outputs, and grading outputs kept changing the criteria. The authors called this criteria drift. It means you cannot design a good eval for a product from a desk. You have to look at what the product does.

The cell below reproduces the example. It stores, for each kind of request, its share of production traffic, its number of suite items, and the pass rates of the two versions.

The plain average over the suite says the candidate gained 2.1 points. The average weighted by production says it lost 1.65. The rest of the section shows how to build a suite whose number you can trust, in the order a team needs the pieces. First you find failures by reading real interactions. Then you weight a lopsided suite like production, spend a labeling budget where it helps most, and write synthetic items that measure something. Then you build a release gate that neither blocks good changes nor ships bad ones. The section ends with flaky items, keeping a suite alive, validating an automatic judge, and monitoring the product once it is live.

Your eval suite is a sample of your users' traffic

An evaluation is an estimate, as §22.1 set out. The quantity you care about is the average true quality of the model's answers on the requests real users send. An evaluation estimates it by grading the model on a sample of test items. The estimate can be wrong in three main ways. The items can differ from real use, the grader can differ from true quality, or the sample can be too small.

In symbols, let $\mathcal{D}_{\text{use}}$ be the distribution of real requests $x$, let $y$ be the model's response, and let $g^\star(x, y) \in [0, 1]$ be its true quality. The target is $\mu^\star = \mathbb{E}[g^\star(x, y)]$, the average over requests $x \sim \mathcal{D}_{\text{use}}$ and the model's sampled responses $y$. An evaluation averages a grader $g$ over $n$ items drawn from a benchmark distribution $\mathcal{D}_{\text{bench}}$. In the terms of §22.1, a gap between $\mathcal{D}_{\text{bench}}$ and $\mathcal{D}_{\text{use}}$ causes validity bias, and a gap between $g$ and $g^\star$ is grader error, one part of measurement bias.

For a public benchmark, someone else fixed $\mathcal{D}_{\text{bench}}$ and $g$. For your own product both are yours. You choose the items, so you can make $\mathcal{D}_{\text{bench}}$ look like $\mathcal{D}_{\text{use}}$. You choose the grader, so you can make $g$ check what your users need. The support team in the opening example had this control and did not use it.

§14.7 gave a short checklist for a private eval set, written for checking a fine-tuned model. Start from real usage, read outputs before writing metrics, split the set by kind of request and size each part, write a grader per item, include regressions and refusals, and keep the set clean and versioned. This section is the full method behind that checklist. It adds the workflow that finds failures, the estimators that turn a skewed suite into a production estimate, and the tests that turn an estimate into a release decision.

Eval-driven development

Eval-driven development means letting measurement decide every change to the system. In ordinary software you can often write the test before the code, because you know what the failure would look like. With an LLM system the possible failures are too many and too surprising to list in advance. That is what criteria drift says. So most of an LLM product's evals come from looking at what the system gets wrong. A smaller part comes from requirements you can state exactly before any output exists, such as "never quote a refund amount that is not in the policy" or "answer in the user's language".

Practitioners agree on most of this loop and disagree on one step, whether to write evals before the failures exist. Anthropic (2026), a guide to agent evals, recommends eval-driven development. It includes evals for capabilities the agent cannot yet perform, which give the team a target to climb toward. Husain and Shankar's evals FAQ answers "Should I practice eval-driven development?" with "Generally no". They mean that writing evaluators for failures you only imagine wastes effort. Their exception is a crisp constraint known in advance, such as never mentioning a competitor. The loop in the box takes the common ground. Most items come from failures you observed, and the rest from requirements you can state exactly.

The loop keeps two kinds of eval, the split that §22.5 defined for agents. A capability eval asks whether the system can do something at all, so its items are ones the system mostly fails today and its pass rate has room to climb. A regression eval asks whether the system still does everything it used to, so its items are ones it should pass nearly every time, and any drop signals a break. Capability items graduate into the regression suite once the system handles them reliably.

The advice to start with 20 to 50 tasks follows from a power calculation. Score each item on the old and the new system. Call an item discordant when one version passes it and the other fails. Only discordant items carry information about the change, since an item both versions pass, or both fail, looks the same either way. Suppose a prompt fix makes 30% of the items pass that used to fail, and breaks none. Then 30% of the items are discordant, all in the new version's favor. About 19 items are enough to detect that change with 80% power in a two-sided test at the 5% level. (Power is the probability that the test detects a change that is really there.)

The general rule comes from the paired test of §0.11, which scores both versions on the same items and analyzes the per-item differences. Give each item the difference $\Delta_i = b_i - a_i$ between its new score $b_i$ and its old score $a_i$. Let $\Delta$ be the true mean change and $s_\Delta$ the standard deviation of $\Delta_i$. A two-sided test at level $\alpha$ with power $1-\beta$ needs

items, where $z_q$ is the $q$ quantile of the standard normal. For pass or fail items each $\Delta_i$ is $-1$, $0$, or $+1$, so $\Delta_i^2$ is 1 exactly on discordant items. If $d$ is the discordant fraction, then $\mathbb{E}[\Delta_i^2] = d$ and $s_\Delta^2 = d - \Delta^2$.

In plain words, the items you need grow with the noise of the per-item difference and shrink with the square of the change you want to see. Check the first case by hand. With $\Delta = 0.3$ and $d = 0.3$, $s_\Delta = \sqrt{0.3 - 0.09} = 0.458$, and $z_{0.975} + z_{0.8} = 1.960 + 0.842 = 2.802$. So $n = (2.802 \cdot 0.458/0.3)^2 = 18.3$, which rounds up to 19.

The formula is a normal approximation, and at 19 items it is optimistic. The exact test for paired pass or fail items, McNemar's test of §0.11, uses only the discordant items and asks whether wins outnumber losses by more than chance. In this example it reaches 80% power only at 25 items. Either way, 20 to 50 items is the right scale.

A change that raises the pass rate by 30 points is detected with 19 to 28 paired items, depending on how many items it also breaks. That is where "20 to 50" comes from. A 10-point change needs about 150 items, and a 2-point change nearly 3,000. So a suite has to grow as the product matures and the changes worth detecting shrink.

An interviewer who asks "how big should our eval be?" wants this answer, with the formula behind it.

Error analysis, from traces to a failure taxonomy

Error analysis is how you find out what to test. You read records of real interactions, note what went wrong in each, and group the notes into a short list of failure modes, the distinct kinds of mistake the system makes. The records are called traces.

Error analysis is reading traces and turning what you find into categories you can count. The method is borrowed from social science. Grounded theory is a way of building categories from interview transcripts instead of deciding them in advance (Glaser and Strauss 1967). Two of its procedures do the work.

Four practical rules make the procedure work, and each has a reason.

• One person with domain judgment codes first. Husain and Shankar's evals FAQ calls this person the "benevolent dictator", a domain expert who decides what counts as a failure. Several coders produce inconsistent notes before a taxonomy exists. Agreement between coders becomes measurable (§22.4) only once the categories are fixed.
• Record the first failure in each trace. Errors compound. A wrong retrieval causes a wrong answer, which causes an angry follow-up, which causes a failed handoff. Counting all four inflates the downstream symptoms. The first failure is the one whose fix removes the rest.
• Use pass or fail per failure mode, not a 1 to 5 score. A binary label forces a decision, is faster to give, and is more consistent across annotators, because the line between a 3 and a 4 is personal. It is also easier to analyze, since a pass rate $p$ on $n$ items has the known standard error $\sqrt{p(1-p)/n}$.
• Count, then rank. Sort the counts of first failures from largest to smallest. The top few modes usually cover most failing traces. That ranking decides what to fix and what to write evals for first. A table sorted this way, with a running total of the shares, is called a Pareto table.

How many traces are enough? Start with one failure mode, and ask how likely you are to miss it. Suppose it is the first failure in 10% of failing traces. Each failing trace misses it with probability 0.9, so 20 failing traces all miss it with probability $0.9^{20} = 0.12$. One time in eight, a mode behind a tenth of your failures stays invisible after 20 failing traces.

The general statement has two parts. The first says how long a rare mode can hide. The second estimates how much you have not yet seen. Assume failing traces are independent draws, and that failure mode $j$ is the first failure in a fraction $q_j$ of them. Call $q_j$ the prevalence of mode $j$.

In plain words, count the modes you have seen exactly once and divide by the failing traces read. That fraction estimates the chance that the next failing trace shows you something new. (Strictly, it estimates that chance as it stood one trace earlier, which is nearly the same once $t$ is in the dozens.) For example, if 3 of your modes have appeared once each in 60 failing traces, the next failing trace has about a $3/60 = 5\%$ chance of being a new mode. Good (1953) derived this estimate for counting animal species not yet observed, crediting the idea to Alan Turing, and it is often called the Good-Turing estimate. It is a better stopping signal than a feeling that "we have seen it all".

Part (a) says how long a rare mode can hide. A mode behind 2% of failures needs about 150 failing traces to appear with 95% probability. At a 40% failure rate that is about 370 traces read.

The simulation below codes traces from a taxonomy of 12 modes with a long tail, a few common modes and many rare ones. Each trace fails with probability 0.4. Coding stops when 40 traces in a row bring nothing new, the kind of stopping rule teams use. It repeats this 500 times and reports what the rule missed.

The quiet-window rule stops after a median of 115 traces, close to the "at least 100" of practice, having found about 9 of the 12 modes. The modes it missed still cause 7.4% of failures on average, and 15.6% in the worst tenth of runs. The singleton estimate at the stopping point, 5.3%, runs low. That is no accident. The rule stops after a quiet stretch, and a quiet stretch is exactly when singletons become doubletons and no new ones appear. Still, the estimate is the right order of magnitude, and it says plainly that the taxonomy is not finished.

The lesson is not that 100 traces are too few. It is that error analysis is never done. Repeat it on fresh traces after every major change and on a schedule. In an interview, "when do you stop reading traces?" has this answer. You stop when new traces stop adding modes, and the share of modes seen only once is the number that backs it up.

def failure_taxonomy(traces, window):
    counts = {}
    for tr in traces:
        for label in tr:                      # placeholder: counts every failure, not the first
            counts[label] = counts.get(label, 0) + 1
    pairs = sorted(counts.items(), key=lambda kv: -kv[1])
    return {"counts": pairs, "cumulative": [], "saturated_at": None}

Collecting failures from production

Once a product has users, its own traffic is the best source of eval items. There are five common ways to find failures in it. Each is biased in its own way, so a good suite draws on several.

The uniform sample matters more than it looks. It is the only source that measures how much of production traffic each kind of request makes up, and the next part needs exactly that. It is also the only one that finds failures nobody reported. Read a uniform sample in every round of error analysis, and use the targeted sources to collect many examples of each mode once you know it exists.

Production traces contain personal data. Get consent through the product's terms. Scrub or pseudonymize identifiers before traces enter the eval store, restrict who can read raw traces, and set a retention period. Keep eval items derived from user data out of every training set. That protects privacy, and it protects the measurement, because a model trained on the suite's items scores well on them without being better (§22.3). Teams that cannot keep raw traffic often write synthetic items modeled on the failure modes they found, which a later part of this section treats.

Weighting the suite like production

The opening example was a weighting failure, and it has a clean fix. Split the requests into kinds, such as login, orders, and refunds. Each kind is a stratum (plural strata). Measure the pass rate on each stratum separately, then average those rates using each stratum's share of production traffic rather than its share of the suite.

The opening numbers show the size of the effect on a single version. The suite's plain average for the shipped version weights each stratum by its share of the suite, $(700 \cdot 0.95 + 200 \cdot 0.80 + 100 \cdot 0.60)/1000 = 0.885$. Weighting by production share gives $0.15 \cdot 0.95 + 0.50 \cdot 0.80 + 0.35 \cdot 0.60 = 0.1425 + 0.40 + 0.21 = 0.7525$. The suite overstates production quality by 13 points.

Now the general case. There are $H$ strata. For stratum $h$, let $w_h$ be its share of production traffic, $p_h$ the system's true pass rate on that traffic, $n_h$ its number of suite items, and $\hat p_h$ the observed pass rate on them, with $n = \sum_h n_h$. The suite's pooled mean is $\sum_h (n_h/n)\,\hat p_h$. Production weights stratum $h$ by $w_h$ instead.

In plain words, measure each kind of request on its own, then average the per-kind pass rates with production's weights rather than the suite's. The suite can be as lopsided as convenience made it, provided each stratum's items are representative of that stratum.

One caution about the plug-in standard error. A stratum whose observed pass rate is exactly 0 or 1 contributes nothing to it, which understates the uncertainty of small strata. A stratum with 20 of 20 passes is not certain to pass every time. For such strata, use the Wilson interval of §0.11, which keeps every pass rate the data do not rule out and so stays sensible near 0 and 1. Or add a pseudo-count, one imagined pass and one imagined fail, before computing $\hat p_h(1-\hat p_h)$.

The proviso "representative of that stratum" is where teams go wrong. Post-stratification corrects the imbalance between strata. It cannot correct selection within a stratum. Items collected because they failed, such as those mined from complaints, are not a random sample of their stratum. Their pass rate says nothing about the stratum's pass rate. Keep failure-mined items in a separate capability or regression set. Estimate production quality only from items sampled at random within each stratum.

The simulation reruns the suite 20,000 times with fresh items. The pooled mean is reliably wrong, and the post-stratified interval covers the truth at its promised rate.

The pooled mean averages 0.885 against a truth of 0.7525, and its 95% interval never covers the truth. The post-stratified estimate is centered on the truth, its spread of 2.23 points matches the formula, and its interval covers the truth 94.6% of the time. In practice, report the post-stratified number and its interval, never the plain suite average. An interviewer who hears "our eval went up but users are unhappy" expects you to ask first whether the suite's mix matches production.

import numpy as np

def post_stratified(p_hat, n, w):
    p_hat, n = np.asarray(p_hat, float), np.asarray(n, float)
    est = (n * p_hat).sum() / n.sum()                      # placeholder: the pooled suite mean
    return float(est), float(np.sqrt(est * (1 - est) / n.sum()))

Spending a labeling budget

Graded items cost money, often an expert's time. So the next question is how to spend a budget of $N$ new labels across strata. The natural default is proportional allocation, $n_h = N w_h$, which mirrors production. It is not the best choice. A stratum whose pass rate is near 100% gives nearly the same answer on every item, so its estimate is already precise and extra labels there are nearly wasted.

The fix is to weight by uncertainty too. The rule is to give each stratum labels in proportion to its traffic share times the spread of its items' scores. Write $\sigma_h = \sqrt{p_h(1-p_h)}$ for the per-item standard deviation in stratum $h$. It is 0.5 for a stratum that passes half the time and 0.22 for one that passes 95% of the time.

In plain words, give each stratum labels in proportion to its share of traffic times the uncertainty of its items, and fewer where labels are expensive. Neyman (1934) derived this rule for survey sampling, and it carries his name. Its advantage over proportional allocation is exactly the spread of the strata's standard deviations. So it pays most when some strata are nearly always right and others are coin flips.

The cell compares this with three other ways to spend the same 1,000 labels.

Neyman allocation reaches a standard error of 1.28 points. Proportional allocation gives 1.31, equal allocation 1.45, and the suite's current shape 2.23. Here Neyman beats proportional only slightly, because the three $\sigma_h$ (0.22, 0.40, 0.49) are not far apart. The big win is leaving the current shape. The last line checks the identity for the difference numerically.

Two cautions apply. First, the $\sigma_h$ are unknown before labeling. Plan with pilot estimates, or with the worst case $\sigma_h = 0.5$ for every stratum, which turns Neyman into proportional allocation. $V$ is flat near its minimum, so rough estimates lose little. Second, Neyman minimizes the variance of the overall estimate, which can starve a stratum with little traffic. Some rare strata matter in their own right, such as self-harm conversations at 0.5% of traffic. Give each of them a floor $n_{\min}$ large enough for its own interval, and allocate the rest by Neyman.

In the widget, the three support strata keep their traffic shares and you set their pass rates. Each bar splits the same 1,000 labels by one scheme, and the number at its right is the standard error it buys. The big number is how many labels proportional allocation needs to match Neyman's error bar. At the defaults it is about 1,050, a small lead. Now make orders, the largest stratum, pass 99% of the time. Neyman cuts its labels from 495 to 196, and proportional allocation needs about 1,490 labels to keep up.

import numpy as np

def neyman_allocation(w, sigma, N, n_min=1):
    w = np.asarray(w, float)
    return np.round(N * w).astype(np.int64)        # placeholder: proportional allocation

def variance_of_allocation(w, sigma, n):
    raise NotImplementedError

Synthetic evals that measure something

Synthetic items are test inputs written by a model or a program rather than collected from users. They fill three gaps. Before launch there is no traffic. Some failure modes are too rare in traffic to collect. Some combinations of conditions never occur together in a small sample. Their danger is that a model writing test items writes the items that are easy for models to write, and those are often easy for models to answer.

Two construction methods have a track record. The first is structured generation. List the dimensions along which requests vary. For a support assistant they might be the intent, the customer's mood, the account type, and the language. Pick combinations of their values, such as (refund, angry, business account, Spanish), at first by hand. Then, as a separate step, have a model write a realistic request for each combination. Husain and Shankar's evals FAQ recommends this over asking a model for "test queries", which yields generic and repetitive output. The second method is generate and filter. Have a model write many candidate items for a behavior, then keep only those that pass automated and human checks.

Software testing supplies a useful template for what to generate. CheckList (Ribeiro et al. 2020) lists the capabilities a model needs, such as handling negation or names, and writes three types of test for each one.

In CheckList's user studies, a team responsible for a commercial sentiment model found new, actionable bugs in a model they had already tested extensively. Practitioners using the method wrote about twice as many tests and found almost three times as many bugs as those without it. Invariance tests are cheap to generate in bulk, because the perturbation is the only thing that needs writing. They also need no reference answer, only agreement between two outputs.

Every synthetic item then passes four filters before it enters the suite.

1. Validity. The item has a correct reference answer and the grader accepts it. Run the reference through the checker, and spot-check a sample by hand. A broken item fails every model and looks like a hard item, which §22.5 showed is a common source of false failures.
2. Difficulty. Drop items that every model passes or every model fails. They say nothing about differences between models, though an item every model passes can still serve as a regression check.
3. Discrimination. Keep items on which stronger models do better (made precise below), and look hard at items where they do not. Those are often ambiguous.
4. Diversity and independence. Remove near-duplicates by embedding similarity. Generate with a model family different from the one being evaluated. Items written by the model under test may play to its strengths, and a judge from the same family prefers that family's outputs (§22.4).

The third filter needs a number. The idea is to ask, across a panel of models, whether the models that pass this item are the ones that do well on the rest of the suite.

In plain words, an item discriminates when the models that pass it are the ones that do well on everything else. Why leave the item out of the total? Because an item always correlates with a total that contains it. Write the total as $T = x_j + R_j$, and let $\sigma_j$ and $\sigma_T$ be the standard deviations of the item and the total across models. If item $j$ is pure noise, independent of every other item, then $\operatorname{Cov}(x_j, T) = \operatorname{Var}(x_j) + \operatorname{Cov}(x_j, R_j) = \sigma_j^2$. Its item-total correlation is $\sigma_j^2/(\sigma_j\sigma_T) = \sigma_j/\sigma_T > 0$, a positive discrimination for an item that measures nothing. Its item-rest correlation is 0. The inflation is largest on short suites, which is where synthetic item sets start.

The cell checks this on simulated data. Forty models of varying ability answer 15 items, where a model's chance of passing an item rises with its ability minus the item's difficulty. Item 0 is replaced by a coin flip that ignores ability. The experiment is repeated 2,000 times.

On a 15-item suite the coin-flip item has an average item-total correlation of 0.16, as the formula predicts ($\sigma_j/\sigma_T = 0.17$), and an item-rest correlation of zero. A real item scores 0.32 item-rest. So on the inflated measure, the coin flip looks half as discriminating as a real item does on the honest one. In practice, sort new synthetic items by item-rest discrimination and read the bottom of the list first.

import numpy as np

def rank_items(P, valid):
    # drop invalid and constant items, then rank the rest by item-rest correlation
    raise NotImplementedError

Release gates that do not lie

Now the decision itself. A release gate is the rule that decides, from the suite, whether a candidate replaces the shipped system. A good gate rarely ships a real regression and rarely blocks a harmless change, and the three rules teams use differ exactly there. The shipped system and the candidate are scored on the same $n$ items. Item $i$ gets an old score $a_i$ and a new score $b_i$, and the paired difference is $\Delta_i = b_i - a_i$. Write $\bar\Delta$ for the mean of the differences, $s_\Delta$ for their sample standard deviation, and $\Delta$ for the true mean change, as in the paired analysis of §0.11.

Most teams' first gate is "ship if the score did not drop", $\bar\Delta \ge 0$. Suppose the candidate changes nothing, so $\Delta = 0$. Its observed $\bar\Delta$ is then as likely to dip below zero as to rise above it, and the rule blocks the release about half the time, at every suite size. Teams learn to ignore a gate that blocks harmless changes half the time, and an ignored gate gates nothing.

The usual statistical fix is "ship unless the drop is significant". It blocks only when a two-sided test rejects $\Delta = 0$ in the negative direction. So the candidate ships unless the data prove it worse, which puts the burden of proof on the gate. A small suite has little power, so it rarely shows a significant drop, and real regressions sail through.

The question a gate should answer is different. It is "is the candidate not meaningfully worse?", with "meaningfully" fixed in advance as a margin. That is a non-inferiority test. Clinical trials use it for new treatments that are cheaper or safer but must not be much less effective (Walker and Nowacki 2011).

In plain words, ship only when the data show, with confidence $1-\alpha$, that the candidate is no more than $\delta$ worse. Failing to show that is a reason to block, whatever the cause, including a suite too small to tell.

Two things differ from the two-sided formula of §0.11. The critical value is one-sided, $z_{0.95} = 1.645$ rather than $1.96$, because only a drop can block. And the effect in the denominator is the distance to the margin, $\Delta + \delta$, rather than an improvement to detect.

Take 15% of items discordant, so $s_\Delta = \sqrt{0.15} = 0.387$, with 80% power at one-sided $\alpha = 0.05$. Ruling out a 1-point drop takes 9,274 paired items, and ruling out a 2-point drop takes 2,319. By hand, the bracket is $1.645 + 0.842 = 2.486$, and $(2.486 \cdot 0.387/0.01)^2 \approx 9{,}270$. Few product suites are that large. That is why the margin, the discordance, and the scoring deserve as much thought as the items.

The simulation runs 10,000 releases at each true change, for a small and a large suite, with margin $\delta = 2$ points and 15% discordance. It compares the three rules.

Read the rows at true change 0 and at $-2$ or $-3$ points. On 2,500 items the naive rule ships a release that changes nothing about half the time (51% here, a little over one half because an exact tie counts as no drop), and ships a 1-point improvement only 90% of the time. The rule "ship unless significantly worse" ships 26% of 2-point regressions on 2,500 items and 86% of them on 300 items. The non-inferiority gate ships 2-point regressions about 5% of the time on both suites, as the theorem promises. It ships the unchanged release about 82% of the time on 2,500 items, but only 23% of the time on 300. That is not a defect. It is the gate reporting that 300 items cannot certify a 2-point margin, which the other two rules hide.

The widget below draws the same comparison as smooth curves, from the normal approximation of the theorem. Each curve is one rule's chance of shipping against the true change, for the naive rule, the significance test ("ship unless significantly worse"), and non-inferiority. The shaded band left of the dashed margin holds the releases that should be blocked.

An interviewer will ask how you would gate releases on your eval. The answer is a one-sided non-inferiority test on paired differences, with a margin agreed in advance and a suite sized by the formula above.

Gating many strata at once

A production suite reports per-stratum results, and a release that is fine on average can still break one stratum. The natural rule is to require non-inferiority in every stratum, each with its own margin. It looks as if this needs a correction for testing many hypotheses at once, like those of §0.11. It does not.

In plain words, a rule that needs every check to pass is harder to pass than any one check, so its false-ship rate is no worse than one check's. The price is paid on the other side. If each stratum's gate ships an unchanged candidate with probability 0.8, five independent strata all pass with probability $0.8^5 = 0.33$. So per-stratum gates need per-stratum power, which is another reason for the floors in the allocation.

A single overall gate uses the post-stratified difference $\sum_h w_h\bar\Delta_h$, with variance $\sum_h w_h^2 s_{\Delta,h}^2/n_h$, the paired version of the post-stratified estimate. A common design combines the two kinds. It runs an overall non-inferiority gate, adds per-stratum gates only for strata where a regression is unacceptable on its own, and adds zero-tolerance checks for safety items. For those, remember the rule of three of §0.11. Zero failures in $n$ trials only bound the failure rate below about $3/n$ at 95% confidence. So a clean run of 300 items still allows a failure rate of about 1%.

import numpy as np

def noninferiority_gate(old, new, delta, alpha=0.05):
    d = np.mean(new) - np.mean(old)
    return {"diff": d, "lower": d, "ship": d >= 0}      # placeholder: the naive rule

def n_required(delta, s_delta, alpha=0.05, power=0.8):
    raise NotImplementedError

Latency and cost belong in the gate

Quality is not the only thing a release can break. A candidate that is non-inferior on every stratum but writes twice as many tokens costs twice as much and is noticeably slower. So the 95th percentile of latency (p95) and the cost per request belong in the release gate beside quality, each with a limit fixed in advance like the margin.

Measure latency by replaying suite traffic at production concurrency, since latency depends on load and on output length. Report p95, not the mean, because the slow tail is what users notice. Count cost per request over every model call a request makes (retries, tool calls, hidden reasoning tokens, and any judge in the serving path) at current prices.

Both numbers carry sampling error. The mean cost has the usual standard error of a mean. The p95 of $n$ latencies has an interval that holds for any distribution, between two of the sorted latencies. The reason is that the number of sampled latencies below the true p95 is binomial with parameters $n$ and 0.95. For $n = 2{,}000$ that count has standard deviation $\sqrt{2000 \cdot 0.95 \cdot 0.05} = 9.7$. So the 95% interval runs from about the 1,881st to the 1,919th sorted latency, roughly the sample's 94th to 96th percentile.

Flaky items and reruns

LLM outputs are sampled, and agent environments add their own randomness (§22.5), so a healthy item can fail by chance. An item that sometimes passes and sometimes fails with no change to the system is called flaky. A suite of 200 items that each pass 98% of the time is entirely green on one run with probability $0.98^{200} = 0.018$. So "all tests pass" cannot be the gate for an LLM system.

Teams then add reruns, and each policy trades one error for another. Suppose a healthy item passes each run with probability $q$, and a broken one with probability $q' \lt q$. On a single run, a healthy item raises a false alarm with probability $1 - q$.

• Pass if any of $r$ runs passes (rerun on failure). The false-alarm rate falls to $(1-q)^r$, but a broken item is caught only with probability $(1-q')^r$. This quietly changes the metric to pass@$r$, the chance that at least one of $r$ runs succeeds, which is the rerun trap of §22.5.
• Pass only if all $r$ runs pass. This is the reliability metric $\text{pass}^k$ of §22.5, the chance that all $k$ runs succeed, with $k = r$. A broken item is caught with probability $1 - q'^r$, but the false-alarm rate rises to $1 - q^r$.

Take $q = 0.9$ and $q' = 0.5$. Rerunning on failure twice (three runs in all) cuts false alarms from 10% to 0.1%. It also cuts the chance of catching the broken item from 50% to 12.5%. Neither policy is a fix, because both judge an item by a handful of coin flips.

The fix is to stop treating an item as pass or fail on one run. Run each item $K$ times on both versions, score it by its pass rate, and gate on the paired differences of those rates with the non-inferiority test. That uses every run, and it lowers $s_\Delta$, because averaging $K$ runs divides the within-item part of the variance by $K$.

Failures caused by the environment rather than the model (a crashed tool, a timeout) are a separate matter. Log them separately and rerun only those, as §22.5 showed. Move items whose answer depends on the date or on a live service out of the gating suite until they are pinned to fixed data.

Keeping the suite alive

A suite is a product with its own maintenance, and most of the ways it decays are slow.

• Version everything the score depends on. The items, the rubric, the judge model and its prompt, the scaffold (the code around the model, its prompts, tools, and control loop), and the decoding settings share one version number. Compare two systems only on the same suite version, run in the same batch, so the comparison stays paired. A score on version 7 and a score on version 8 are different measurements.
• Never edit an item in place. Retire it and add a corrected one, so history stays comparable.
• Separate capability items from regression items. An item every candidate passes cannot be discordant, so it says nothing about a change. It still guards against regressions, so move it to the regression suite rather than deleting it. Report capability and regression results separately.
• Hold out a rotating slice. A team that keeps tuning against its own suite overfits it. This is Goodhart's law from §22.1, that a measure which becomes a target stops measuring, at the scale of one team. Keep a slice that is scored only at release time and refreshed from new traffic. Kapoor et al. (2024) found this discipline missing from many agent benchmarks.
• Refresh from production. Traffic shifts, so re-estimate the weights $w_h$ and rerun error analysis on a schedule, adding new failure modes as new strata or items.
• Track criteria drift. When the team's idea of a good answer changes, that is a new rubric version. Re-grade or retire the old labels made under the previous rubric.
• Keep it out of training. Decontaminate training and fine-tuning data against the suite. Put a canary string, a unique random marker, in the suite's files, so that a model that has trained on them can be caught (§22.3).

One judge per failure mode

Many failure modes cannot be checked by code, such as "the reply blamed the customer". They need a judge, a language model prompted to grade a response. §22.4 covered judges in depth. For a product suite, the practice that works is narrow judges. Write one binary judge per failure mode, each with a short rubric for that mode alone, and validate each against human labels for that mode. A single judge that rates "overall quality" from 1 to 5 is hard to validate, drifts, and cannot tell you which failure is rising.

Validation is a small supervised learning problem, with the same discipline. Collect human pass or fail labels for the mode and split them three ways. A train split supplies examples placed in the judge's prompt. A dev split is used to iterate on the prompt. A test split is touched once, at the end, because every change made after looking at dev results fits the judge to the dev set.

What you measure on the test split is how often the judge is right on each kind of response. A judge that catches 27 of 30 real failures and passes 114 of 120 good responses is right 90% of the time on failures and 95% of the time on passes. Those two numbers have names.

In plain words, TPR is the share of real failures the judge catches, and TNR is the share of good responses it lets through, as in §19.4. Both are conditional on the true label, so they do not depend on how common failures are in the labeled set. You may therefore enrich the labeled set with known failures, and you need to, because a failure mode at 10% prevalence gives only 15 positives in 150 random labels. Precision, by contrast, depends on prevalence and changes when you enrich.

A judge that catches 27 of 30 human-labeled failures has a TPR of 0.90, with a Wilson interval (§0.11) from 0.74 to 0.97. On a test split of 15 failures, 13 caught gives an interval from 0.62 to 0.96, which certifies little. Pinning TPR to within 5 points takes about 140 labeled failures. That arithmetic is behind the practitioner advice to label 100 to 200 examples per failure mode (evals FAQ), and it is the reason to label failures preferentially.

A judge's raw flag rate is the share of traces it marks as failures. Once TPR and TNR are known, the flag rate on a new batch can be corrected into an estimate of the true failure rate. Take a judge with TPR 0.90 and TNR 0.95 on traffic where 2% of traces truly fail. It flags $0.90 \cdot 0.02 = 1.8\%$ of traces as caught failures and $0.05 \cdot 0.98 = 4.9\%$ as false alarms, 6.7% in all. Most of what it reports is false alarms.

The correction is the Rogan-Gladen estimator, which §19.4 derived for attack success rates, with sensitivity and specificity in place of TPR and TNR. Here it is in this section's terms. If the true failure rate is $\pi$, a judge flags a fraction $f = \mathrm{TPR}\,\pi + (1-\mathrm{TNR})(1-\pi)$ of traces, the caught failures plus the false alarms on passes. When $\mathrm{TPR} + \mathrm{TNR} > 1$, solving for $\pi$ turns an observed flag rate $\hat f$ into the estimate

The estimator was first used to correct disease rates measured with an imperfect screening test (Rogan and Gladen 1978). On the example, $(0.067 + 0.95 - 1)/0.85 = 0.02$, which recovers the true 2%. The other route is prediction-powered inference (§14.7). It takes a random slice of traces that both humans and the judge labeled. It averages the human label minus the judge's label on that slice and adds this average to the judge's rate on all traces. Either way, keep a small human-labeled slice of every release's traffic, since a judge validated on last quarter's failures can miss this quarter's.

Monitoring in production

A release gate certifies a candidate on the traffic its suite was drawn from. After release, the traffic and the system both keep moving. New kinds of request arrive. A provider updates the model behind a fixed name. The document index is rebuilt, or someone edits a prompt outside the release process. Monitoring is the same measurement run continuously on live traffic, so that a rising failure mode is caught in days rather than through complaints. It reuses the pieces above in four steps.

1. Sample a fixed share of live traces at random, say 1% of each day's conversations, scrubbed as in the section on collecting failures. The share is fixed in advance and the choice is random, because a sample picked by a rule (only conversations with a thumbs-down) cannot estimate a rate. If strata are sampled at different rates, reweight by the production shares $w_h$ as in post-stratification.
2. Run every narrow judge on every sampled trace. A window of $n_{\text{live}}$ traces, say one day's sample, gives each failure mode a flag count $k$ and a raw flag rate $\hat f = k/n_{\text{live}}$. Because each judge looks for one mode, an alert names the failure that is rising.
3. Correct each flag rate with the formula above, using the TPR and TNR measured on that judge's test split.
4. Alert when the corrected rate's lower confidence bound exceeds a threshold $\tau$, fixed in advance as the largest failure rate the product tolerates for that mode. Using the lower bound rather than the point estimate is the release gate's logic turned around. The monitor pages only when the data show, with confidence $1-\alpha$, that the rate is above $\tau$.

The corrected rate's interval has a part that more live traffic cannot shrink, because the judge's error rates were estimated from a fixed set of labels.

In plain words, a corrected rate is uncertain for two reasons, today's sample and the judge's validation, and only the first goes away with more traffic. At a low failure rate the specificity term is the larger validation term, since its factor $(1-\pi)^2$ is nearly 1 while the sensitivity term's factor $\pi^2$ is nearly 0. So the way to sharpen a monitor is to label more passes, and a random human-labeled slice of traffic supplies them in volume, since most traces pass. And because the validation error is the same every day, it acts on a given monitor as a fixed bias, not as fresh noise.

The bound is a test, so every look can raise a false alarm. When the true rate sits exactly at $\tau$, each daily look fires with probability about $\alpha$. At $\alpha = 0.05$, 30 looks a month would page about 1.5 times for nothing, and with independent looks at least once with probability $1 - 0.95^{30} = 0.79$. Choose $\alpha$ per look from the false-alarm rate the on-call team can bear. At $\alpha = 0.01$ that is about one false page in 100 days at the threshold, and far fewer when the system runs well below it.

Two more checks keep the monitor honest. First, a change from a baseline window is sharper than a level. With the same judge in both windows, $\hat\pi_1 - \hat\pi_0 = (\hat f_1 - \hat f_0)/\hat Y$. The offset $\mathrm{TNR} - 1$ cancels, and $\hat Y$ only rescales the difference. So "did this failure mode rise since release?" is a comparison of two raw flag rates that needs no validation labels at all. The correction is needed only to say by how much.

Second, that argument assumes the judge's TPR and TNR are the same in both windows, and this is the monitor's weak point. They are conditional on the true label, so a change in how common failures are leaves them alone. A change in the kind of trace (a new market, a new language, a failure that looks different) can move them. So recompute them on the human-labeled slice every week or release. Watch input drift directly too, through the production shares $w_h$ and the share of traces the stratum classifier cannot place. The same monitor tracks p95 latency and cost per request, with their own thresholds.

The cell below checks the example and then simulates 20,000 monitors, each with its own validation labels, over 60 days in which the true failure rate jumps from 2.5% to 8% on day 40.

The formula's standard error, 1.15 points, matches the spread of the corrected rate over the 20,000 simulated monitors. A jump from 2.5% to 8% pages on its first day 87% of the time and within three days 97% of the time. Only 0.3% of monitors raise a false alarm in the 40 quiet days before it.

A month spent exactly at the threshold fires on 1.2% of looks, close to the nominal 1%, and on 13% of monitors at least once. Independent looks would give 26%.

An interviewer asking how to monitor an LLM feature wants four pieces. They are a random sample of live traces, narrow validated judges, the corrected rate with an honest interval, and a per-look level chosen for the on-call budget.

Evals for retrieval and agentic products

The method above is the same for any product, but two kinds of product add structure. A retrieval-augmented assistant over company documents finds passages and then writes an answer from them, and the two stages fail separately. So its suite measures each stage on its own, with the metrics §7.6 defined. For retrieval, recall at $k$ is the share of questions whose needed passage is among the top $k$ retrieved. For generation, faithfulness asks whether every claim in the answer is supported by the retrieved passages. Answer correctness and correct abstention on unanswerable questions complete the set. Stratify it by document type and by whether the question is answerable, multi-hop, or out of scope.

An agentic product acts over many steps, calling tools that change the world. Its suite follows §22.5. It grades the final state of the environment rather than what the transcript claims. It runs each task several times and reports $\text{pass}^k$, the chance that all $k$ runs succeed, for reliability. And it pins the environment so that reruns see the same world. Building such agents is the subject of Chapter 24.

Where an output can be checked by code (an exact answer, a unit test, a schema, a database state), prefer that grader to a judge. Code graders can still be gamed, as §18.3 showed, so read some of the outputs they pass. And a product that may refuse needs both a harmful-request suite and a benign suite that looks harmful, so that over-refusal is measured too (§19.4).

The real way, with an evaluation framework

Teams rarely build the harness from scratch. Open frameworks such as Inspect, from the UK AI Security Institute, and OpenAI's evals package datasets, solvers, scorers, and logs. The sketch below is a stratified regression task in Inspect. It has a code-based scorer for one failure mode, per-stratum accuracy, and four samples per item. Its standard errors are clustered by trace. Items cut from the same conversation tend to pass or fail together, so the error bar treats each trace, not each item, as one independent unit. It is shown, not run, since it calls a model provider.

The per-sample scores in the log are what the gate needs. Export them for the shipped and the candidate system on the same suite version. Average the four samples of each item into a per-item pass rate. For each stratum with its own gate, pass that stratum's two arrays to noninferiority_gate. For the overall gate, combine the per-stratum mean differences with the production weights, as in the section on gating many strata.

Where it breaks

• Overfitting to your own suite. Every prompt tweak tested on the same items fits them a little. The rotating holdout and a release-only slice are the defense.
• Within-stratum selection. Post-stratification fixes the mix of strata, not items chosen because they failed. Estimate production quality only from items sampled at random within strata.
• Stale weights and new kinds of traffic. A new feature creates traffic no stratum covers. Watch the share of production traces the stratum classifier cannot place.
• Suites that only grow. An ever-larger suite of items every version passes costs money and dilutes discordance. Retire capability items once they graduate, and keep the regression suite lean.
• Judges that drift. A new judge model or prompt is a new instrument. Pin it, version it, and revalidate TPR and TNR when it changes.
• The offline-online gap. Even a perfect offline suite measures only what you thought to test. Monitoring catches failure modes that rise after release, but only those its judges look for, so people still read the uniform sample in every round of error analysis. The final measurement is in production, by comparing the old and new systems on live users with an A/B test and product metrics. Those tests have statistics of their own. Checking a running test repeatedly and stopping at the first good result inflates false positives, which is why online teams use sequential tests built for repeated looks. Offline gates decide what is safe to try online, and online tests decide what was worth shipping.

In interviews these pieces come together in a design round, where you lay out an evaluation and safety system for a whole product, from offline gates and online monitoring to the escalation of safety issues.

Read more

• Shankar et al. (2024). Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences. The EvalGen study, criteria drift, and aligning automatic graders with a user's own grades.
• Husain and Shankar. AI Evals: Everything You Need to Know. Practitioner guidance on error analysis, open and axial coding, binary labels, and validating judges with TPR and TNR on dev and test splits.
• Anthropic (2026). Demystifying evals for AI agents. A roadmap from zero to a trusted suite, capability and regression evals, and grader bugs found by reading transcripts.
• Perez et al. (2022). Discovering Language Model Behaviors with Model-Written Evaluations. 154 generated and filtered datasets, their validation by crowdworkers, and the behaviors they uncovered.
• Ribeiro et al. (2020). Beyond Accuracy: Behavioral Testing of NLP models with CheckList. Minimum functionality, invariance, and directional tests as a template for behavioral suites.
• Good (1953). The Population Frequencies of Species and the Estimation of Population Parameters. The singleton estimate of unseen mass used here as a stopping signal for error analysis.
• Neyman (1934). On the Two Different Aspects of the Representative Method: The Method of Stratified Sampling and the Method of Purposive Selection. The origin of optimal allocation in stratified sampling.
• Walker and Nowacki (2011). Understanding Equivalence and Noninferiority Testing. A short, clear account of non-inferiority margins, one-sided bounds, and their interpretation.
• Berger (1982). Multiparameter Hypothesis Testing and Acceptance Sampling. The intersection-union principle, which is why requiring every stratum to pass needs no correction for multiple tests.
• Rogan and Gladen (1978). Estimating Prevalence from the Results of a Screening Test. The correction of an observed positive rate for a test's sensitivity and specificity, used here to correct a judge's flag rate.
• Page (1954). Continuous Inspection Schemes. The CUSUM chart, which accumulates evidence across windows to detect a sustained shift in a monitored rate.
• Kapoor et al. (2024). AI Agents That Matter. Why evaluations need proper holdout sets, and how missing ones let systems overfit their benchmarks.

Watch next

▶

▶

▶