A lab announces that its new model's attack success rate on a jailbreak benchmark fell from 6% to 3%. The safety team is pleased and the release goes ahead. How much did you just learn? If the benchmark had 100 prompts, the 95% confidence interval for the old model runs from about 2.8% to 12.5% and for the new one from about 1.0% to 8.5%. The intervals overlap heavily. The announcement is consistent with no change at all, and with a model that got worse.

Safety evaluations carry more weight than almost any other measurement in machine learning, because release decisions, public claims, and frontier safety commitments rest on them. They are also unusually easy to get wrong. The positive cases are rare, the judges are imperfect, the model may behave differently when it suspects a test, and the quantity we care about (would this model meaningfully help someone cause harm?) is not something any benchmark measures directly. This section covers the two everyday safety evals, refusal and over-refusal, with the statistics to report them honestly, then the evaluations for extreme risks, how frontier safety frameworks turn them into decisions, the human studies that measure real-world uplift, and the pitfalls that make safety numbers mislead.

Two numbers, always together

Every behavioral safety eval reduces to two error rates. Let $\mathcal{H}$ be a set of harmful requests, ones the model should refuse or answer only safely, and $\mathcal{B}$ a set of benign requests that may look sensitive. Then

In plain words, how often the model helps with what it should not, and how often it refuses what it should help with. On a jailbreak benchmark the first is the attack success rate of §19.3. The two trade off along exactly the frontier of §19.1, and a model that refuses everything scores perfectly on the first and terribly on the second. Reporting only the first is the most common way safety evaluations mislead.

Over-refusal benchmarks. XSTest (Röttger et al. 2024) is the standard small one. It has 250 safe prompts in ten categories built to trip a model that pattern-matches on surface features, such as homonyms (asking how to "kill" a computer process), figurative language, safe targets, safe contexts (a question about violence in a video game), definitions, historical events, and privacy questions about public or fictional people, plus 200 unsafe contrast prompts that look similar but should be refused. Responses are hand-labeled as full compliance, full refusal, or partial refusal. The paper's headline was that Llama 2 70B chat, with its original system prompt, fully refused 38% of the safe prompts and partially refused another 21.6%, while still refusing all the unsafe ones. GPT-4 struck the best balance of the models tested. OR-Bench (Cui et al. 2024) scaled the idea to 80,000 seemingly harmful but benign prompts and 600 genuinely harmful ones across 32 models, and found a Spearman rank correlation of 0.89 between a model's safety on the harmful prompts and its over-refusal on the benign ones. Across today's models, most of what looks like extra safety is bought with extra refusals.

Harmfulness benchmarks. HarmBench (§19.3) and StrongREJECT (Souly et al. 2024), with 313 forbidden prompts in six categories, are typical. StrongREJECT's contribution is its judge, which scores both whether the model was willing to answer and how useful the answer would be to someone with harmful intent. Under that judge, many jailbreaks that looked successful under looser metrics turned out to degrade the answers they extracted.

Response labels are not binary. Partial refusals, safe completions (answering the benign core of a request and declining the harmful part), and refusals that lecture the user are all different behaviors. Collapse them only after deciding what each means for your policy.

Error bars for rates

Section 0.11 built the statistics, and safety evals use them at their least forgiving end, rates near zero. Here is the short version. An eval gives $k$ positives out of $n$ prompts and the estimate $\hat p = k/n$. The textbook Wald interval $\hat p \pm z\sqrt{\hat p(1-\hat p)/n}$, with $z = 1.96$ for 95%, collapses to $[0, 0]$ when $k = 0$ and covers the truth far less than 95% of the time for rare events. The Wilson interval evaluates the standard error at the hypothesized rate instead of at $\hat p$ and keeps every rate the data do not reject, which centers it at $\big(\hat p + z^2/2n\big)/\big(1 + z^2/n\big)$, pulled toward $\tfrac12$, and keeps it inside $[0, 1]$ (§0.11 derives it, and Exercise 0.11d implements it). With zero failures its upper end is $\frac{z^2/n}{1 + z^2/n} \approx 3.84/n$, the two-sided cousin of the rule of three, $3/n$ for a one-sided 95% bound.

In plain words, zero failures in 50 prompts does not mean a zero failure rate. It means the rate is probably below about 7%. To claim a rate below 1% with zero observed failures you need roughly 380 prompts, and below 0.1% roughly 3,800. The cell applies §0.11's interval to the numbers that safety teams actually announce.

The opening example, 6/100 against 3/100, has Wilson intervals $[0.028, 0.125]$ and $[0.010, 0.085]$. At ten times the sample size, 60/1000 against 30/1000, the intervals $[0.047, 0.076]$ and $[0.021, 0.043]$ separate. And at $0/50$ the Wald interval is the useless $[0, 0]$ while Wilson gives $[0, 0.071]$.

import numpy as np

def safety_report(prompt_type, label):
    raise NotImplementedError

Comparing two models on the same prompts

When two models are run on the same prompts, the comparison is paired, and §0.11 showed why that matters. Only the discordant prompts, where exactly one model failed, carry information about which is safer. With $b$ prompts where only A failed, $c$ where only B failed, and $n$ in total, the difference in failure rates is $(b - c)/n$, its standard error is the standard deviation of the per-prompt differences over $\sqrt n$, which works out to $\frac1n\sqrt{(b + c) - (b - c)^2/n}$, and McNemar's exact test asks whether $c$ is a plausible $\operatorname{Binomial}(b + c, \tfrac12)$ draw (Exercise 0.11e implements both, with a paired bootstrap). Hard prompts tend to defeat both models, so pairing usually shrinks the error a lot. For the opening announcement, here is what pairing does when the two models share most of their failures.

Pairing cuts the standard error by about a quarter, and still nothing is shown, because five discordant prompts are five coin flips. The cure for a rare-event comparison is more prompts, and §0.11's power formula says how many. To detect a 3-point drop with 80% power when about 5% of prompts are discordant takes about $\big(2.80 \cdot \sqrt{0.05}/0.03\big)^2 \approx 440$ paired prompts, and a 1-point drop about 3,900.

What is different about generative models. Safety evals sample their answers, often several per prompt, and their prompts come in clusters, the same harmful behavior wrapped in a dozen jailbreak templates, or one agentic scenario run with many seeds. Section 14.7 worked out what both do to an error bar. Sampled answers add a within-prompt variance term that more samples shrink and more prompts shrink too, so treating every sample as independent divides by the wrong $\sqrt n$. And prompts that share a behavior or a template fail together, so the honest interval is the clustered standard error, which sums residuals within each cluster before squaring and can be several times the naive one. For a jailbreak benchmark, cluster by the underlying behavior, not by the attacked prompt, because a hundred wrappers of one request are closer to one data point than to a hundred.

The judge is a measuring instrument

Nearly every number above came from a judge, a person, a classifier, or a language model deciding whether each response was harmful. Early papers used string matching ("I'm sorry", "I cannot") to detect refusals, which counts a refusal that then complies anyway as safe and a helpful answer that begins with an apology as a refusal. Modern evals use LLM judges, which are better but not perfect, and their errors bias the rates in a predictable direction.

Suppose the judge has sensitivity $s_e$ (the probability it flags a truly harmful response) and specificity $s_p$ (the probability it passes a truly harmless one). If the true harmful rate is $p$, the rate it reports is

In plain words, the observed rate is the true positives it catches plus the false alarms on everything else. Solving for $p$ gives the Rogan-Gladen correction,

valid when $s_e + s_p > 1$ (a judge better than a coin) and clipped to $[0, 1]$. The false-alarm term $(1 - s_p)(1-p)$ dominates when $p$ is small. A judge with 95% specificity reports at least about 5% harmful even for a model that is never harmful, so on a strong model most of a reported ASR can be judge error. Estimate $s_e$ and $s_p$ by having people label a few hundred responses, and propagate their uncertainty too, since an error in $s_p$ moves $\hat p$ one for one.

There is a second way to use the same human labels, which §14.7 built for judge win rates, prediction-powered inference. Judge every response, average the human-minus-judge difference on a random human-labeled subset, and add that correction to the judge's overall rate. The result is unbiased for the human rate with no model of the judge's errors at all. The two corrections suit different situations. Prediction-powered inference needs the labeled responses to be a random sample of the very eval being scored. Rogan-Gladen needs only that the judge's sensitivity and specificity carry over, which lets a team measure them once on a pooled labeled set and apply them to a new model whose true rate is different, at the risk that a new model's failures look different to the judge.

import numpy as np

def corrected_rate(q, sens, spec):
    return q                                  # placeholder: trusts the judge

def judge_error_rates(judge, human):
    raise NotImplementedError

import numpy as np

def corrected_rate_ci(judge_eval, judge_cal, human_cal, n_boot, rng, level=0.95):
    q = float(np.mean(judge_eval))
    return q, q, q                            # placeholder: trusts the judge, no uncertainty

Evaluating for extreme risks

Refusal evals ask whether a model will help with a harmful request. For the most severe risks the more important question is whether it could, if its safeguards were removed or bypassed, which is what an attacker with stolen weights, a fine-tuning API, or a working jailbreak would get. Shevlane et al. (2023) drew the distinction that frontier labs now use. Dangerous capability evaluations measure what a model is able to do in areas such as biological and chemical weapons, offensive cyber operations, persuasion, and autonomous operation. Alignment evaluations measure its propensity to use those capabilities for harm. A model can pass the second and fail the first, and a model that fails neither today may fail the first tomorrow.

Some examples show the range.

Phuong et al. found no strong dangerous capabilities in Gemini 1.0 but flagged early warning signs, which is the typical result for a given year's frontier and the reason these evaluations are repeated for every new model.

Reading a capability trend, the 50% time horizon

The METR measurement is the most quoted trend in the field, and it is worth knowing exactly how it is computed, because it is a fitted curve and not a count. Kwa et al. (2025) timed skilled professionals on 170 software and research tasks, from a few seconds to many hours, and took each task's length $t$ to be the geometric mean of the successful human times. For each model they fit a logistic curve of success against the logarithm of task length,

where $h$ is the model's 50% time horizon, the task length at which the curve crosses one half, and $\beta$ says how sharply success falls off around it. In plain words, a model with a one-hour horizon succeeds on about half of the tasks that take an expert an hour, more often on shorter ones and less often on longer ones. Fitting it is ordinary logistic regression, the same fit as the Bradley-Terry reward models of §16.3, on the single feature $\log_2 t$. If the fit is $\sigma(w_0 + w_1\log_2 t)$, then $w_1 = -\beta$, and the curve crosses a success rate $u$ where $w_0 + w_1\log_2 t = \log\frac{u}{1-u}$, so the $u$ horizon is $h_u = 2^{(\log\frac{u}{1-u} - w_0)/w_1}$. The trend is then a straight line through $\log_2 h$ against release date, whose slope is doublings per year. Across models released since 2019 the horizon doubled roughly every seven months, and Claude 3.7 Sonnet's was about 50 minutes (Kwa et al. 2025). The cell reproduces the pipeline on simulated models whose horizon doubles every seven months.

The fit recovers a 50-minute horizon as 52 minutes and the seven-month doubling as 7.3 months. Three details matter when you read the real numbers. The 80% horizon is several times shorter than the 50% one (15 minutes here, and several-fold shorter in the paper too), so "can do hour-long tasks" means "half the time". The oldest models' horizons, a few seconds, lie below the shortest task in this toy set, so they are extrapolations of the fitted curve, which is why METR added very short tasks to its suite. And the uncertainty is clustered, since tasks come in families that succeed or fail together and each task is run several times, so METR's intervals come from a bootstrap over task families, tasks, and runs, the clustered resampling of §14.7 (they also down-weight large task families). A doubling time with an interval of a few months either way still says that the horizon grows about tenfold every two years, which is what makes the trend decision-relevant.

import numpy as np
from sklearn.linear_model import LogisticRegression

def time_horizon(minutes, success, level=0.5):
    m = np.asarray(minutes)[np.asarray(success) == 1]
    return float(np.median(m))                # placeholder: the median length of the tasks it solved

def doubling_time_months(release_years, horizons):

Two properties make dangerous capability evals different from benchmarks. First, they are lower bounds. A capability eval measures what the evaluators managed to elicit, with their prompts, tools, and time, and the capability-versus-behavior distinction of §14.1 applies with full force. More scaffolding, more samples, or a little fine-tuning can raise the score a lot, so serious evaluations spend real effort on elicitation, and a model that deliberately underperforms (§19.6) would defeat a careless one. Second, the questions themselves can be hazardous. WMDP's questions were written to measure precursor knowledge without publishing a recipe, and the hardest evaluations are kept private.

Frontier safety frameworks

How do evaluations become decisions? Since 2023 the major labs have published frameworks with a shared structure, which METR's survey of them calls the common elements of frontier AI safety policies.

1. Capability thresholds. Levels of dangerous capability, defined in advance, at which current safeguards would no longer be adequate.
2. Evaluations run as models are developed, including during training, designed to trigger before a threshold is crossed, with a safety buffer for the capability gained between evaluations and for weak elicitation.
3. Required safeguards for each threshold, covering both deployment (misuse protections such as the classifiers of §19.7) and security (protecting the weights from theft).
4. A commitment to pause training or deployment if the safeguards cannot be put in place.

Karnofsky (2024) called this structure an if-then commitment. If a model reaches capability X, then safeguards Y must be in place before proceeding. The appeal is that it lets people who disagree about when dangerous capabilities will arrive agree on what to do when they arrive.

The specific frameworks differ in their categories and vocabulary, and they are revised often, so read the details below as a snapshot of the published versions as of late 2025, and check each lab's current document before quoting one in an interview. Anthropic's Responsible Scaling Policy (first published September 2023 and revised several times since) defines AI Safety Levels modeled loosely on biosafety levels, each with required safeguards, and in May 2025 Anthropic activated its ASL-3 protections for Claude Opus 4 as a precautionary step, because it could not rule out that the model had crossed the relevant threshold. OpenAI's Preparedness Framework (beta December 2023, version 2 April 2025) tracks biological and chemical capability, cybersecurity, and AI self-improvement, with High and Critical thresholds. Google DeepMind's Frontier Safety Framework (May 2024, revised twice in 2025) defines Critical Capability Levels with early-warning evaluations. Two cautions apply to all of them. They are self-imposed, so their force depends on how they are interpreted and enforced. And thresholds expressed in terms of evaluations inherit every pitfall in the last section of this page.

Uplift studies

A model that scores well on a hazardous-knowledge quiz may still add nothing to what a determined person could find with a search engine. The question that matters for misuse is marginal uplift, how much more likely someone is to succeed at a harmful task with the model than without it. That can only be measured with people, in a randomized controlled trial. Participants are randomly assigned to a control arm with the usual resources (the internet) or a treatment arm with the same resources plus the model, attempt a task (writing a plan, answering questions about a protocol), and experts score the results blind to the arm.

Two published studies illustrate the design. RAND (Mouton et al. 2024) had 15 red-team cells of three people work for seven weeks on plans for biological attack scenarios, some with internet access only and some with one of two language models as well, and expert adjudicators scored each plan's viability. They found no statistically significant difference between plans made with and without a model. OpenAI (2024) gave 100 participants (50 biology PhDs with wet-lab experience and 50 students) tasks covering the stages of a biological threat, with a research version of GPT-4 that did not refuse for the experts. Scores rose slightly with model access (for the experts, accuracy rose 0.88 points on a 10-point scale), and none of the differences was statistically significant.

"Not statistically significant" is the phrase to read carefully. A study can fail to find an effect because there is none, or because it was too small to see one. Before running a study you can compute how many participants it needs. Section 0.11 derived the sample size for a paired comparison, two models on the same questions, where only the discordant items add noise. An uplift trial cannot pair, because each arm is a different group of people, so the variances of the two arms add. Running the same derivation with the test statistic standardized by the pooled variance under the null and by the separate variances under the alternative gives the standard sample size per arm for comparing two proportions, for a two-sided test at level $\alpha$ with power $1-\beta$ to detect a change in success rate from $p_0$ to $p_1$,

where $z_q$ is the $q$ quantile of the standard normal ($z_{0.975} = 1.96$, $z_{0.8} = 0.84$). In plain words, the required sample grows with the inverse square of the effect you want to detect. Halve the effect and you need four times the participants.

With 50 participants per arm, a study can reliably detect a jump in task success from 30% to 60%, but it needs about 93 per arm to detect 30% to 50%, about 356 for 30% to 40%, and far more for small changes in rare outcomes. A null result from a small study rules out only large effects. That is not a criticism of the studies above, which were careful and said so, but it is the right way to read their conclusions, and it is why uplift studies are combined with capability evals rather than replacing them.

import numpy as np

def n_per_arm(p0, p1, z_a=1.959964, z_b=0.841621):
    raise NotImplementedError

def min_detectable_p1(p0, n, z_a=1.959964, z_b=0.841621):
    raise NotImplementedError

How safety evaluations mislead

Lenient or brittle judges. String-matching refusal detectors and permissive "did it refuse?" judges mismeasure both axes. Validate the judge against people and correct for its error rates.

Too few prompts, counted wrongly. Rare events need large samples, and comparisons need paired designs. Report Wilson intervals, and do not announce differences that live inside them. A hundred jailbreak wrappers of ten behaviors are closer to ten data points than to a thousand, so cluster by behavior when computing the error bar (§14.7).

One number. Harmfulness without over-refusal hides the cost of safety, and over-refusal without harmfulness hides its benefit. OR-Bench's 0.89 correlation says the two usually move together.

Measuring capability, calling it safety. Ren et al. (2024) correlated dozens of safety benchmarks with a general capability score across models and found that many, including TruthfulQA and ETHICS, track capability at correlations around 0.8 (0.81 and 0.82 across chat models). Improvements on such benchmarks can come from making the model generally smarter, and presenting such gains as safety progress is what they called safetywashing. Others, such as sycophancy ($-0.67$) and robustness to several jailbreak attacks, were negatively correlated with capability, so they measure something that capability alone does not fix and can even get worse as models improve. Before trusting a safety benchmark, check its correlation with capability.

Weak elicitation. A dangerous capability eval run with a bare prompt and one sample underestimates what the model can do with tools, many samples, and fine-tuning. A low score is evidence only to the extent the elicitation was strong.

Evaluation awareness. Models can often tell when they are being tested. Needham et al. (2025) built a benchmark of 1,000 transcripts from 61 datasets, half from evaluations and half from real use, and found that Gemini 2.5 Pro distinguished them with an AUC of 0.83, against 0.92 for human experts. If a model behaves differently when it believes it is being evaluated, evaluations measure the evaluated behavior, a problem §19.6 returns to.

Contamination. Public safety benchmarks end up in training data, and a model can learn to refuse the specific benchmark prompts. Keep held-out private sets, and use contamination checks and canaries of the kind §19.7 builds.

Read more

• Röttger et al. (2024). XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Models. The standard over-refusal test suite and its findings.
• Souly et al. (2024). A StrongREJECT for Empty Jailbreaks. Why jailbreak evaluation needs a judge that scores answer quality.
• Shevlane et al. (2023). Model evaluation for extreme risks. Dangerous capability and alignment evaluations, and how they fit into governance.
• Phuong et al. (2024). Evaluating Frontier Models for Dangerous Capabilities. A full program of dangerous capability evaluations on a frontier model.
• Li et al. (2024). The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning. A hazardous-knowledge proxy benchmark and an unlearning method.
• Kwa et al. (2025). Measuring AI Ability to Complete Long Software Tasks. The time-horizon metric and its doubling trend.
• Ren et al. (2024). Safetywashing: Do AI Safety Benchmarks Actually Measure Safety Progress?. Which safety benchmarks are capability benchmarks in disguise.
• Karnofsky (2024). If-Then Commitments for AI Risk Reduction. The logic behind frontier safety frameworks.

Watch next

▶

▶