Chapter 22 · Section 22.3 · 85 min

Contamination

Contamination means that a model has seen the test before it is scored. The test questions, their answers, or close copies of them were in its training data, or the model looked them up while it was being tested. It matters because a contaminated score looks exactly like an honest one. A model that memorized the answers and a model that can solve the problems post the same number, and the number alone cannot tell you which one you have.

Two experiments from 2024 show how large the effect can be. The first wrote a new copy of a benchmark. GSM8K is a set of grade-school math word problems, and nearly every lab reports its 1,319-question test set. Scale AI paid writers to produce 1,205 new problems in the same style, matched to GSM8K in how often people solve them, how many steps they take, and how large their answers are. The new set, GSM1k, was kept private (Zhang et al. 2024). A model that has learned grade-school math should score about the same on both sets. Several did not. The worst models lost up to 8 points on GSM1k. The Phi and Mistral families lost points at almost every size and version, and the Yi, Xwin, Gemma, and CodeLlama families showed the same pattern more weakly. Across models, the gap grew with how likely the model was to write out GSM8K's own test problems.

The second experiment used the calendar. LiveCodeBench collects programming-contest problems and tags each one with the date it was released (Jain et al. 2024). DeepSeek's 33-billion-parameter base model for code passed about 60% of the LeetCode problems released in May 2023. It passed almost none of those released in September 2023, the month the DeepSeek models came out. Problems from a second contest site, AtCoder, showed no such drop. The likely reading is that the older LeetCode problems were in its training data.

§22.1 treated an evaluation as an estimator. The benchmark score $\hat\mu$ , the model's average grade on the published items, is meant to estimate $\mu_{\text{bench}}$ , its expected grade on fresh items written the same way, items it cannot have seen. Average $\hat\mu$ over everything random in it, the draw of items and the sampled answers. When that average differs from $\mu_{\text{bench}}$ , the measurement is biased. Contamination is the part of that bias that comes from the model having met the test items before.

This section answers three questions you can ask from outside the lab that trained the model. Did the model see the test? Can you prove it to a skeptic, with a p-value that means what it says? And how many points of the score did the leak buy? Each question has its own tools, and each tool is blind to some kinds of leak, so we start by naming the kinds.

What leaks, and how

Think of a student who saw the exam beforehand. They might have seen the exact paper, the paper with the answers filled in, a study guide that rephrased the questions, a translated copy, or only the answer key. Each leaves a different trace. A student who memorized the paper can recite the questions word for word. One who studied a rephrased guide cannot recite them, but still gets them right. One who saw only the key can give the answers but not the questions. A detective who checks only for word-for-word recitation catches the first student and misses the rest.

Language models leak in the same ways, and in a few more. Benchmarks are posted on the web, quoted in papers, pasted into homework forums, and copied into instruction datasets. A model that saw a benchmark can write synthetic training problems that carry pieces of it into a second model, one that never saw the original (§8.4). The risk compounds when models train on their own outputs, as in synthetic-data and self-training loops. And an agent that can browse the web may look the answer up while it is being tested.

📐 Contamination and the leakage bias

Let a benchmark have items $x_1, \dots, x_n$ with reference answers $a_1, \dots, a_n$ , drawn from a distribution $\mathcal{D}_{\text{bench}}$ . Item $i$ is contaminated (or dirty) for a model if information specific to $(x_i, a_i)$ , which a fresh draw from $\mathcal{D}_{\text{bench}}$ would not carry, reached the model before or during its evaluation. Items that are not dirty are clean. The main kinds of leak are

verbatim questions, the text of $x_i$ without its answer,
questions with answers, $x_i$ together with $a_i$ or a worked solution,
paraphrases and translations of $x_i$ , with or without the answer,
answers only, an answer key tied to item identities ("problem 17, answer 204"),
derived data, such as synthetic problems written by a model that had seen the benchmark, or training text selected because it resembles the benchmark,
order and metadata, the benchmark file in its published order, its name, its wrong answer options,
test-time leaks, answers reached during the evaluation itself, by a browsing agent or through a retrieval index that contains the benchmark.

Let $\mu_{\text{bench}}$ be the model's expected score on fresh draws from $\mathcal{D}_{\text{bench}}$ (items it cannot have seen), and $\hat\mu$ its score on the published items, both graded correctly in a working environment. The leakage bias is

\Delta_{\text{leak}} = \mathbb{E}[\hat\mu] - \mu_{\text{bench}} .

In plain words, the leakage bias is how much better the model does on the items that exist than it would do on new items written the same way. It is usually positive. Nothing in the definition forces a sign, though, and a model can see an item and gain nothing from it.

That last point is the most common confusion in this area. Seeing an item is not the same as benefiting from it. Magar and Schwartz (2022) tested this directly. They pretrained BERT encoders (§7.1) on Wikipedia mixed with labeled examples from downstream tasks, then fine-tuned them on each task. Comparing examples seen in pretraining with examples not seen, they measured two separate effects. Memorization means the model recognizes the leaked examples. Exploitation means it scores higher on them. Some tasks showed both. Others showed memorization with no exploitation. How strong each effect was depended on factors such as the number of copies and the size of the model.

So every audit has two questions. Detection asks whether an item was seen. Effect asks how much seeing it helped. Here is how the kinds of leak meet the detectors this section builds. Read the table now as a map, and again at the end as a summary.

kind of leak	n-gram overlap with training data	per-example membership inference	order test	fresh, perturbed, or time-split items
verbatim questions	yes	weakly, needs many copies	yes, if the file kept its order	yes, if the copy helped
questions with answers	yes	weakly	yes, if ordered	yes
paraphrases, translations	rarely	barely	rarely	yes
answers only	no	no	no	yes
derived synthetic data	no	no	no	yes
test-time leaks	no	no	no	only if fresh items are not online
needs	the training data	log-probabilities	log-probabilities, an exchangeable benchmark	new items of the same difficulty

The last column is different in kind. The first three detectors look for the cause, traces of the items in the model or in its data. The last one measures the effect, a score that does not carry over to new items. Only the effect matters for the leakage bias. Only the causes can be checked without writing new items. An interviewer who asks "how would you check for contamination" wants to hear both halves.

Overlap checks, and what they cannot see

If you have the training data, the first check is a text search. An n-gram is a run of $n$ consecutive words. The overlap check of §8.3 lowercases every text, strips its punctuation, lists every n-gram of every test item, and looks each one up in the training data. Two rules turn the matches into a verdict.

The any-match rule flags a test item if even one of its n-grams appears in training. GPT-3's contamination study used it, with $n$ set per benchmark between 8 and 13 words (Brown et al. 2020).
The coverage rule flags an item when more than half of its tokens sit inside 8-grams that match one single training example. Tulu 3 used it on its fine-tuning data (Lambert et al. 2024), as §14.2 describes.

Decontamination is the fix on the training side. You remove every training document that overlaps a benchmark you plan to report. §14.2 also showed the hole in it. Yang et al. (2023) trained 13-billion-parameter models on rephrased and translated copies of test sets, Llama 2 for MMLU (multiple-choice questions in 57 subjects) and GSM8K, and CodeLlama for HumanEval (Python functions written from a description). The copies passed n-gram decontamination, and the models matched or beat GPT-4 on all three. Both earlier sections built these tools. Here we look at what their reports mean.

✏️ Five items and three crawled pages

A benchmark has five word problems, and a web crawl has three pages. Page 1 is a homework thread that quotes item 1 word for word, with its answer. Page 2 paraphrases item 2 ("every night Maya gets through twelve pages") and then quotes the first sentence of item 3. Page 3 is an answer key for all five items, and it repeats item 5's instruction "round your answer to the nearest whole number of cents". For each item, count how many of its 8-grams appear in the crawl and what fraction of its words they cover.

An item of $w$ words has $w - 8 + 1$ word 8-grams. Item 1 has 20 words, hence 13 8-grams, and all 13 are on page 1, so its coverage is 100%. Item 2's paraphrase shares no run of eight words with it, so 0 of its 8 8-grams match. Item 3 has 19 words and shares its first 12 with page 2. Those 12 words contain $12 - 8 + 1 = 5$ of its 12 8-grams and cover $12/19 = 63\%$ of its words. Item 4's question appears nowhere, 0 of 17. Item 5 has 25 words and shares the 10-word instruction, which gives 3 of its 18 8-grams and covers $10/25 = 40\%$ of its words.

The cell checks the counts. Each item's matches all come from one page, so the coverage against the whole crawl equals the coverage against a single training example.

Python · cell 1

Now read the output as an auditor would. The any-match rule with $n = 8$ flags items 1, 3, and 5. Item 5 is a false positive. Its question never leaked, and the match is a stock phrase. The coverage rule (more than 50%) clears it and still flags items 1 and 3. Item 2 is a false negative under both rules, a leaked item that passes, because the paraphrase changed every run of eight words. And the answer key on page 3 leaked all five answers without tripping either rule, since an n-gram check reads only the question text.

Three lessons carry over to real audits. The threshold trades false alarms against misses, and both are real. A clean report covers only what the check can see. And none of this works without the training data, which rules it out for every closed model.

Paraphrases have a practical fix (Yang et al. 2023). Embed each test item as a vector, retrieve its nearest training examples, and ask a language model whether each pair is a rephrasing. With that check, the authors found rephrasings of 8.5% of HumanEval's 164 problems in a sample of RedPajama-Data-1T and of 15.9% in a sample of StarCoder-Data, two public pretraining sets.

Membership inference, a test for one example

Without the training data, the only witness is the model. Membership inference asks it a narrow question about one piece of text. Was this exact text in your training set? A member is a text that was in the training data, and a non-member is one that was not. We frame the question as a hypothesis test and grade it with the rates of §0.11. The true positive rate is the fraction of real members that the attack flags. The false positive rate is the fraction of non-members it flags by mistake.

📐 Membership inference as a test

For a text $x$ and a model, the null hypothesis $H_0$ says $x$ is a non-member, and the alternative $H_1$ says it is a member. A membership inference attack is a score $s(x)$ , oriented so that larger means "more likely a member", and a threshold $t$ . It flags $x$ when $s(x) \ge t$ . On a labeled evaluation set of members and non-members, its true positive rate is $\mathrm{TPR}(t) = \mathbb{P}(s \ge t \mid \text{member})$ and its false positive rate is $\mathrm{FPR}(t) = \mathbb{P}(s \ge t \mid \text{non-member})$ . It is summarized by the area under the ROC curve,

\text{AUC} = \mathbb{P}(s^+ > s^-) + \tfrac12\mathbb{P}(s^+ = s^-),

for a random member score $s^+$ and an independent random non-member score $s^-$ , and by the TPR at a fixed low FPR such as 1% or 5%.

In plain words, an attack ranks texts by how familiar they look to the model. Its AUC is how often it ranks a real member above a real non-member, with ties counted as half. The AUC alone can mislead (Carlini et al. 2022). An attack can reach a decent AUC by being slightly better than chance on every example while being unable to flag any single example with confidence. A contamination audit needs the confident flags, so report the TPR at a low FPR as well.

The simplest score is the loss attack, the mean log-probability of the text's tokens. For a text of $T$ tokens it is

\frac{1}{T}\sum_{t=1}^{T}\log p_\theta(x_t \mid x_{\lt t}),

where $p_\theta(x_t \mid x_{\lt t})$ is the model's probability of token $t$ given the tokens before it. This is the held-out likelihood of §6.6. Training raises the likelihood of training text, so members should score higher. The trouble is that some texts are likely under any model, such as a license header or a list of numbers. Others are unlikely under any model, such as a page of rare names. So the raw likelihood mixes "seen in training" with "easy to predict". Better attacks remove the second part, and two are standard.

The zlib ratio calibrates against a generic compressor. Text that zlib also compresses well is easy for everyone, so dividing by its compressed length cancels part of the "easy" signal. Carlini et al. (2021) used it to pull memorized text out of GPT-2, the extraction attack of §19.7. We use the total log-likelihood over the compressed length in bytes,

\frac{\sum_t \log p_\theta(x_t \mid x_{\lt t})}{\lvert\operatorname{zlib}(x)\rvert} .

Up to the constant $8\ln 2$ , this is minus the ratio of the model's code length for $x$ to zlib's. Carlini et al. divide the per-token mean instead of the total. The two differ by the token count, and implementations vary, so say which one you use.

Min-K% Prob calibrates against the text itself (Shi et al. 2023). Its hypothesis is that an unseen text contains a few outlier tokens with very low probability, and a seen text has fewer of them. So it averages only the lowest-probability tokens.

Why should the low tail carry the signal? Because one training step raises low probabilities far more than high ones. Take a token the model gives probability 0.9 and another it gives 0.01. The result below says one gradient step of size $\eta$ adds at most $2\eta(0.1)^2 = 0.02\eta$ to the first token's log-probability. It adds at least $\eta(0.99)^2 \approx 0.98\eta$ to the second's, both to first order, at least 49 times as much. The cleanest version of the argument is for a step taken directly on the logits, the scores that the softmax turns into probabilities.

∂ Proof of the one-step bound

From the softmax derivative of §0.2, $\partial\log p_y/\partial z_j = \mathbb{1}[j = y] - p_j$ , so $\nabla_{\mathbf{z}}\log p_y = \mathbf{e}_y - \mathbf{p}$ , where $\mathbf{e}_y$ is the one-hot vector for $y$ . The step is $\Delta\mathbf{z} = \eta(\mathbf{e}_y - \mathbf{p})$ . The first-order change is the gradient dotted with the step, $\eta\lVert\mathbf{e}_y - \mathbf{p}\rVert^2 = \eta\big((1-p_y)^2 + \sum_{j\ne y}p_j^2\big)$ . The sum is at least 0. It is at most $\big(\sum_{j\ne y}p_j\big)^2 = (1 - p_y)^2$ , because the $p_j$ are nonnegative. That gives both bounds. ∎

In plain words, a token the model already predicts with probability near 1 gains almost nothing from being trained on. A token it found surprising gains between about $\eta$ and $2\eta$ in log-probability. So training on a text erases its surprises first. The mean log-probability averages those gains with many near-zero gains on easy tokens, which look the same for members and non-members. The low tail concentrates them. A real step moves the parameters, not the logits, and it changes other positions too. So this is the idealization behind the heuristic, not a guarantee.

🔬 Min-K%++, scoring a token against what the model expected

Zhang et al. (2024) push the calibration one step further. Min-K%++ scores each token by how far its log-probability sits above what the model expected at that position,

\frac{\log p_\theta(x_t \mid x_{\lt t}) - \mu_t}{\sigma_t}, \qquad \mu_t = \mathbb{E}_{v \sim p_\theta(\cdot \mid x_{\lt t})}\big[\log p_\theta(v \mid x_{\lt t})\big].

Here $\mu_t$ is minus the entropy of the next-token distribution, and $\sigma_t$ is the standard deviation of $\log p_\theta(v \mid x_{\lt t})$ under the same distribution. It then averages the lowest fraction $f$ of those scores. A token is suspicious if it is a peak of the model's prediction, not merely likely. On WikiMIA, a benchmark described below, the authors report AUC gains of 6.2 to 12.3 points over the runner-up, Min-K% Prob, averaged over five models, with the size of the gain depending on the length of the texts. The PyTorch block near the end of the section computes it.

Core

Min-K% Prob and the zlib ratio

Implement three membership scores and their evaluation. All three scores must be oriented so that a higher score means "more likely a member".

min_k_prob(token_logprobs, frac=0.2) returns $\operatorname{MinK}_f$ , the mean of the $\max(1, \lceil fT\rceil)$ lowest of the $T$ given log-probabilities. Treat $fT$ as exact, so a product that lands on a whole number in exact arithmetic is not rounded up by floating-point error.
zlib_ratio(logprob_sum, text) returns the total log-likelihood divided by the length in bytes of zlib.compress(text.encode("utf-8")).
auroc(scores, labels) returns the ROC AUC of scores for binary labels (1 for a member), counting ties as one half, without calling a library AUC function. It raises ValueError if either class is empty.

import zlib
import numpy as np

def min_k_prob(token_logprobs, frac=0.2):
    raise NotImplementedError

def zlib_ratio(logprob_sum, text):
    raise NotImplementedError

Tests, hints, verified solutions, and AI help, with Pro at $7.99 a month.

A membership test you can run

Let us try the three scores on a model small enough to train in a cell. The model is a character 5-gram model, one of the count-based language models of §3.1. It predicts each character from the four before it, using counts from its training text. Add- $k$ smoothing adds a small count $k$ to every possible next character, so a character never seen after a context still gets a small probability. The training text is 1,500 machine-written word problems plus half of a 300-item benchmark of the same kind. That half was chosen at random and leaked once. Members and non-members come from the same generator, so the only difference between them is membership.

Python · cell 2

All three attacks work, with AUCs between 0.69 and 0.73. All three are also weak in the sense that matters. At a 5% false positive rate they catch about one member in ten. If you had to say of one particular item "this was in the training set", you could not.

These AUCs happen to match the range reported on WikiMIA (Shi et al. 2023), a benchmark built to test these attacks on real models. There, on the original texts, the loss and zlib attacks scored 0.70 to 0.72 on LLaMA-30B and LLaMA-65B, and Min-K% Prob scored 0.74. That looked like a success. It was not quite one.

Why per-example membership inference fails on pretraining data

WikiMIA's members are Wikipedia articles about events, created before 2017, which many models trained on. Its non-members are articles about events, created in 2023 or later, which those models cannot have seen. That split guarantees the labels are right. It also makes the two groups talk about different things.

Das, Zhang, and Tramèr (2024) asked whether you need the model at all. They built blind baselines, classifiers that never query any model and look only at the text. A bag-of-words classifier separates WikiMIA's members from its non-members with an AUC of 0.99 and a TPR of 94.7% at 5% FPR. The best published attack reached 0.84 and 43.2%. On all 8 published membership-inference evaluation sets they examined, a blind baseline beat the best attack.

Here is the same thing in miniature. The non-members now come from a "later" generator whose problems drift to a different mix of topics. The model has seen every one of those topics in its training text. The blind baseline is a naive Bayes classifier. It learns from half the items how much more often each word appears in members than in non-members, and it scores a text by adding up those log-odds.

Python · cell 3

The blind classifier reaches an AUC of 0.92 and beats every attack that queried the model. It learned only that "eggs" and "cards" appear in later problems and "apples" and "books" in earlier ones. On an evaluation set where a blind classifier wins, you cannot tell how much an attack learned from the model, because the attack can score well by detecting the drift.

Duan et al. (2024) ran the clean experiment. They used Pythia, a suite of open models from 160 million to 12 billion parameters trained on the Pile, an open corpus of about 800 GB of text. They drew members and non-members from the same domains and the same time, and found almost every attack near chance.

📌 Per-example membership inference on pretraining data is near chance

On Pythia models trained on the Pile, no attack that Duan et al. (2024) tested reached an AUC above 0.6 in any domain except GitHub code. Most were within a few points of 0.5. They trace the failure to two properties of pretraining. The dataset is enormous and seen for about one epoch. And the boundary between members and non-members is fuzzy. Non-members from the Pile's Wikipedia and arXiv test sets share over 30% of their 7-grams with training documents on average. Earlier successes came from evaluation sets whose non-members were shifted in time, and Das et al. (2024) showed that blind baselines beat published attacks on all 8 evaluation sets they examined. A membership-inference result reported without a same-distribution non-member set and a blind baseline is not evidence.

In an interview, this is the answer to "why not just run Min-K% on the benchmark?" On one item seen once, it cannot tell you anything you can defend.

One weak signal, many items

An AUC of 0.55 is useless for one item and can be decisive for a thousand. A benchmark audit does not need to classify items. It only needs to decide whether the benchmark as a whole was seen, and weak evidence on each item adds up across many items. The next result says how fast.

∂ Proof of the AUC and z-test formulas

For independent $s^+ \sim \mathcal{N}(\mu_1, \sigma^2)$ and $s^- \sim \mathcal{N}(\mu_0, \sigma^2)$ , the difference $s^+ - s^-$ is $\mathcal{N}(\mu_1 - \mu_0, 2\sigma^2)$ . So $\mathbb{P}(s^+ > s^-) = \Phi\big((\mu_1 - \mu_0)/(\sqrt2\sigma)\big) = \Phi(d'/\sqrt2)$ , and ties have probability zero. The difference of two means of $n$ scores each has mean $\mu_1 - \mu_0$ and variance $2\sigma^2/n$ . So the $z$ statistic has mean $(\mu_1 - \mu_0)/(\sigma\sqrt{2/n}) = d'\sqrt{n/2}$ . ∎

The separation $d'$ is fixed by the attack, but the expected $z$ grows with the square root of the number of items. So an attack barely better than chance becomes decisive on a large enough set, as the next example shows.

The cell checks the arithmetic and then runs 500 simulated audits of that size.

Python · cell 4

The simulated audits average a $z$ near 4, and 93% of them reach $p \lt 0.01$ . A signal too weak to flag any single item is strong enough to convict the set. This is the idea behind dataset inference (Maini et al. 2024). It combines several attacks that each carry a little signal for the data at hand, and it runs one statistical test over a whole set of documents. It separated the training and test splits of different Pile subsets with p-values below 0.1 and no false positives.

The catch is in the theorem's setup. You need non-members from the same distribution, such as a held-out split written at the same time. That is exactly what membership-inference evaluations kept getting wrong. The next test removes the need.

The order test, which proves contamination

A benchmark is published as a file, and a file has an order. Suppose the order carries no meaning, so any shuffle of the items is as natural as the published one. A model that never saw the file then has no reason to prefer the published order. A model that trained on the file, start to finish, has learned that item 18 follows item 17, and it does prefer it. Oren et al. (2023) turned that asymmetry into a test with a guaranteed false positive rate, using nothing but log-probabilities.

Here is the idea with three items, A, B, and C, published in the order ABC. They can be arranged in $3! = 6$ orders. Score each order by the model's log-probability of the three items written out in that order. If the model never saw the file, the published order ABC is just one of six orders, and it lands on top with probability $1/6$ . A model that read ABC many times puts it on top every time. A file of three items can never give a p-value below $1/6$ , which is why real tests use many items and many shuffles.

The argument needs one property of the benchmark, defined next.

The test is a lottery among orderings. Score the published order and $P$ random shuffles by the model's log-probability of the whole file. If the model never saw the file, the published order is one more random order, and it lands at the very top with probability $1/(P+1)$ . In symbols, write $\ell(X) = \log p_\theta(\operatorname{seq}(X))$ for the model's log-probability of the examples concatenated in the given order, with a fixed separator between them. Draw independent uniformly random permutations $\varsigma_1, \dots, \varsigma_P$ , and set $\ell_0 = \ell(X)$ and $\ell_k = \ell(X_{\varsigma_k})$ . The null hypothesis is that the model's parameters $\theta$ are independent of $X$ , which is what "the model never saw this benchmark" means statistically. The permutation p-value is

\hat p = \frac{1 + \#\{k \in \{1, \dots, P\} : \ell_k \ge \ell_0\}}{P + 1} .

In plain words, the p-value is the published order's rank among all $P + 1$ orderings, counting from the most likely, divided by $P + 1$ . Ties are counted against the published order.

∂ Proof that the permutation p-value is valid

Step 1, the $P + 1$ orderings are exchangeable. Let $\varsigma_0$ be one more uniform permutation, independent of everything else. For a fixed $\varsigma_0$ , exchangeability gives $X_{\varsigma_0} \overset{d}{=} X$ , and $\theta$ and the shuffles are independent of $X$ , so $(\theta, X, \varsigma_1, \dots, \varsigma_P) \overset{d}{=} (\theta, X_{\varsigma_0}, \varsigma_1, \dots, \varsigma_P)$ . Reordering a reordered dataset composes the permutations, $(X_{\varsigma_0})_{\varsigma_k} = X_{\varsigma_0\circ\varsigma_k}$ , so

(X, X_{\varsigma_1}, \dots, X_{\varsigma_P}) \overset{d}{=} (X_{\varsigma'_0}, X_{\varsigma'_1}, \dots, X_{\varsigma'_P}), \qquad \varsigma'_0 = \varsigma_0,\ \varsigma'_k = \varsigma_0\circ\varsigma_k .

Given $\varsigma_0$ , each $\varsigma'_k$ for $k \ge 1$ is uniform, because composing with a fixed permutation is a bijection of the permutation group, and they are independent. So $\varsigma'_0, \varsigma'_1, \dots, \varsigma'_P$ are $P + 1$ i.i.d. uniform permutations, independent of $(\theta, X)$ . Relabeling i.i.d. variables does not change their joint law. So the sequence on the right is exchangeable, and so is $(\ell_0, \ell_1, \dots, \ell_P)$ , which applies the same function $\ell$ with the same $\theta$ to each entry.

Step 2, a counting fact. For $j \in \{0, \dots, P\}$ let $R_j = \#\{k \in \{0, \dots, P\} : \ell_k \ge \ell_j\}$ , which counts $j$ itself, so $\hat p = R_0/(P+1)$ . For any real numbers and any integer $q$ , the set $J = \{j : R_j \le q\}$ has at most $q$ elements. Indeed, if $J$ is nonempty, pick $j^\star \in J$ with the smallest $\ell_{j^\star}$ . Every $j \in J$ has $\ell_j \ge \ell_{j^\star}$ , so $R_{j^\star} \ge \lvert J\rvert$ , and $R_{j^\star} \le q$ .

Step 3, combine. By exchangeability every $R_j$ has the same distribution as $R_0$ , so

\mathbb{P}(R_0 \le q) = \frac{1}{P+1}\sum_{j=0}^{P}\mathbb{P}(R_j \le q) = \frac{\mathbb{E}\,\lvert J\rvert}{P+1} \le \frac{q}{P+1}.

Since $\hat p$ takes values in multiples of $1/(P+1)$ , $\mathbb{P}(\hat p \le \alpha) = \mathbb{P}\big(R_0 \le \lfloor\alpha(P+1)\rfloor\big) \le \lfloor\alpha(P+1)\rfloor/(P+1) \le \alpha$ . Without ties the $R_j$ are a permutation of $1, \dots, P+1$ , so $\lvert J\rvert = q$ exactly and $R_0$ is uniform. ∎

This is a theorem. It holds for any model and any exchangeable benchmark, with no assumption about how the model works. Three consequences follow.

The $+1$ matters. Without it, $\#\{\ell_k \ge \ell_0\}/P$ reports $p = 0$ when the published order beats every shuffle, a claim of certainty that $P$ samples cannot support. It also rejects too often. With $P = 20$ and $\alpha = 0.05$ it rejects whenever at most one shuffle beats the published order. Under the null that happens with probability $2/21 = 0.095$ , nearly twice the promised 5%. The corrected test rejects only when no shuffle does, with probability $1/21 = 0.048$ . The next cell checks both numbers.
The smallest possible p-value is $1/(P+1)$ . With $P = 99$ shuffles and a published order that beats all of them, report $p = 0.01$ , not $p = 0$ . Now suppose you audit 50 benchmarks on 5 models, which is 250 tests. A Bonferroni correction divides the level by the number of tests, here $0.05/250 = 0.0002$ . Reaching it needs $P \ge 4{,}999$ full passes over each benchmark before a rejection is even possible.
The test needs exchangeability, and it sees only one kind of leak. A file sorted by difficulty, or with numbered items, can make the published order more likely to an innocent model. And a leak that did not keep the published order leaves the test nothing to detect, for example shuffled copies, paraphrases, or items scattered across forum posts. The test can prove contamination. It can never prove its absence.

The cell simulates 4,000 innocent models. Each scores a 30-item benchmark and 20 shuffles, and the cell counts how often each version of the test rejects.

Python · cell 5

The sharded test, and why it has power

The floor of $1/(P+1)$ is a limit set by compute. A p-value of $10^{-6}$ needs a million shuffles, each a full pass of the model over the benchmark. Oren et al. get around it with a sharded likelihood comparison test. Split the $n$ examples into $r$ contiguous shards, blocks of consecutive examples whose sizes differ by at most one. In shard $S_i$ , compare the published-order log-likelihood with the average over $m$ shuffles of that shard,

s_i = \ell(S_i) - \frac{1}{m}\sum_{k=1}^{m}\ell\big((S_i)_{\varsigma_{ik}}\big), \qquad i = 1, \dots, r .

Then test whether the $s_i$ have mean zero with a one-sided $t$ -test. The statistic is $t = \bar s/(\hat\sigma_s/\sqrt r)$ , where $\bar s$ is the mean and $\hat\sigma_s$ the sample standard deviation of the $s_i$ . It is compared against Student's $t$ distribution with $r - 1$ degrees of freedom.

Why is this valid? Under the null, each $s_i$ has mean zero. Step 1 of the proof above, applied within one shard, says the published shard and each of its shuffles have the same distribution. When the examples are i.i.d., the $s_i$ are also independent, because the shards are disjoint and the model is fixed. So the $t$ -test is valid in the limit as the number of shards grows (Theorem 2 of Oren et al.). Unlike the permutation test, its guarantee is asymptotic rather than exact.

Why does it have power? The gain is not more signal. In the toy model below, a memorized order adds a bonus $\delta$ to the log-likelihood for every adjacent pair of items in published order. A random order of $k$ items has on average $(k-1)/k$ such pairs, so the published order of the whole benchmark gains about $(n - 2)\delta$ over a random one. Suppose the other effects of context on the log-likelihood have standard deviation $\sigma_c$ per pair. Then either test sees a standardized signal of roughly $\sqrt n\,\delta/\sigma_c$ . The gain is that the $t$ -test turns that signal into a p-value directly, with no floor. A $z$ of 7 gives $p \approx 10^{-12}$ without $10^{12}$ shuffles.

Each shard's mean needs only a few dozen shuffles. Oren et al. found diminishing returns beyond about 25 shuffles per shard and a sweet spot of 10 to 20 shards for 1,000 examples. Their audits of real models used 50 shards with 250 shuffles each.

✏️ The order test fires on a memorized benchmark

A toy model scores a 1,000-item benchmark. Its log-likelihood of an ordering is a sum of random context effects, $\sigma_c = 1$ nat per adjacent pair, plus a memorization bonus $\delta$ per pair in published order. Run the permutation test with $P = 99$ and the sharded test with 50 shards of 20 items and 25 shuffles each.

At $\delta = 0$ , a model that never saw the file, both p-values are unremarkable (0.37 and 0.70).
At $\delta = 0.05$ nats the expected $z$ is $\sqrt{1000} \times 0.05 = 1.6$ . Neither test is conclusive (0.05 and 0.14).
At $\delta = 0.15$ nats the expected $z$ is 4.7. The permutation test hits its floor of 0.01. The sharded test reports $p \approx 2\times10^{-5}$ , about 500 times smaller.

The cell runs the three cases.

Python · cell 6

📌 What the order test found on real models

Oren et al. (2023) pretrained a 1.4-billion-parameter model on Wikitext, a corpus of Wikipedia articles, with benchmarks inserted at known numbers of copies. Test sets inserted once were not detected (sharded p-values from 0.16 to 0.48). Detection began at 2 to 4 copies, with a threshold near 4. At 10 copies the sharded p-values were vanishingly small, $2\times10^{-11}$ for the MNLI benchmark and $3\times10^{-13}$ for TruthfulQA. At 50 copies and more, every test set's p-value was numerically zero. The permutation test with 100 shuffles was stuck at its floor near 0.01.

They then audited LLaMA2-7B, Mistral-7B, Pythia-1.4B, GPT-2 XL, and BioMedLM on eight benchmarks. BioMedLM, trained only on PubMed biomedical text, served as a negative control, a model known not to have seen the benchmarks. They found little evidence of pervasive contamination. The one striking result was Mistral-7B on ARC-Easy, a set of grade-school science questions, with $p = 0.001$ . They caution that this sits at the edge of significance once you correct for the several dozen tests in the table.

The widget runs the same toy model on a 1,000-item benchmark, with the sharded test on 50 shards of 20 items and 25 shuffles each. Each copy of the file in training adds a little to the bonus $\delta$ . On the left, the bars are the log-likelihoods of the shuffled orders and the line is the published order. On the right are both p-values as the copies grow.

Interview

Prove contamination from order

You get a benchmark as a list examples and a black box seq_logprob(list_of_examples) -> float, which returns the model's log-probability of those examples concatenated in the given order. Implement both tests.

permutation_pvalue(examples, seq_logprob, P, rng) returns the permutation p-value with the $+1$ correction. Draw each shuffle with rng.permutation.
sharded_test(examples, seq_logprob, n_shards, m, rng) returns the p-value of the sharded test. Split the examples into n_shards contiguous shards whose sizes differ by at most one, the larger ones first (as np.array_split does). For each shard, compute its published-order log-probability minus the mean over m shuffles of that same shard. Return the one-sided p-value of a $t$ -test that these differences have a mean greater than zero.

import numpy as np
from scipy.stats import ttest_1samp

def permutation_pvalue(examples, seq_logprob, P, rng):
    raise NotImplementedError

def sharded_test(examples, seq_logprob, n_shards, m, rng):
    raise NotImplementedError

Tests, hints, verified solutions, and AI help, with Pro at $7.99 a month.

Here is the test as you would run it on an open model with Hugging Face. It cannot run in your browser, but every line maps to the cell above. The one practical constraint is the context window. Each shard is scored as a single sequence, so choose a number of shards that makes each shard fit.

Before you trust a small p-value from this code, run it on a negative control, a model trained before the benchmark existed or on unrelated data. Oren et al. used BioMedLM and GPT-2. If the control also fires, the file is not exchangeable, and the result says nothing about contamination. On MMLU, they set aside the 14 of 57 subject files on which a negative control fired.

Closed models that return only text need a different kind of test, one that asks the model to complete or fill in a test item. Neither test below gives a p-value with a guarantee, but both work through an API.

Guided instruction (Golchin and Surdeanu 2023) prompts the model with a dataset's name, the split, and the first part of an item. It checks whether the model completes the rest exactly or nearly. The best version agreed with human experts' verdicts in 92 to 100% of cases on seven datasets.
Testset slot guessing (Deng et al. 2023) hides one wrong option of a multiple-choice item and asks the model to supply it. Wrong options are arbitrary, so a model that reproduces them has seen the item. ChatGPT and GPT-4 reproduced MMLU's hidden wrong options with exact-match rates of 52% and 57%.

Fresh, perturbed, and time-split items

The tests so far look for traces of the leak. The most decisive evidence looks for its effect instead. Score the model on new items written to the same specification. If its score holds up, the leak did not matter, whatever the model saw. If the score drops, something did not generalize, and you have measured how much.

There are three ways to get new items.

Write them, as GSM1k did.
Perturb old ones. GSM-Symbolic (Mirzadeh et al. 2024) turned GSM8K problems into templates whose names and numbers can be changed. All 25 models it tested scored lower on the new versions than on the originals, and changing the numbers hurt more than changing the names. Adding one irrelevant but plausible clause cut performance by up to 65%.
Let time write them. Collect problems released after a model's training cutoff, the date after which no data entered its training set. LiveCodeBench does this (Jain et al. 2024), with 511 problems from LeetCode, AtCoder, and CodeForces released between May 2023 and May 2024 in the paper's latest version. So does LiveBench (White et al. 2024), which adds questions from recent sources every month and grades them against objective answers. Roberts et al. (2023) used the same natural experiment on GPT models with Codeforces and Project Euler problems. They found statistically significant trends of pass rate with a problem's popularity on GitHub and with its release date relative to the cutoff.

📌 Fresh copies of benchmarks reveal overfitting

On GSM1k, 1,205 new problems matched to GSM8K, the worst models scored up to 8 points lower than on GSM8K. The Phi and Mistral families lost points at nearly every size, and Yi, Xwin, Gemma, and CodeLlama did so to a lesser extent. Across models, the gap correlated with the per-character log-likelihood of generating GSM8K's test problems (Spearman correlation 0.36, $p = 0.03$ ). Frontier models showed little or no gap (Zhang et al. 2024).

On LiveCodeBench, DeepSeek's 33B base model fell from a pass rate of about 60% on LeetCode problems from May 2023 to about 0 on September 2023 problems. GPT-4o dropped on LeetCode problems released after its late-2023 cutoff. Performance on AtCoder problems stayed roughly stable across months (Jain et al. 2024).

A drop on new items is not yet a measurement of contamination, because the new items may just be harder. GSM1k matched GSM8K's human solve rates, step counts, and answer sizes for exactly this reason. Time splits cannot be matched that way. Next month's problems are whatever next month brings, harder or easier. The fix is a control group.

Time splits as a difference in differences

Take a reference model that cannot have seen any of the problems, for example one whose training data predates them all. Score it on the same problems. Whatever happened to difficulty over time happened to it too. Subtract its before-minus-after change from the target model's, and the shared trend cancels. Econometricians call this a difference in differences.

The example subtracted four accuracies. The general statement writes the same estimate as an average of per-problem differences between the two models, which gives its error bar, and it names the one assumption the subtraction rests on.

∴ Difference in differences for a time split

Let problems $i = 1, \dots, n$ have release dates $t_i$ , and let model A have cutoff $c$ . The problems with $t_i \le c$ ("before", $n_b$ of them) could have leaked, and those with $t_i \gt c$ ("after", $n_a$ of them) could not. Let $o^A_i, o^R_i \in \{0, 1\}$ be the pass indicators of A and of a reference model R that saw none of the problems. Write A's clean pass probability, had nothing leaked, as $p^0_{A,i}$ , its actual one as $p^0_{A,i} + \Delta_i$ for before-problems, and R's as $p_{R,i}$ . Assume parallel trends, $\bar p^{\,0}_{A,b} - \bar p^{\,0}_{A,a} = \bar p_{R,b} - \bar p_{R,a}$ , where bars average over the problems in each period. Then the estimator

\widehat{\mathrm{DiD}} = (\bar o^A_b - \bar o^A_a) - (\bar o^R_b - \bar o^R_a) = \bar d_b - \bar d_a, \qquad d_i = o^A_i - o^R_i,

has expectation $\bar\Delta_b$ , the average leak effect on the before-problems. Its variance is $\operatorname{Var}(d \mid b)/n_b + \operatorname{Var}(d \mid a)/n_a$ when problems are independent. The proof follows.

∂ Proof of the difference-in-differences identity

Take expectations problem by problem. $\mathbb{E}[\bar o^A_b - \bar o^A_a] = \bar p^{\,0}_{A,b} + \bar\Delta_b - \bar p^{\,0}_{A,a}$ and $\mathbb{E}[\bar o^R_b - \bar o^R_a] = \bar p_{R,b} - \bar p_{R,a}$ . Subtracting and using parallel trends leaves $\bar\Delta_b$ . The rearrangement $\widehat{\mathrm{DiD}} = \bar d_b - \bar d_a$ is algebra. The two period means are over disjoint sets of independent problems, so their variances add. ∎

In plain words, the reference model measures how much harder the later problems are, and whatever drop is left over is the leak. The pairing matters. A and R are scored on the same problems, so a problem that is hard for both cancels inside $d_i$ . That makes the interval tighter than one built from four separate accuracies, and the more often the two models pass and fail the same problems, the larger the gain.

The assumption matters too. Parallel trends says that without the leak, A's accuracy would have fallen by the same amount as R's. It is most plausible when R is about as strong as A. A much weaker reference near the floor cannot drop as much as A does, which biases the estimate up. A reference that also saw the old problems, such as a newer model, makes the estimate too small, because older problems have had more time to spread across the web. LiveCodeBench's AtCoder curves play the same role from a different angle. They show the same model on a second platform whose problems did not leak.

The cell simulates 500 time-split benchmarks of 40 problems a month for two years. Problems get harder over time, and model A memorized 30% of the before-problems it would otherwise fail.

Python · cell 7

The naive drop of about 31 points is mostly the difficulty trend. The difference in differences recovers the planted effect of about 11.6 points on average, and its intervals cover it 94% of the time, close to the promised 95%. Notice the standard error, about 4 points with 960 problems. Ignoring the pairing would give 4.4 points. The gain is modest here, because most of the spread in a single pass or fail is the luck of one attempt, which pairing cannot cancel. The standard error shrinks like one over the square root of the number of problems, so LiveCodeBench's 511 problems give about $4\sqrt{960/511} \approx 5.5$ points, an interval near $\pm 11$ points on a single model. Only large leaks, like DeepSeek's, are visible this way. That is the number to quote when an interviewer asks whether a 5-point post-cutoff drop proves anything.

Interview

Did the model see these problems? A time-split analysis

Implement time_split(release, cutoff, model_pass, ref_pass, n_boot, rng). release holds each problem's release date as an ISO string such as "2024-03-17", and cutoff is the model's training cutoff in the same format. A problem released on the cutoff date counts as before. model_pass and ref_pass are the 0/1 outcomes of the model and of a reference model that saw none of the problems. Return a dict with five entries.

"before" and "after", each a tuple (accuracy, lo, hi) for the model, with the Wilson interval from the wilson helper.
"naive_drop", the model's before accuracy minus its after accuracy.
"did", the difference in differences.
"did_ci", a 95% percentile interval (lo, hi) from n_boot bootstrap resamples. Each resample draws before-problems from the before period and after-problems from the after period, with replacement and keeping each period's size. It uses the same resampled problems for the model and the reference.

import numpy as np

def time_split(release, cutoff, model_pass, ref_pass, n_boot, rng):
    raise NotImplementedError

Tests, hints, verified solutions, and AI help, with Pro at $7.99 a month.

How much did the leak help

Suppose an overlap check, or a vendor's report, has split a benchmark into dirty and clean items. The tempting estimate of what contamination bought is the dirty items' accuracy minus the clean items' accuracy. Before we test that estimate, here is the identity that says what we are estimating.

∴ Leakage bias is the dirty share times the effect on dirty items

Let $p_i$ be the model's pass probability on item $i$ as it is, and $p^0_i$ what it would be had item $i$ never leaked, so that $p_i = p^0_i$ for clean items. Let $\nu_{\text{dirty}}$ be the fraction of dirty items and $\Delta_{\text{dirty}}$ the mean of $p_i - p^0_i$ over them. Then

\mathbb{E}[\hat\mu] - \frac{1}{n}\sum_{i=1}^{n}p^0_i = \nu_{\text{dirty}}\,\Delta_{\text{dirty}} .

If the items are a sample from $\mathcal{D}_{\text{bench}}$ , the second term on the left is an unbiased estimate of $\mu_{\text{bench}}$ , so $\nu_{\text{dirty}}\Delta_{\text{dirty}}$ estimates $\Delta_{\text{leak}}$ . The proof follows.

∂ Proof of the leakage identity

$\mathbb{E}[\hat\mu] = \frac1n\sum_i p_i$ . The sum $\sum_i (p_i - p^0_i)$ has nonzero terms only for dirty items, and those terms add up to $n\nu_{\text{dirty}}\Delta_{\text{dirty}}$ . A fresh draw from $\mathcal{D}_{\text{bench}}$ is clean, and the $n$ items are such draws before any leak, so $\mathbb{E}[\frac1n\sum_i p^0_i] = \mu_{\text{bench}}$ . ∎

So we need $\Delta_{\text{dirty}}$ . In the language of causal inference it is an average treatment effect on the treated, the mean effect of a treatment over the units that actually received it. Here the treatment is the leak and the treated units are the dirty items. The naive dirty-minus-clean gap estimates it only if dirty and clean items would have been equally hard without the leak, and they rarely are. Long items have more n-grams, so they are flagged more often. Popular items get quoted more. Older items have had longer to spread. Each of those properties also predicts difficulty.

✏️ A negative gap that hides an eight-point leak

A 1,000-item benchmark has 420 easy items, with clean pass rate 0.9, and 580 hard ones, with clean pass rate 0.3. The leak raises the pass rate of every dirty item by 0.08. Hard items are long and get quoted, so 180 of the 200 dirty items are hard. That leaves 400 easy and 400 hard clean items.

Dirty accuracy is $(20 \times 0.98 + 180 \times 0.38)/200 = 0.44$ . Clean accuracy is $(400 \times 0.9 + 400 \times 0.3)/800 = 0.60$ . The naive gap is $-16$ points, and a report would conclude that contaminated items are harder for the model, so contamination did not help.

Within each difficulty level the comparison is fair. Easy items gain $0.98 - 0.90 = 0.08$ , and hard items gain $0.38 - 0.30 = 0.08$ . This reversal is Simpson's paradox, where every group shows a gain and the pooled comparison shows a loss. It happens because being dirty goes with being hard. Of the dirty items 90% are hard, against 50% of the clean ones, so pooling compares mostly hard dirty items with half-easy clean ones. Weighting by where the dirty items are gives $0.1 \times 0.08 + 0.9 \times 0.08 = 0.08$ , the true effect. The headline inflation is $\nu_{\text{dirty}}\Delta_{\text{dirty}} = 0.2 \times 0.08 = 1.6$ points.

The clean subset is biased too. Its accuracy of 0.60 overstates the true $\mu_{\text{bench}} = (420 \times 0.9 + 580 \times 0.3)/1000 = 0.552$ , because the clean items are easier. "Report the clean subset" is right only if you reweight it to the benchmark's mix of difficulty.

The fix is to compare only items of similar difficulty. Group the items into strata, bins of similar difficulty. Compare dirty with clean inside each bin, and average the comparisons.

∴ Stratified estimate of the leak effect

Split the items into strata $h = 1, \dots, H$ by a difficulty proxy that the leak cannot affect, such as a reference model's accuracy, item length, or the human solve rate. In stratum $h$ let $n_{h,d}$ and $n_{h,c}$ be the numbers of dirty and clean items, and $\hat p_{h,d}$ and $\hat p_{h,c}$ the model's accuracies on them. Keep only strata that have both kinds, and set $w_h = n_{h,d}/\sum_{h'} n_{h',d}$ over the kept strata. Assume that within each stratum, dirty and clean items have the same mean clean pass rate, $\mathbb{E}[p^0 \mid \text{dirty}, h] = \mathbb{E}[p^0 \mid \text{clean}, h]$ . Then

\hat\Delta_{\text{dirty}} = \sum_h w_h\big(\hat p_{h,d} - \hat p_{h,c}\big), \qquad \operatorname{SE}(\hat\Delta_{\text{dirty}}) = \sqrt{\sum_h w_h^2\left(\frac{\hat p_{h,d}(1 - \hat p_{h,d})}{n_{h,d}} + \frac{\hat p_{h,c}(1 - \hat p_{h,c})}{n_{h,c}}\right)}

is unbiased for the effect on the dirty items of the kept strata, with that standard error when outcomes are independent. The proof follows.

∂ Proof of the stratified estimate

In stratum $h$ , $\mathbb{E}[\hat p_{h,d}] = \mathbb{E}[p^0 \mid \text{dirty}, h] + \Delta_h$ , where $\Delta_h$ is the mean effect on that stratum's dirty items, and $\mathbb{E}[\hat p_{h,c}] = \mathbb{E}[p^0 \mid \text{clean}, h]$ . By the assumption, the difference has expectation $\Delta_h$ . Summing with weights $w_h$ , the dirty items' shares, gives $\sum_h w_h\Delta_h$ , the mean effect over dirty items. Given the counts, the $2H$ accuracies are independent averages of Bernoulli outcomes. So the variance is the sum of their variances, each multiplied by $w_h^2$ . ∎

In plain words, compare dirty and clean items only among items that are equally hard, then average those comparisons in proportion to where the dirty items are. The weights are the dirty items' shares because the question is what the leak did to the dirty items. Weighting by clean items or by all items answers a different question, and when the effect differs by stratum it gives a different number.

The assumption is the weak point, as in every observational analysis. A proxy that misses the real cause of difficulty leaves bias behind. So stratify on the best difficulty measure you have, and say which one you used.

The cell runs the estimator on the example's numbers.

Python · cell 8

The estimate recovers the 8-point effect, but its 95% interval is about $\pm 7.6$ points with these counts. Even a correct estimator needs many dirty items to pin down a small effect.

Two published methods tie detection to effect more tightly. ConTAM (Singh et al. 2024) chooses the overlap threshold so that the flagged items are the ones that actually show a gain. ConStat (Dekoninck et al. 2024) defines contamination directly as performance that does not generalize. It compares a model's score on a benchmark with its score on a second benchmark for the same task, relative to a set of reference models. It reported high levels of contamination in several popular models, including Mistral, Llama, Yi, and the top three models of the Open LLM Leaderboard at the time.

Core

How much did the leak help?

Implement leak_effect(dirty, correct, difficulty) for 0/1 arrays dirty and correct and a real-valued difficulty proxy. Form five strata by quintiles, with edges = np.quantile(difficulty, [0.2, 0.4, 0.6, 0.8]) and h = np.searchsorted(edges, difficulty, side="right"), which takes values 0 to 4. Return a dict with four entries.

"naive", the dirty accuracy minus the clean accuracy.
"stratified", the estimate from the theorem. Skip strata that lack dirty or clean items and renormalize the weights over the rest.
"se", its standard error.
"inflation", the fraction of dirty items times "stratified", the estimated leakage bias of the headline score.

import numpy as np

def leak_effect(dirty, correct, difficulty):
    raise NotImplementedError

Tests, hints, verified solutions, and AI help, with Pro at $7.99 a month.

Canaries

Every test so far is detective work after the fact. A canary is the benchmark author's foresight, a unique random string placed in the benchmark so that its spread can be traced later. BIG-bench, a collaborative benchmark of more than 200 tasks, put one random identifier into every file (Srivastava et al. 2022). The string lets anyone building a training corpus filter BIG-bench out, and the authors asked every publication that quotes BIG-bench tasks to include it too. The string serves two purposes. Honest trainers can filter on it. And anyone can later test whether a model has memorized it, which would mean that documents carrying it were trained on.

The test is the canary test of §19.7 applied to a benchmark. Score the true canary and $M$ random strings of the same format by the model's log-probability. Find the canary's rank among all $M + 1$ , counting from the most likely, with ties counted against it. If the model never saw the canary, the canary is one more random string of that format, exchangeable with the others. The argument of the permutation theorem then gives the valid p-value

\hat p = \frac{1 + \#\{j : \log p_\theta(\text{random}_j) \ge \log p_\theta(\text{canary})\}}{M + 1} = \frac{\text{rank}}{M + 1} .

In plain words, a canary ranked first among itself and 999 random strings has $p = 0.001$ . §19.7 measured memorization by exposure, the number of bits of the secret string the model gives away, which is $\log_2$ of the number of candidates minus $\log_2$ of the canary's rank. Over these $M + 1$ candidates, $-\log_2\hat p$ is exactly that exposure, with ties counted against the canary.

Canaries are cheap, and they are not enough. They work only if every crawler and trainer honors them. Dropping documents with the string removes only the documents that kept it. Quoting a few items in a blog post without the canary, reformatting the files, or pasting a problem into a forum all leave the canary behind. A positive result is strong evidence that canaried documents were trained on, though not which ones. A negative result proves nothing, since a long random string is hard to memorize from a handful of copies. The order test, by comparison, needed about 4 copies of a whole file.

Prevention goes further. Jacovi et al. (2023) proposed three strategies for public test data.

Publish it encrypted with a public key, under a license that forbids redistributing it.
Refuse to evaluate on closed APIs that cannot exclude test data from training.
Avoid test items whose solutions already appear online.

GSM1k took the strongest option and kept its problems private. Its authors committed to release them once three open models from different lineages scored above 95%, or in June 2025, whichever came first.

Warm-up

Canary check

Implement canary_test(canary_logp, random_logps, alpha=0.05, n_tests=1) for M random strings. It returns (rank, p, reject).

The rank is 1 plus the number of random strings with log-probability greater than or equal to the canary's.
The p-value is rank / (M + 1).
reject is whether p <= alpha / n_tests, a Bonferroni correction for checking several models or benchmarks.

If even rank 1 could not reach the corrected level, raise ValueError, since the test is then powerless by design.

import numpy as np

def canary_test(canary_logp, random_logps, alpha=0.05, n_tests=1):
    raise NotImplementedError

Tests, hints, verified solutions, and AI help, with Pro at $7.99 a month.

Scoring texts in PyTorch

In practice every per-token attack comes from one forward pass. Here is the batched version, with padding, of the log-probabilities, Min-K% Prob, and the Min-K%++ score, computed from a causal language model's logits and checked against a plain loop. With a real model you would pass model(ids).logits, and nothing else changes.

🔥 In PyTorchfor a machine with PyTorch

What labs do

No single check covers every kind of leak, so serious evaluation stacks them. Here are the options side by side.

method	what you need	catches	misses	gives you
n-gram overlap, decontamination (§8.3, §14.2)	the training data	verbatim and lightly edited copies	paraphrases, translations, answer keys, synthetic data	a flag per item
embedding search plus a model judge (Yang et al. 2023)	the training data, compute	paraphrases, translations	answer keys, distant derivations	a flag per item
per-example membership inference	log-probabilities, same-distribution non-members	heavily duplicated items	almost everything at one copy	a weak score per item
dataset inference (Maini et al. 2024)	log-probabilities, same-distribution non-members	a seen dataset, even with weak per-item signal	leaks without surface traces	a p-value per dataset
order test (Oren et al. 2023)	log-probabilities, an exchangeable file	the file seen in order a few times	shuffled or scattered copies	a p-value with a guarantee
guided completion, slot guessing	text from an API	memorized text and wrong options	leaks the model will not recite	strong hints, no guarantee
fresh, perturbed, or time-split items	new items, a reference model	the effect of any leak	nothing, if items are truly new and matched	an estimate of the inflation
canaries (BIG-bench)	the author's foresight, trainers' cooperation	trained-on canaried documents	copies without the string	a p-value per model
private held-out sets (GSM1k)	trust in the holder	everything	nothing, until they leak	a clean score

The practice that follows from the table has five parts.

Decontaminate training data against every benchmark you will report, including fine-tuning and synthetic data. Publish the method, the $n$ , and the threshold.
Keep private held-out sets, and refresh them, since every evaluation through a closed API sends the items to someone else's servers. How to build and guard your own is part of building your own evals.
Prefer benchmarks with time stamps or regular refreshes (LiveBench, LiveCodeBench) and perturbed variants (GSM-Symbolic), and report results on the post-cutoff slice.
When you report a contamination analysis, report clean and dirty results stratified by difficulty, the leakage-bias estimate with an interval, and the detector's known blind spots.
Block test-time leaks. A browsing agent can look up a benchmark. The Holistic Agent Leaderboard (Kapoor et al. 2025) found agents searching Hugging Face for the benchmark instead of solving the task. Blocking that is part of evaluating agents.

The widget turns the map from the start of the section into numbers you can change. It is an illustrative model, with parameters set to match the findings above, not measured data. Each row is a kind of leak, with every item of a 1,000-item benchmark leaked that way into the training data of a model whose clean accuracy is 60%. The n-gram and Min-20% Prob columns are the share of leaked items flagged, with the Min-K% threshold set for 5% false alarms. The order test column is the chance that the test fires on the whole file at level 0.05. The last column is the points the leak adds. Click a row to read it.

🧭 What to remember

Contamination is any item-specific information that reaches the model before or during evaluation, from verbatim copies and answer keys to paraphrases, synthetic data, and test-time lookups. Its cost is the leakage bias $\Delta_{\text{leak}} = \mathbb{E}[\hat\mu] - \mu_{\text{bench}}$ , which equals the dirty share times the effect on dirty items, $\nu_{\text{dirty}}\Delta_{\text{dirty}}$ .
n-gram overlap needs the training data and sees only surface copies. Its threshold trades false alarms for misses, and a clean report says nothing about paraphrases, answers, or synthetic data.
Per-example membership inference (loss, zlib ratio, Min-K%, Min-K%++) is near chance on real pretraining data. Its published successes came largely from distribution shift, so always run a blind baseline, and aggregate weak evidence over many items instead.
The order test proves contamination with a guaranteed false positive rate when the benchmark is exchangeable. Keep the $+1$ , remember the $1/(P+1)$ floor, use the sharded $t$ -test for small p-values, and run a negative control.
The effect of a leak is measured on fresh, perturbed, or post-cutoff items. Time splits need a difference in differences against a reference model, and dirty-versus-clean gaps need stratification by difficulty with dirty-item weights.
Canaries, encryption, and private, refreshed held-out sets prevent leaks better than any detector finds them, and none of them works without cooperation.
Leakage corrupts what the model knows before it is graded. The next section turns to the grader itself, and to what happens when the grader is a language model, which can be biased, noisy, and even contaminated with benchmark answers of its own.

Oren et al. (2023). Proving Test Set Contamination in Black Box Language Models. The exchangeability argument, the sharded test, and the canary experiments that show how many copies it takes.
Shi et al. (2023). Detecting Pretraining Data from Large Language Models. Min-K% Prob and the WikiMIA benchmark, best read together with the two critiques below.
Duan et al. (2024). Do Membership Inference Attacks Work on Large Language Models? The careful same-distribution evaluation on Pythia and the Pile, and why membership is fuzzy.
Das, Zhang, and Tramèr (2024). Blind Baselines Beat Membership Inference Attacks for Foundation Models. Short and decisive, a model-free classifier beats published attacks on every evaluation set.
Maini et al. (2024). LLM Dataset Inference: Did you train on my dataset? How to aggregate weak per-example signals into a valid dataset-level test.
Zhang et al. (2024). A Careful Examination of Large Language Model Performance on Grade School Arithmetic. GSM1k, the model for building a matched fresh benchmark and keeping it private.
Jain et al. (2024). LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code. Time-stamped problems and the cutoff drops that reveal contamination.
Singh et al. (2024). Evaluation data contamination in LLMs: how do we measure it and (when) does it matter? ConTAM, which judges an overlap metric by whether the items it flags actually score higher.
Jacovi et al. (2023). Stop Uploading Test Data in Plain Text: Practical Strategies for Mitigating Data Contamination by Evaluation Benchmarks. Encryption, training-exclusion controls, and other prevention strategies.

Made for larger screens

Contamination

What leaks, and how

Overlap checks, and what they cannot see

Membership inference, a test for one example

Min-K% Prob and the zlib ratio

A membership test you can run

Why per-example membership inference fails on pretraining data

One weak signal, many items

The order test, which proves contamination

The sharded test, and why it has power

Prove contamination from order

Fresh, perturbed, and time-split items

Time splits as a difference in differences

Did the model see these problems? A time-split analysis

How much did the leak help

How much did the leak help?

Canaries

Canary check

Scoring texts in PyTorch

What labs do

Read more

Watch next