Chapter 22 · Section 22.4 · 100 min

LLM as a judge

An LLM judge is a language model that grades another model's answers. You give it a question, one or two answers, and instructions. It returns a score, or it says which answer is better. Judges are how the field grades open-ended work, such as explanations, emails, medical advice, and plans, where no program can check the answer. They are fast and cheap. They are also often wrong, in ways that are easy to miss. This section is about how far to trust a judge, and how to measure it.

Two experiments show why that matters. Three popular leaderboards grade with a judge. AlpacaEval 2.0 and Arena-Hard-Auto have a GPT-4 model compare each answer with a fixed reference model's answer to the same prompt, and MT-Bench has GPT-4 score each answer from 1 to 10. A win rate is the share of those comparisons that the judge says the submitted model wins. A length-controlled win rate is the win rate the judge would give if both answers had the same length, which removes its taste for longer answers (§16.6). In October 2024 a team built a "model" that ignores every instruction and returns the same fixed text. It scored an 86.5% length-controlled win rate on AlpacaEval 2.0, an 83.0% win rate on Arena-Hard-Auto, and 9.55 out of 10 on MT-Bench, top-ranked numbers on all three (Zheng et al. 2024). The text had been written to confuse the judge, and it did.

The second experiment is simpler. Wang et al. (2023) asked ChatGPT to grade answers from Vicuna-13B against answers from ChatGPT itself, on 80 questions. With Vicuna's answer shown first, Vicuna won 2.5% of the questions. With the same two answers in the other order, it won 82.5%. Nothing about the answers changed, only their place on the page.

Both stories are about the grader. §22.1 wrote every evaluation as an estimate. You want the average true quality of a model's responses $y$ to prompts $x$ . What you compute is the average of some grader's scores over $n$ benchmark items,

\mu^\star = \mathbb{E}\big[g^\star(x, y)\big], \qquad \hat\mu = \frac{1}{n}\sum_{i=1}^{n} g(x_i, y_i).

In plain words, $g^\star$ is the true quality of a response and $g$ is the grader you actually use. Whenever $g$ differs from $g^\star$ on average, the estimate $\hat\mu$ inherits that average difference, however many items you grade. Exact match, unit tests, and the answer-checking programs of §18.3 are graders whose mistakes you can list in advance. A judge is a grader whose mistakes you have to measure.

So this section treats a judge as a measuring instrument. It answers five questions in order. How should you ask a judge, and how do you write instructions that a judge and two humans apply the same way? What biases does it have, and how do you measure each one? How do you measure whether two graders agree? How precise is that measurement, and which statistic fits which data? And how can a judge be poor on single answers yet rank models well? It ends with the ways judges fail on the hardest problems, and what labs do about it.

What earlier sections established

Several earlier sections used judges. Here is what each one showed, in a sentence.

MT-Bench, 80 questions graded by GPT-4, showed that a strong judge can agree with expert humans about as often as experts agree with each other. The same section named four judge biases. A judge may favor the first or second slot, favor longer answers, favor its own model family, and grade badly the problems it cannot solve. It removed the first by judging every pair in both orders (§14.7).
A judge's average can be corrected with a few hundred human labels. You add the judge's average error on the labeled items to its average on all items. This is prediction-powered inference (§14.7).
A judge's flag rate can be corrected when you know two rates. Its sensitivity is how often it catches a true failure, and its specificity is how often it passes a true success. Inverting the two rates gives the true failure rate (§19.4).
The length-controlled win rate comes from a logistic regression of the judge's verdicts on the length difference. It reports what the judge would prefer at equal length (§16.6).
Position bias can be modeled as a constant added to the log-odds of whichever answer is shown first (§19.2).
A reward model is a judge that outputs a number instead of a verdict (§16.3).

Those sections needed a working judge. This one studies the instrument itself. It models all the biases at once and measures each one. It says how precise an agreement statistic is, and which statistic to use when the scale is ordered or the raters are many. It explains why a judge can be bad at single answers and still rank models well. And it shows how to tell when a judge has stopped measuring anything.

Four ways to ask a judge

You can ask a judge four kinds of question. Give this answer a score from 1 to 10. Which of these two answers is better? Does this answer agree with the correct one? Does this answer meet each item on this checklist? Most real judges combine two of them.

📐 Pointwise, pairwise, reference-guided, and rubric judging

Let $x$ be the prompt and $y$ , $y'$ two responses.

Pointwise (single-answer) grading returns a score $s(x, y)$ on a fixed scale, such as 1 to 10. Its main weakness is that the scale drifts across prompts, judges, and judge versions, and the scores bunch on a few values.
Pairwise comparison returns one of three verdicts on $(x, y, y')$ . The first is better, the second is better, or they tie. Some versions grade the verdict ("much better", "slightly better"). Its main weaknesses are position bias and cost, since comparing $m$ systems with each other takes on the order of $m^2$ comparisons.
Reference-guided grading gives the judge a correct answer $y^\ast$ and asks whether $y$ agrees with it, as a score or a verdict. Its main weakness is that it needs a reference, and the judge may mark down correct answers that are written differently.
Rubric (checklist) grading gives the judge a list of criteria $c_1, c_2, \dots$ , each a yes or no question about $y$ . It returns the vector of answers, often combined into a weighted score. Its main weakness is that someone must write the criteria for each task, and a rubric measures only what it lists.

Why is a comparison more sensitive than a score? A small example shows it. Model A scores 9 on an easy prompt and 4 on a hard one. Model B scores 8 and 3. Each model's scores swing by 5 points with the prompt's difficulty, and the 1-point gap between the models hides inside that swing. Compare them prompt by prompt instead, and A wins both times by exactly 1. The difficulty moves both answers together, so it cancels in the comparison. This is why a paired test beats an unpaired one (§0.11), and why preference data is collected as comparisons (§16.2).

Comparisons have two costs. Their verdicts can depend on which answer is shown first, as Wang et al.'s experiment showed. And the number of pairs grows like $m^2$ , so most benchmarks compare every system with one fixed baseline instead. Scores still have their place. A score is the only format for an absolute decision, such as whether an answer is safe to send, and its cost grows only linearly with the number of systems.

A reference matters most where judges are weakest. Zheng et al. (2023) gave GPT-4 ten math questions with two answers each. Each pair was judged in both orders, so there were 20 verdicts, and they counted how often GPT-4 called a wrong answer correct. With their default prompt it failed 14 times out of 20. Asking it to solve the problem first, in the same prompt, cut the failures to 6. Even then it often repeated the mistake of the answer it had just read. So they had GPT-4 solve each problem in a separate call, without seeing the answers, and showed that solution to the judge as a reference. That cut the failures to 3. A judge that cannot solve a problem cannot reliably grade it, and a wrong answer in front of it makes things worse. The end of the section returns to this.

Rubrics are what large expert evaluations use. HealthBench (Arora et al. 2025) holds 5,000 health conversations. Each has its own rubric written by physicians, 48,562 criteria in all from 262 physicians. Each criterion carries between $-10$ and $10$ points, with negative points for behavior that should not appear. A model grader decides, one criterion at a time, whether the response meets it. At release, o3 scored 60%. The grader was itself checked against physicians, and that check is what makes the number mean something.

✏️ Scoring one conversation with a rubric

Here is HealthBench's scoring rule on a made-up conversation with three criteria. "Advises calling emergency services" is worth 5 points. "Asks how long the pain has lasted" is worth 3. "Recommends a prescription drug without seeing a doctor" is worth $-4$ , because it describes behavior that should not appear.

A response that advises calling emergency services, never asks about the duration, and recommends the drug earns $5 + 0 - 4 = 1$ point. The most any response could earn is $5 + 3 = 8$ , the sum of the positive criteria. So this response scores $1/8 = 0.125$ . A single response can score below zero, so the benchmark averages over all conversations first and then clips the mean to the range from 0 to 1.

The table sums up when to use which format.

format	returns	sensitivity to small differences	main bias	use it for
pointwise score	a number per response	low, scale drift and bunching	leniency, scale drift	absolute gates, cheap monitoring
pairwise	which of two is better	high, prompt difficulty cancels	position, length	comparing checkpoints or systems
reference-guided	agreement with $y^\ast$	high where a reference exists	form over substance	math, factual QA, extraction
rubric or checklist	yes or no per criterion	high per criterion	criteria omitted, leniency	expert domains, per-failure-mode gates

In practice, compare checkpoints pairwise, gate single responses with a rubric, and give the judge a reference whenever one exists. An interviewer will ask you to defend that choice, and the prompt-difficulty argument above is the answer.

Scores from probabilities

A judge that outputs "4" hides how sure it was. If the model exposes the probability of each output token, you can read that too. Say a judge puts the probabilities $(0.05, 0.05, 0.20, 0.45, 0.25)$ on the scores 1 to 5. The most likely score is 4. The expected score is $0.05 \cdot 1 + 0.05 \cdot 2 + 0.20 \cdot 3 + 0.45 \cdot 4 + 0.25 \cdot 5 = 3.8$ . A second answer with probabilities $(0, 0.02, 0.08, 0.48, 0.42)$ also has 4 as its most likely score, but its expected score is 4.3. The expectation tells the two answers apart, and it moves smoothly as quality changes.

Liu et al. (2023) built a judge called G-Eval on exactly this idea. Write $s_1, \dots, s_L$ for the possible scores and $\lambda_1, \dots, \lambda_L$ for the judge's log-probabilities of the tokens that write them. Then

\bar s = \sum_{k=1}^{L} \hat p_k\, s_k, \qquad \hat p_k = \frac{e^{\lambda_k}}{\sum_{k'=1}^{L} e^{\lambda_{k'}}}.

In plain words, rescale the judge's probabilities over the score tokens alone so that they add up to one, then take the expected score. The rescaling matters, because some probability always goes to tokens that are not scores. When the API gave no probabilities, G-Eval estimated them from 20 samples at temperature 1. On news summaries it reached a Spearman rank correlation of 0.514 with human ratings, well above earlier automatic metrics. (Spearman's correlation is the ordinary correlation of the two lists' ranks, §1.6.) Its authors also warned that LLM graders may prefer text written by LLMs, the self-preference measured later in this section.

Warm-up

Expected scores from a judge's probabilities

Implement expected_score(logprobs, scores). The array logprobs holds the judge's log-probabilities for the score tokens, and scores holds the score each token stands for. The probabilities need not add up to one, and some log-probabilities may be -np.inf. Return the pair (expected, most_likely). The first entry is the probability-weighted score, after rescaling the probabilities over these tokens so they sum to one. The second is the score of the most likely token. The function must not overflow when the log-probabilities are large.

import numpy as np

def expected_score(logprobs, scores):
    raise NotImplementedError

Tests, hints, verified solutions, and AI help, with Pro at $7.99 a month.

Checking that the probabilities mean what they say

An expected score, or a pass probability compared with a threshold, is only as good as the probabilities behind it. Nothing guarantees that a judge's probabilities match what humans would say. Post-training tends to make models overconfident. GPT-4's calibration error on a subset of MMLU grew about tenfold after post-training, from 0.007 to 0.074 (OpenAI 2023, §19.8). And a judge's probability describes the judge's own uncertainty, not a human's.

📐 A calibrated judge, and two repairs

A judge's probabilities are calibrated against humans if, among the responses it passes with probability about $p$ , humans pass a fraction about $p$ , for every $p$ . The expected calibration error (ECE) measures the miss. Sort the items into bins by the judge's $\mathbb{P}(\text{pass})$ . In each bin, take the gap between the mean probability and the human pass rate. Average the sizes of those gaps, weighting each bin by its number of items (§0.11).

Two maps that keep the order of the items repair a miscalibrated judge. Write $\operatorname{logit}(p) = \log\big(p/(1 - p)\big)$ for the log-odds. Temperature scaling divides the judge's log-odds by one fitted positive number $T$ and reports $\sigma\big(\operatorname{logit}(\hat p)/T\big)$ . Platt scaling fits two numbers $a > 0$ and $b$ and reports

\hat p' = \sigma\big(a \operatorname{logit}(\hat p) + b\big), \qquad \sigma(z) = \frac{1}{1 + e^{-z}}.

The two numbers come from a logistic regression of the human labels on the judge's log-odds. The extra number $b$ repairs a lean toward passing as well as overconfidence.

Check calibration on the human-labeled items that validation needs anyway. Fit the repair on one half of them and test it on the other half. Both maps keep the order of the items. So they move thresholds and averages, but they never change which of two responses the judge rates higher.

Here is a simulated judge that is too sure and too lenient. Its log-odds are twice the human ones, plus 0.5 toward passing.

Python · cell 1

Only after such a repair is the judge's average probability an estimate of the human pass rate. Prediction-powered inference, which returns at the end of the section, corrects the average without assuming calibration at all. In practice, never threshold a judge's raw probabilities. An interviewer who hears "we filter at $p > 0.9$ " will ask what fraction of those items humans actually pass.

Writing a rubric that people and judges apply the same way

A rubric is the part of a judge you control most directly. These eight habits come from the papers above and from practice, and each has a reason.

Decide what the judge is for before writing a word. A judge that decides about single responses (send or block, keep or filter, reward or not) must be right item by item. A judge that compares systems only needs to be right on average. That choice decides which agreement you must prove, as the part on system-level agreement shows.
Prefer yes or no criteria over one overall score. "Does the response state the dose in milligrams?" can be checked. "Rate the medical quality from 1 to 10" cannot. Yes or no criteria let you measure the judge's error on each criterion and fix the worst one. They also avoid the drift of long scales.
Describe every level of a scale. If you need a scale, write what a 1, a 3, and a 5 look like, with an example of each. Then two raters mean the same thing by "3".
Ask for reasoning first and the verdict last, in a fixed format. Reasoning before the verdict helps on problems that need it (the 14 math failures that fell to 6 above). A fixed final format makes the verdict easy to parse. Read the verdict, not the reasoning.
Give a reference when one exists, and say what counts as equivalent (units, rounding, order).
Do not trust an instruction to remove a bias. The judge prompts of both Zheng et al. (2023) and Wang et al. (2023) told the judge to ignore the order of the answers. Both judges kept a strong position bias. Measurement and design, such as swapping orders, are what work.
Treat the response as data, not as instructions. Put it between delimiters and tell the judge that the text inside them is the thing to grade. Then test the judge with responses that address it directly, like the fixed text from the opening and the prompt injections of §19.3.
Pilot it with two humans and the judge on 50 to 100 items. Measure how well the two humans agree first. Where they disagree, the rubric is ambiguous, so fix its wording before blaming the judge. Then measure the judge's agreement with the humans against that ceiling, with the statistics later in this section.

Here is what such a judge looks like in code. It grades with a rubric and a reference, judges pairs in both orders, and can poll a small panel of judges, called a jury. It calls a generic model client, so it is shown, not run.

A model of a biased judge

To measure biases you need a model of how they enter a verdict. Start from the Bradley-Terry model of §16.3. Each system $i$ has one number, its rating $r_i$ . It beats system $j$ with probability $\sigma(r_i - r_j)$ , where $\sigma$ is the logistic sigmoid. If system $i$ 's rating is 0.4 higher, it wins $\sigma(0.4) = 0.60$ of its comparisons with $j$ . A biased judge adds three terms to that rating difference. It adds a bonus for whichever answer sits in the first slot, a bonus for being longer, and a bonus for coming from its own model family.

📐 The biased-judge model

A judge compares the response shown first, from system $i$ , with the response shown second, from system $j$ . It prefers the first with probability

\mathbb{P}(\text{first wins}) = \sigma\big(r_i - r_j + b_{\text{pos}} + \gamma_{\text{len}}(\ell_{\text{first}} - \ell_{\text{second}}) + \gamma_{\text{self}}(o_i - o_j)\big).

Here $r_i$ are the systems' quality ratings on the log-odds scale, assumed the same as a human's. The position bias $b_{\text{pos}}$ is a log-odds bonus for the first slot. $\ell$ is the log of a response's length in tokens, and $\gamma_{\text{len}}$ is the judge's extra preference per unit of log-length difference. $o_i$ is 1 if system $i$ comes from the judge's own model family and 0 otherwise, and $\gamma_{\text{self}}$ is the self-preference. A human rater in the same model has $b_{\text{pos}} = \gamma_{\text{len}} = \gamma_{\text{self}} = 0$ .

In plain words, the judge's verdict is a coin whose bias depends on true quality, plus three thumbs on the scale. The first thumb follows the slot. The other two follow the response.

That difference decides what swapping orders can fix. Collect everything that travels with the responses into one number,

u_{ij} = r_i - r_j + \gamma_{\text{len}}(\ell_i - \ell_j) + \gamma_{\text{self}}(o_i - o_j),

where $\ell_i$ and $\ell_j$ are the two responses' log-lengths. It is system $i$ 's margin as the judge sees it, before the slot.

∴ What judging both orders removes

Under the biased-judge model, the verdict for $i$ averaged over the two orders is

\tfrac{1}{2}\big(\sigma(u_{ij} + b_{\text{pos}}) + \sigma(u_{ij} - b_{\text{pos}})\big).

It equals $\tfrac12$ exactly when $u_{ij} = 0$ , and otherwise it lies on the same side of $\tfrac12$ as $u_{ij}$ . So swapping removes the position bias from the direction of the verdict. It leaves $\gamma_{\text{len}}$ and $\gamma_{\text{self}}$ untouched, because they are part of $u_{ij}$ in both orders.

Proof. With $i$ shown first, $i$ wins with probability $\sigma(u_{ij} + b_{\text{pos}})$ . With $j$ shown first, the first-shown response is $j$ 's, so the log-odds that $j$ wins are $u_{ji} + b_{\text{pos}} = -u_{ij} + b_{\text{pos}}$ . Then $i$ wins with probability $1 - \sigma(-u_{ij} + b_{\text{pos}}) = \sigma(u_{ij} - b_{\text{pos}})$ , using $1 - \sigma(-z) = \sigma(z)$ . At $u_{ij} = 0$ the two terms are $\sigma(b_{\text{pos}})$ and $1 - \sigma(b_{\text{pos}})$ , which average to $\tfrac12$ . Both terms increase with $u_{ij}$ , so the average crosses $\tfrac12$ only there. This is the argument of §19.2, with $u_{ij}$ in place of the fair log-odds. ∎

✏️ Swapping fixes the slot and keeps the length

Take $b_{\text{pos}} = 0.6$ . Two equally good, equally long answers get $\sigma(0.6) = 0.646$ in one order and $\sigma(-0.6) = 0.354$ in the other. These average to exactly 0.5.

Now let $i$ be better, $u_{ij} = 0.5$ . The two orders give 0.750 and 0.475, which average to 0.613. A judge with no position bias would say $\sigma(0.5) = 0.622$ . So the average is pulled slightly toward one half, but it points the right way.

Finally, let the answers be equally good, but $i$ 's answer longer by 0.3 in log-length, with $\gamma_{\text{len}} = 0.8$ . Then $u_{ij} = 0.8 \cdot 0.3 = 0.24$ , and the average over both orders is 0.555. The longer answer still wins 55% of the time after swapping.

So swapping is the right design, but averaging the two orders hides how large the position bias is. Before fitting any model, two simple counts describe it directly.

Consistency is a noisy measure of bias. Take a pair of equally good answers, $u_{ij} = 0$ , and assume the two verdicts are independent. The pair is consistent with probability $2\sigma(b_{\text{pos}})\sigma(-b_{\text{pos}})$ . That is 0.458 at $b_{\text{pos}} = 0.6$ , and still only $\tfrac12$ at $b_{\text{pos}} = 0$ , because a fair judge facing a tie flips a coin twice. So low consistency mixes noise with bias. The first-position rate rises above $\tfrac12$ only through $b_{\text{pos}}$ , provided every pair is shown in both orders. The cleanest single number is the fitted $b_{\text{pos}}$ , which the next part estimates.

Adding a few worked examples to the judge prompt raised GPT-4's consistency to 77.5%. The authors still used the plain prompt, because the longer one costs more and might bring biases of its own.

Length bias has its own published test. Zheng et al. took 23 MT-Bench answers that contained a numbered list. They made each one longer without adding information. GPT-4 rephrased the list, and the rephrased copy was inserted before the original. Then judges compared the padded answer with the original. GPT-3.5 and Claude-v1 preferred the padded answer in 91.3% of cases, and GPT-4 in 8.7%. Style biases run deeper than length. Feuer et al. (2024) found that judge preferences on alignment benchmarks did not track concrete measures of safety, world knowledge, or instruction following, and that judges weighted style over substance.

Measuring every bias at once

Let's build a judge with known biases and see what each measurement recovers. There are four systems. For every pair of them, both systems answer the same 2,000 prompts. Systems 0 and 1 are equally good, but system 1 writes answers about 35% longer ( $e^{0.3} \approx 1.35$ ) and comes from the judge's own family. System 2 is better and more concise, and system 3 is worse. The judge has $b_{\text{pos}} = 0.6$ , $\gamma_{\text{len}} = 0.8$ , and $\gamma_{\text{self}} = 0.4$ , and it judges every pair in both orders. A human with no biases judges each pair once.

Python · cell 2

The judge picks the first slot in 62.7% of verdicts and is consistent on only 53.0% of pairs. Against its equally good sibling, which is longer and from the judge's family, system 0 wins 48.2% when shown first and 22.7% when shown second. Averaging the orders gives 35.4%, while humans give 48.8%. Swapping removed the position effect and left a 13-point gap from length and self-preference. Against system 2, the averaged judge says system 0 wins 44.2%, while humans say 39.7%. System 2's short answers cost it under the judge's taste for length.

Swapping is a fix in the design. To measure the biases, fit the model. Each verdict becomes one row of a logistic regression. The row has $+1$ in the column of the system shown first and $-1$ in the column of the system shown second. It has a constant 1, whose coefficient is $b_{\text{pos}}$ , and the log-length difference, whose coefficient is $\gamma_{\text{len}}$ . Only differences of ratings affect a verdict, so we drop system 0's column, which sets $r_0 = 0$ .

We fit by Newton's method,

\mathbf{w} \leftarrow \mathbf{w} + (\mathbf{X}^\top\mathbf{S}\mathbf{X})^{-1}\mathbf{X}^\top(\mathbf{y} - \mathbf{p}), \qquad \mathbf{S} = \operatorname{diag}\big(p_t(1 - p_t)\big).

Here $\mathbf{X}$ stacks the rows, $\mathbf{y}$ holds the verdicts, $\mathbf{p}$ holds the fitted probabilities $p_t$ , and $\mathbf{w}$ holds the coefficients. This is the same iteration §16.6 used for the length-controlled win rate.

Python · cell 3

✏️ Auditing a judge

Read the four rows of output from the bottom. Humans recover the true ratings, $(0, 0.02, 0.41, -0.47)$ against $(0, 0, 0.4, -0.5)$ . The judge's full fit recovers $\hat b_{\text{pos}} = 0.60$ and $\hat\gamma_{\text{len}} = 0.81$ , both on target. It rates systems 1 and 2 equally, at 0.42. System 2's rating is right. System 1's is its true 0 plus the self-preference 0.4. So the gap between the judge's and the humans' rating of system 1, $0.42 - 0.02 = 0.40$ , measures $\gamma_{\text{self}}$ .

Drop the length term (the row "+ position") and system 1 looks like the best system by a wide margin, 0.63 against 0.24 for system 2. That is the wrong order. Drop position as well ("ratings only") and the ratings barely change. Every pair was shown in both orders, so the missing position term only shrinks the ratings slightly toward zero (0.63 to 0.59 for system 1). This is a known property of logistic regression. Leaving out a variable shrinks the other coefficients toward zero, even when the variable is unrelated to them. On a schedule that is not balanced, the missing term biases the ratings instead, which the next exercise tests.

Why not estimate $\gamma_{\text{self}}$ by adding an $o_i - o_j$ column to the fit? Because that column is already built from the rating columns. With one own-family system it is identical to that system's column.

∴ Self-preference cannot be measured from the judge alone

In the biased-judge model, the self-preference $\gamma_{\text{self}}$ and the ratings enter every verdict only through the shifted ratings $r_i + \gamma_{\text{self}}\,o_i$ . No number of verdicts from that judge separates $\gamma_{\text{self}}$ from the own-family systems' ratings.

Proof. Regroup the log-odds of a verdict as

\big(r_i + \gamma_{\text{self}}\,o_i\big) - \big(r_j + \gamma_{\text{self}}\,o_j\big) + b_{\text{pos}} + \gamma_{\text{len}}(\ell_{\text{first}} - \ell_{\text{second}}).

Raise $\gamma_{\text{self}}$ by any amount $c$ and lower the rating of every own-family system by the same $c$ . Every verdict keeps exactly the same probability, so the likelihood cannot tell the two parameter values apart. In the regression, the column $o_{\text{first}} - o_{\text{second}}$ adds up the $\pm 1$ columns of the own-family systems, so it is a combination of the rating columns. With one own-family system, say system 1, it equals system 1's column ( $+1$ when system 1 is shown first, $-1$ when second, 0 otherwise). In every case the information matrix $\mathbf{X}^\top\mathbf{S}\mathbf{X}$ is singular. ∎

The other two biases can be estimated. Length varies within a system, from response to response, so its column differs from every rating column. Position varies within every pair. Self-preference is a property of a whole system, and from the judge's verdicts alone it looks exactly like that system being better. It can only be measured against another grader, such as humans or a judge from a different family, on the same pairs. This is a favorite interview point, because it explains why a judge that "likes its own model" cannot be caught by looking at its verdicts alone.

Interview

Ratings and position bias from a judge, fit jointly

A judge compared $m$ systems in pairs. For verdict $t$ , first[t] and second[t] are the systems shown first and second, and y[t] is 1 if the first-shown response won. Implement fit_judge_ratings(first, second, y, m). It fits the model $\mathbb{P}(y_t = 1) = \sigma(r_{\text{first}} - r_{\text{second}} + b_{\text{pos}})$ by Newton's method, with the constraint $r_0 = 0$ . It returns (r, b_pos), with r shifted to sum to zero. Real schedules can be lopsided, for example with the new model always in the first slot. Your fit must not let that system's rating absorb the position bias.

import numpy as np

def fit_judge_ratings(first, second, y, m, iters=50):
    raise NotImplementedError

Tests, hints, verified solutions, and AI help, with Pro at $7.99 a month.

Self-preference needs a second grader

So the self-preference audit compares the judge with humans on the same pairs. Take pairs in which one response comes from the judge's own family and the other does not. Collect the judge's verdict on each pair, averaged over both orders, and a human verdict on the same pair. Then compare the two own-family win rates.

The common mistake is to compare the judge's own-family win rate with $\tfrac12$ . The own-family responses may really be better, and only humans on the same pairs say by how much. Then ask how much of the excess is length. Refit both rates with the log-length difference as an extra input, and compare them at equal length, as the length-controlled win rate does.

The evidence that self-preference is real, and not only a matter of style, is strongest for summaries. Panickssery et al. (2024) found that GPT-4, GPT-3.5, and Llama 2 can tell their own summaries from others' better than chance, with no training. Fine-tuning GPT-3.5 and Llama 2 on 500 examples pushed this self-recognition above 90%. Across fine-tuned variants, the strength of self-preference rose linearly with self-recognition. Controlled experiments ruled out the obvious confounders, which the authors present as initial evidence that recognition causes the preference. Zheng et al. (2023) saw GPT-4 give itself a 10% higher win rate than humans gave it, and Claude-v1 25%. They noted that their data could not separate this from real quality differences, which is exactly the identifiability problem above.

Core

A self-preference audit

You have $n$ matched pairs, each an own-family response against another family's. judge_own[i] is the judge's verdict that the own-family response wins, averaged over both orders, so it is 0, 0.5, or 1. human_own[i] is a human's verdict on the same pair (0 or 1). dlen[i] is the log-length ratio, own over other. Implement self_preference_audit(judge_own, human_own, dlen, n_boot, rng, level=0.95). It returns a dict with these entries.

"judge_rate" and "human_rate", the two own-family win rates.
"excess", the judge rate minus the human rate.
"ci", a percentile paired-bootstrap interval for the excess. Draw idx = rng.integers(0, n, size=(n_boot, n)) once and apply it to both arrays.
"lc_excess", the excess at equal length, $\sigma(\hat\theta_0) - \sigma(\hat\phi_0)$ . Here $\hat\theta$ and $\hat\phi$ are logistic fits of the judge's and the humans' verdicts on the inputs $[1, \texttt{dlen}]$ , from the fit_logistic helper.
"length_share", the part of the excess explained by length, $1 - \texttt{lc\_excess}/\texttt{excess}$ .

import numpy as np

def self_preference_audit(judge_own, human_own, dlen, n_boot=2000, rng=None, level=0.95):
    raise NotImplementedError

Tests, hints, verified solutions, and AI help, with Pro at $7.99 a month.

In production the same joint model is a few lines of PyTorch. The same fit works for any input you want to control for, such as length, markdown, or a refusal flag. This block simulates a judge comparing six systems. It recovers the ratings, the position bias, and the length slope with L-BFGS, an optimizer that approximates Newton's method from recent gradients.

🔥 In PyTorchfor a machine with PyTorch

Agreement between two graders

Every claim that a judge "works" is a claim that it agrees with some reference, usually humans. The simplest measure is raw agreement, the fraction of items on which two graders give the same label. It flatters graders who both use one label most of the time. Cohen's kappa corrects for that. It compares the agreement you saw with the agreement two graders would reach by chance, if they labeled independently and each kept their own label frequencies.

A tiny example first. Two raters each say "pass" on half of the items, and they agree on 80% of them. Two raters who flipped coins with those same rates would agree half the time, since $0.5 \cdot 0.5 + 0.5 \cdot 0.5 = 0.5$ . So the real raters beat chance by 0.3, out of the 0.5 by which anyone could beat it. Kappa is $0.3/0.5 = 0.6$ .

This is Cohen's kappa, defined in §14.7, and here is its general form again. Two raters each assign every one of $n$ items to one of $C$ categories. Let $p_o$ be the fraction of items on which they agree, and $p_{1c}$ , $p_{2c}$ the fractions of items each rater put in category $c$ . The chance agreement and kappa are

p_e = \sum_{c=1}^{C} p_{1c}\,p_{2c}, \qquad \kappa = \frac{p_o - p_e}{1 - p_e}.

In plain words, kappa is the share of the possible improvement over chance that the raters achieved. It is 0 at chance and 1 for perfect agreement. §16.2 wrote the same quantities as $A$ and $A_e$ . Three questions remain. When does kappa mislead? How precise is a measured kappa? And what should replace it when categories are ordered or raters are many?

The kappa paradox

Here is the case that makes people distrust kappa. A judge and a human grade 200 answers as pass or fail. Each says "pass" 95% of the time, and they agree on 92% of the answers.

✏️ The kappa paradox by hand

Let $a$ , $b$ , $c$ , $d$ be the shares of items that both pass, that only the judge passes, that only the human passes, and that both fail. Each rater's pass rate gives $a + b = a + c = 0.95$ . Agreement gives $a + d = 0.92$ , and the four shares sum to 1. So $b = c$ . Then $a + 2b + d = 1$ gives $b = c = 0.04$ , and so $a = 0.91$ and $d = 0.01$ .

Chance agreement is $p_e = 0.95^2 + 0.05^2 = 0.9025 + 0.0025 = 0.905$ , so

\kappa = \frac{0.92 - 0.905}{1 - 0.905} = \frac{0.015}{0.095} = 0.158.

Raw agreement is 92% and kappa is 0.16, "slight" agreement on the conventional scale of Landis and Koch (1977). Now split agreement by class. Of the $2a + b + c$ pass labels the two raters gave, $2a$ sit on items the other rater also passed. So when one rater passes an answer, the other passes it too with probability $2a/(2a + b + c) = 1.82/1.90 = 95.8\%$ . When one rater fails an answer, the other fails it too with probability only $2d/(2d + b + c) = 0.02/0.10 = 20\%$ . The raters agree that good answers are good, and almost never agree on which answers are bad. Kappa is low because the failures, the items a judge exists to catch, are where they disagree.

This is the first of two paradoxes that Feinstein and Cicchetti (1990) described. When one category is rare, the chance agreement $p_e$ approaches 1. The same raw agreement then leaves less and less room above chance. Hold $p_o$ at 0.92 and let both raters pass a fraction $q$ of the items. Kappa falls from $(0.92 - 0.5)/0.5 = 0.84$ at $q = 0.5$ to 0.158 at $q = 0.95$ . It crosses zero near $q = 0.958$ , where $q^2 + (1-q)^2 = 0.92$ . At $q = 0.96$ it is slightly negative, $-0.04$ . That is the largest pass rate compatible with 92% agreement, and at it the raters never agree on a single failure.

🔬 The second paradox, unequal marginals

A rater's marginals are its label frequencies. The second paradox runs the other way. Two raters whose marginals differ get a higher kappa than two raters with equal marginals at the same $p_o$ , because unequal marginals lower $p_e$ .

Take two tables of 100 items, with counts $(45, 15; 25, 15)$ and $(25, 35; 5, 35)$ , rows for rater 1 and columns for rater 2. Both have $p_o = 0.60$ . In the first, both raters lean to the first category (rater 1 puts 60% there, rater 2 puts 70%). So $p_e = 0.6 \cdot 0.7 + 0.4 \cdot 0.3 = 0.54$ and $\kappa = 0.13$ . In the second they lean opposite ways (60% and 30%). So $p_e = 0.6 \cdot 0.3 + 0.4 \cdot 0.7 = 0.46$ and $\kappa = 0.26$ . The same agreement earns twice the kappa because the raters disagree about the base rate.

Is the low kappa wrong? Not exactly. It is honest about the disagreement on failures. But it packs two numbers (the agreement on each class) and a prevalence into one. The result depends on the prevalence of the sample you happened to grade. One alternative changes only the chance model. Scott's pi (Scott 1955) computes chance agreement from the pooled marginals, $p_e = \sum_c \bar p_c^2$ with $\bar p_c = (p_{1c} + p_{2c})/2$ . That treats the two raters as interchangeable, and Thakur et al. (2024) used it to compare judges with humans. It shares the prevalence paradox.

🔬 Two corrections that ignore prevalence

PABAK, the prevalence- and bias-adjusted kappa (Byrt, Bishop, and Carlin 1993), sets $p_e = 1/C$ . For two categories this gives $\text{PABAK} = 2p_o - 1$ , here 0.84. It is a rescaling of raw agreement and ignores the marginals entirely.

Gwet's AC1 (Gwet 2008) defines chance agreement as $p_e^{\text{G}} = \frac{1}{C - 1}\sum_c \bar p_c(1 - \bar p_c)$ . For two categories this is $2\bar p(1 - \bar p)$ , here $2(0.95)(0.05) = 0.095$ . That gives $\text{AC1} = (0.92 - 0.095)/(1 - 0.095) = 0.912$ . It stays high at extreme prevalence. Its users like that. Its critics point out that extreme prevalence is exactly where chance agreement is largest.

The honest report for a judge validated against humans includes all of it. Give the prevalence, the raw agreement $p_o$ , kappa with its interval, and the agreement on each class. Better still, give the judge's sensitivity and specificity for the failure class against the human labels, which are what the correction of §19.4 needs.

HealthBench's authors scored their grader in this spirit. They computed the F1 score (the harmonic mean of precision and recall) of each class, "met" and "not met", and averaged the two, so that a rare class counts as much as a common one. On its 34 consensus criteria, their GPT-4.1 grader reached 0.709 against physicians' labels. That beat the average physician in five of seven themes.

The widget below runs the paradox. Two raters each pass 97% of good answers and fail the share of bad answers you set, independently of each other, and you move the share of good answers. The first preset is the worked example. With the careful raters, who fail 90% of bad answers, slide good answers from 50% to 99%. Raw agreement climbs from 88% to 94% while kappa falls from 0.76 to 0.20, and kappa follows the agreement on fails all the way.

In practice, oversample failures when you build a validation set, so that agreement on them is measured precisely. Interviewers love this paradox, and the third interview question below is the classic version.

How precise is a kappa

A kappa from 200 items is an estimate, and reporting it without an interval invites the mistakes of §0.11. The tool for its variance is the delta method. Near the truth, a smooth function of a noisy estimate is almost a linear function of it. So its variance is almost the variance of that linear function. In symbols, let an estimate $\hat{\mathbf{p}}$ have covariance $\boldsymbol{\Sigma}/n$ and let $f$ be smooth. The first-order Taylor expansion $f(\hat{\mathbf{p}}) \approx f(\mathbf{p}) + \nabla f(\mathbf{p})^\top(\hat{\mathbf{p}} - \mathbf{p})$ gives $\operatorname{Var} f(\hat{\mathbf{p}}) \approx \nabla f^\top\boldsymbol{\Sigma}\,\nabla f / n$ .

The crude version treats $p_e$ as a known constant. Then $\hat\kappa$ is a linear function of $\hat p_o$ . That is a proportion over $n$ independent items, with variance $p_o(1 - p_o)/n$ , so

\operatorname{SE}(\hat\kappa) \approx \frac{\sqrt{p_o(1 - p_o)/n}}{1 - p_e}.

This is the approximation in the paper that introduced kappa (Cohen 1960). It ignores that $p_e$ is estimated from the same data. The full delta method does not.

∴ Large-sample variance of Cohen's kappa

Let $p_{ab}$ be the proportion of items that rater 1 put in category $a$ and rater 2 in category $b$ , with row sums $p_{a\cdot}$ and column sums $p_{\cdot b}$ . Then for $n$ independent items

\operatorname{Var}(\hat\kappa) \approx \frac{1}{n(1-p_e)^4}\Big[\sum_{a,b} p_{ab}\,g_{ab}^2 - \Big(\sum_{a,b} p_{ab}\,g_{ab}\Big)^2\Big], \qquad g_{ab} = \mathbb{1}[a = b](1 - p_e) - (p_{\cdot a} + p_{b\cdot})(1 - p_o),

with $\sum_{ab} p_{ab}g_{ab} = p_o + p_op_e - 2p_e$ . This is the formula of Fleiss, Cohen, and Everitt (1969).

∂ The delta method for kappa

The counts in the $C \times C$ table are multinomial. Flatten the table into a vector $\mathbf{p}$ . The cell proportions $\hat p_{ab}$ then have covariance $(\operatorname{diag}(\mathbf{p}) - \mathbf{p}\mathbf{p}^\top)/n$ . Hence for any smooth $f$ , $n\operatorname{Var} f(\hat{\mathbf{p}}) \approx \sum_{ab} p_{ab}\,(\partial_{ab} f)^2 - \big(\sum_{ab} p_{ab}\,\partial_{ab} f\big)^2$ , writing $\partial_{ab} = \partial/\partial p_{ab}$ .

Now differentiate $\kappa = (p_o - p_e)/(1 - p_e)$ . The cell $p_{ab}$ enters $p_o = \sum_c p_{cc}$ only if $a = b$ , so $\partial_{ab}p_o = \mathbb{1}[a = b]$ . It enters $p_e = \sum_c p_{c\cdot}\,p_{\cdot c}$ twice. It is part of the row sum $p_{a\cdot}$ , which is multiplied by $p_{\cdot a}$ . It is also part of the column sum $p_{\cdot b}$ , which is multiplied by $p_{b\cdot}$ . So $\partial_{ab}p_e = p_{\cdot a} + p_{b\cdot}$ . By the quotient rule,

\partial_{ab}\kappa = \frac{(\partial_{ab}p_o - \partial_{ab}p_e)(1 - p_e) + (p_o - p_e)\,\partial_{ab}p_e}{(1 - p_e)^2} = \frac{\mathbb{1}[a=b](1 - p_e) - (p_{\cdot a} + p_{b\cdot})(1 - p_o)}{(1 - p_e)^2} = \frac{g_{ab}}{(1-p_e)^2}.

Substituting into the multinomial formula gives the theorem. For the mean term, $\sum_{ab} p_{ab}\mathbb{1}[a = b] = p_o$ , $\sum_{ab}p_{ab}\,p_{\cdot a} = \sum_a p_{a\cdot}\,p_{\cdot a} = p_e$ , and likewise $\sum_{ab}p_{ab}\,p_{b\cdot} = p_e$ . So $\sum_{ab}p_{ab}g_{ab} = p_o(1 - p_e) - 2p_e(1 - p_o) = p_o + p_op_e - 2p_e$ .

Here are both formulas on the paradox table, checked against 20,000 simulated studies of 200 items each.

Python · cell 4

Why does the crude formula overestimate here? At extreme prevalence the estimated $p_o$ and $p_e$ move together, since a sample with more passes raises both. So part of their noise cancels in $p_o - p_e$ , and the crude formula misses that. Either way the lesson is the same. A kappa from a couple of hundred items at extreme prevalence is barely a measurement.

When no formula fits, use the bootstrap of §0.11. Resample items, each with both raters' labels, and recompute kappa each time. Never resample the two raters' labels separately, because that destroys the pairing that agreement is about.

Weighted kappa for ordered scales

On a 1 to 5 scale, a judge that says 4 when the human says 5 is closer than one that says 1. Cohen's kappa counts both as plain disagreement. Weighted kappa (Cohen 1968) charges each disagreement by its size on the scale.

📐 Weighted kappa

Two raters label $n$ items on an ordered scale of $C$ categories, indexed $a, b \in \{1, \dots, C\}$ by position on the scale. Let $O_{ab}$ be the observed proportion of items with labels $(a, b)$ . Let $E_{ab} = p_{1a}\,p_{2b}$ be the proportion expected if the raters labeled independently with their own marginals. Let $w_{ab} \ge 0$ be a disagreement weight with $w_{aa} = 0$ . Then

\kappa_w = 1 - \frac{\sum_{a,b} w_{ab}\,O_{ab}}{\sum_{a,b} w_{ab}\,E_{ab}}.

Linear weights are $w_{ab} = \lvert a - b\rvert/(C - 1)$ , and quadratic weights are $w_{ab} = (a - b)^2/(C - 1)^2$ .

In plain words, weighted kappa is one minus the ratio of observed disagreement to chance disagreement, where disagreement is measured in steps on the scale. Scaling every weight by the same constant leaves $\kappa_w$ unchanged, since the constant cancels in the ratio. The division by $C - 1$ matters only in the equivalent "agreement weight" form $1 - w_{ab}$ that some software uses.

Quadratic weights have a second justification. With them, $\kappa_w$ is closely related to the intraclass correlation of the two raters' numeric scores (Fleiss and Cohen 1973). So it measures how well the judge's numbers track the human's, like a correlation that also penalizes a constant offset. Here is a judge and a human on a 3-point scale (1 bad, 2 acceptable, 3 good) over 100 items, checked against scikit-learn.

Python · cell 5

✏️ Weighted kappa on a 3 by 3 table

The judge's marginals are $(0.26, 0.43, 0.31)$ and the human's are $(0.28, 0.43, 0.29)$ . Raw agreement is $p_o = 0.71$ , and $p_e = 0.26 \cdot 0.28 + 0.43 \cdot 0.43 + 0.31 \cdot 0.29 = 0.3476$ . So $\kappa = 0.3624/0.6524 = 0.5555$ .

With quadratic weights, an off-by-one cell costs $\tfrac14$ and an off-by-two cell costs 1. Observed disagreement is $\tfrac14(0.05 + 0.06 + 0.07 + 0.08) + (0.01 + 0.02) = 0.095$ . Chance disagreement is $\tfrac14(0.1118 + 0.1204 + 0.1247 + 0.1333) + (0.0754 + 0.0868) = 0.28475$ . So $\kappa_w = 1 - 0.095/0.28475 = 0.6664$ . Linear weights give 0.6072.

Most disagreements here are by one step, so the weighted values are higher than the unweighted 0.5555. Quadratic weights forgive a single step the most, so here they give the highest value.

In practice, report quadratic weighted kappa for a 1 to 5 judge scale, and say which weights you used, because the choice changes the number.

Interview

Cohen's kappa with an interval, and weighted kappa

This is the function you would write before trusting a judge with a launch decision. It has two parts.

First, implement weighted_kappa(r1, r2, categories, weights="quadratic") for two raters' labels. categories lists the scale in order, and it may include levels nobody used. weights is "linear", "quadratic", or None for Cohen's kappa. Return 1.0 when the chance disagreement is zero, which happens when both raters used one and the same category.

Second, implement kappa_with_ci(r1, r2, categories, weights=None, n_boot=2000, rng=None, level=0.95). It returns (kappa, lo, hi), where (lo, hi) is a percentile bootstrap interval over items. Draw the replicates as idx = rng.integers(0, n, size=(n_boot, n)).

import numpy as np

def weighted_kappa(r1, r2, categories, weights="quadratic"):
    raise NotImplementedError

def kappa_with_ci(r1, r2, categories, weights=None, n_boot=2000, rng=None, level=0.95):
    raise NotImplementedError

Tests, hints, verified solutions, and AI help, with Pro at $7.99 a month.

Many raters and missing ratings

Real labeling projects have three judges and five humans, and not everyone labels every item. Cohen's kappa is defined for exactly two raters. Fleiss' kappa handles any fixed number of ratings per item. Its raters need not be the same people from item to item, which is how crowd labeling works. Krippendorff's alpha handles any number of raters, missing ratings, and any type of scale. It is the standard choice when the data are messy.

📐 Fleiss' kappa

Each of $n$ items is rated $R \ge 2$ times in $C$ unordered categories, and $n_{uc}$ is the number of ratings of item $u$ in category $c$ , so $\sum_c n_{uc} = R$ . Item $u$ has $R(R - 1)$ ordered pairs of two distinct ratings. Its agreement is the fraction of those pairs whose two ratings match,

P_u = \frac{\sum_{c} n_{uc}(n_{uc} - 1)}{R(R - 1)} = \frac{\sum_c n_{uc}^2 - R}{R(R - 1)}.

Let $p_c = \frac{1}{nR}\sum_u n_{uc}$ be the pooled share of category $c$ . The observed and chance agreements are $\bar P = \frac{1}{n}\sum_u P_u$ and $\bar P_e = \sum_c p_c^2$ , and Fleiss' kappa (Fleiss 1971) is

\kappa_{\text{F}} = \frac{\bar P - \bar P_e}{1 - \bar P_e}.

In plain words, $P_u$ is the probability that two distinct ratings of item $u$ , drawn at random, agree. $\bar P_e$ is the probability that two ratings drawn at random from the whole pool agree. With $R = 2$ , $P_u$ is 1 when the two ratings agree and 0 when they do not, so $\bar P = p_o$ . And $p_c = (p_{1c} + p_{2c})/2$ is the pooled marginal, so $\kappa_{\text{F}}$ is exactly Scott's pi. It inherits the pooled chance model, and with it the prevalence paradox.

Krippendorff's alpha starts from a different idea. Forget which rater gave which value, and look only at pairs of values within one item (Krippendorff calls an item a unit). Unit $u$ with $n_u \ge 2$ ratings has $n_u(n_u - 1)$ ordered pairs of values from different raters. Give each pair the weight $1/(n_u - 1)$ , so that each of the $n_u$ values carries total weight 1. Units with a single rating have no pairs and are dropped.

📐 Krippendorff's alpha

The coincidence matrix $o_{ck}$ is the weighted number of within-unit ordered pairs with values $(c, k)$ ,

o_{ck} = \sum_{u\,:\,n_u \ge 2}\frac{\#\{\text{ordered pairs of different raters in } u \text{ with values } (c, k)\}}{n_u - 1}.

Let $n_c = \sum_k o_{ck}$ be the number of pairable values equal to $c$ , and $N = \sum_c n_c$ the number of pairable values. Take a disagreement weight $w_{ck}$ , with $w_{ck} = \mathbb{1}[c \ne k]$ for unordered categories and $(c - k)^2$ for numeric data. The observed and expected disagreements and Krippendorff's alpha are

D_o = \frac{1}{N}\sum_{c,k} w_{ck}\,o_{ck}, \qquad D_e = \frac{1}{N(N-1)}\sum_{c,k} w_{ck}\,n_c\,n_k, \qquad \alpha_{\text{K}} = 1 - \frac{D_o}{D_e}.

In plain words, $D_o$ is how often two values from the same item disagree. $D_e$ is how often two values drawn at random from the whole pool, without replacement, would disagree. For unordered categories $D_e = (N^2 - \sum_c n_c^2)/(N(N-1))$ . Alpha is 1 for perfect agreement, and 0 when values within an item are no more alike than random values.

✏️ Fleiss' kappa and Krippendorff's alpha by hand

Three raters label four items, with ratings $(1, 1, 1)$ , $(1, 1, 2)$ , $(2, 2, 2)$ , and $(1, 2, 2)$ . The counts per item are $(3, 0)$ , $(2, 1)$ , $(0, 3)$ , and $(1, 2)$ . So $P_u$ is $(9 - 3)/6 = 1$ , then $(4 + 1 - 3)/6 = \tfrac13$ , then 1, then $\tfrac13$ , and $\bar P = \tfrac23$ . Six of the twelve ratings are 1s, so $p_1 = p_2 = \tfrac12$ and $\bar P_e = \tfrac12$ . Hence $\kappa_{\text{F}} = (\tfrac23 - \tfrac12)/\tfrac12 = \tfrac13$ .

Now drop one of item 2's 1s, as if that rater skipped it. The ratings are $(1, 1, 1)$ , $(1, 2)$ , $(2, 2, 2)$ , and $(1, 2, 2)$ . The first item has $n_u = 3$ and six ordered pairs, all $(1,1)$ , each of weight $\tfrac12$ , so $o_{11}$ gains 3. The second has two ordered pairs, $(1,2)$ and $(2,1)$ , each of weight 1. The third adds 3 to $o_{22}$ . The fourth has the pairs $(1,2)$ twice, $(2,1)$ twice, and $(2,2)$ twice, each of weight $\tfrac12$ . So

o = \begin{pmatrix}3 & 2\\ 2 & 4\end{pmatrix}, \qquad n_1 = 5, \quad n_2 = 6, \quad N = 11,

since all 11 ratings are pairable. Then $D_o = 4/11$ , $D_e = (121 - 25 - 36)/110 = 6/11$ , and $\alpha_{\text{K}} = 1 - 4/6 = \tfrac13$ .

When no rating is missing, alpha is Fleiss' kappa up to a small-sample factor.

∂ Proof that alpha is Fleiss' kappa up to the factor (N - 1)/N

Every unit has $n_u = R$ , so unit $u$ adds $n_{uc}(n_{uk} - \mathbb{1}[c = k])/(R - 1)$ to $o_{ck}$ . Its off-diagonal mass is $\big(R^2 - \sum_c n_{uc}^2\big)/(R - 1) = R(1 - P_u)$ , so $D_o = \frac{1}{N}\sum_u R(1 - P_u) = 1 - \bar P$ . The pairable counts are $n_c = Np_c$ , so $D_e = (N^2 - \sum_c n_c^2)/(N(N - 1)) = (1 - \bar P_e)\,N/(N - 1)$ . Then $\alpha_{\text{K}} = 1 - \frac{N-1}{N}\cdot\frac{1 - \bar P}{1 - \bar P_e}$ , and $\frac{1 - \bar P}{1 - \bar P_e} = 1 - \kappa_{\text{F}}$ . ∎

For the four-item example with all twelve ratings, $N = 12$ and $\alpha_{\text{K}} = 1 - \tfrac23 \cdot \tfrac{11}{12} = \tfrac{7}{18} \approx 0.389$ , slightly above $\kappa_{\text{F}} = \tfrac13$ . The factor $(N-1)/N$ goes to 1 as the data grow. Alpha also inherits the paradox, since it too corrects for chance with pooled marginals. Krippendorff (2011) works through the computation for every type of scale.

Warm-up

Fleiss' kappa, and how it meets Scott and Krippendorff

Implement fleiss_kappa(counts). The array counts has shape (n_items, C), and counts[u, c] is the number of ratings of item u in category c. Every row must sum to the same $R \ge 2$ . Raise ValueError otherwise. Return $\kappa_{\text{F}}$ , or 1.0 when $\bar P_e = 1$ (every rating in one category).

import numpy as np

def fleiss_kappa(counts):
    raise NotImplementedError

Tests, hints, verified solutions, and AI help, with Pro at $7.99 a month.

Core

Krippendorff's alpha with missing ratings

Implement krippendorff_alpha_nominal(R) for unordered categories. R is a raters by units array of numeric labels, with np.nan for a missing rating. Build the coincidence matrix with the $1/(n_u - 1)$ weighting, drop units with fewer than two ratings, and return $\alpha_{\text{K}}$ .

import numpy as np

def krippendorff_alpha_nominal(R):
    raise NotImplementedError

Tests, hints, verified solutions, and AI help, with Pro at $7.99 a month.

The table collects the statistics of this part.

statistic	raters	missing ratings	scale	chance model	watch out for
percent agreement $p_o$	2	no	any	none	inflated by a common class
Cohen's $\kappa$	2	no	unordered	each rater's own marginals	prevalence paradox, rewards unequal marginals
Scott's $\pi$ , Fleiss' $\kappa_{\text{F}}$	2, or a fixed $R$ per item	no	unordered	pooled marginals	prevalence paradox
weighted $\kappa_w$	2	no	ordered	own marginals	choice of weights changes the number
Krippendorff's $\alpha_{\text{K}}$	any	yes	unordered, ordered, numeric	pooled values	prevalence paradox, needs enough pairable values
Gwet's AC1, PABAK	2 (AC1 extends to more)	AC1 extends	unordered	chosen to stay high at extreme prevalence	close to raw agreement at extreme prevalence
per-class agreement, sensitivity and specificity	judge against reference	no	binary per criterion	none	needs the reference to be right

Item-level and system-level agreement

A judge can agree with humans on single answers, which is item-level agreement. Or it can agree with them on which system is better overall, which is system-level agreement. These are different properties, and a judge can have one without the other.

Here is the puzzle that shows it. Thakur et al. (2024) tested 13 judge models on questions where humans agree with each other well. Only the largest judges aligned reasonably with humans on single answers. Even those gave overall scores up to 5 points away from the human scores. Yet when the task was to rank nine "exam-taker" models, smaller judges did reasonably well. So did a plain string check that asks whether the answer contains the reference.

At the other end is Arena-Hard-Auto (Li et al. 2024). It grades 500 hard prompts with a judge against a fixed baseline. Its reference is Chatbot Arena, a public site where people compare two anonymous models' answers and vote (§22.7 builds its ranking). Against the arena's human ranking with the effect of answer style removed, Arena-Hard-Auto's ranking has a Spearman correlation of 98.6% when it removes style the same way, and 94.9% without that adjustment. MT-Bench reaches 90.7% and length-controlled AlpacaEval 88.1%. Arena-Hard-Auto also separates 87.4% of model pairs with non-overlapping confidence intervals, against 22.6% for MT-Bench, at about 20 US dollars a run.

How can a judge be mediocre on items and excellent on systems? Because averages forgive noise and do not forgive bias. Here is a tiny example. Two systems differ in true mean score by 0.2. The judge's error on one item has a standard deviation of 1, five times that gap, so single verdicts are unreliable. Average 500 items and the error of each system's mean shrinks to $1/\sqrt{500} \approx 0.045$ , small next to 0.2. Now give the worse system a constant bonus of 0.2. Averaging never shrinks it, and the two systems now look tied however many items you grade.

To state this precisely we need a measure of how well two rankings agree. Kendall's tau looks at every pair of systems and asks whether the two rankings order the pair the same way. It is the share of pairs ordered the same way minus the share ordered oppositely. So it is 1 for identical rankings, $-1$ for reversed ones, and near 0 for unrelated ones.

∴ System-level agreement from noisy item-level judgments

Let $m$ systems each have $n$ graded items. Model the human score of system $s$ on item $i$ as $h_{si} = \mu_s + v_{si}$ , and the judge's as $j_{si} = h_{si} + b_s + \eta_{si}$ . The system means $\mu_s$ have variance $\sigma_\mu^2$ across systems. The item deviations $v_{si}$ have variance $\sigma_v^2$ . The judge's per-system bias $b_s$ has variance $\sigma_{\text{bias}}^2$ , and its item noise $\eta_{si}$ has variance $\sigma_\eta^2$ . All are independent with mean zero. Then the item-level and system-level correlations are

\rho_{\text{item}} = \sqrt{\frac{\sigma_\mu^2 + \sigma_v^2}{\sigma_\mu^2 + \sigma_v^2 + \sigma_{\text{bias}}^2 + \sigma_\eta^2}}, \qquad \rho_{\text{sys}}(n) = \sqrt{\frac{\sigma_\mu^2 + \sigma_v^2/n}{\sigma_\mu^2 + \sigma_v^2/n + \sigma_{\text{bias}}^2 + \sigma_\eta^2/n}}.

As $n \to \infty$ , $\rho_{\text{sys}} \to \sigma_\mu/\sqrt{\sigma_\mu^2 + \sigma_{\text{bias}}^2}$ , which is 1 if and only if the judge has no system-level bias. If the system means are jointly normal, the expected Kendall tau between the judge's and the humans' system means is $\tfrac{2}{\pi}\arcsin\rho_{\text{sys}}$ . Here $\rho$ is the ordinary (Pearson) correlation, not Spearman's.

∂ Proof of the attenuation formulas

Average over the $n$ items of system $s$ , writing bars for means, $\bar h_s = \mu_s + \bar v_s$ and $\bar j_s = \bar h_s + b_s + \bar\eta_s$ . The mean of $n$ independent terms of variance $\sigma^2$ has variance $\sigma^2/n$ , so $\operatorname{Var}(\bar h_s) = \sigma_\mu^2 + \sigma_v^2/n$ . Since $b_s$ and $\bar\eta_s$ are independent of $\bar h_s$ ,

\operatorname{Cov}(\bar j_s, \bar h_s) = \operatorname{Var}(\bar h_s), \qquad \operatorname{Var}(\bar j_s) = \operatorname{Var}(\bar h_s) + \sigma_{\text{bias}}^2 + \sigma_\eta^2/n .

Therefore $\operatorname{Corr}(\bar j_s, \bar h_s) = \operatorname{Var}(\bar h_s)/\sqrt{\operatorname{Var}(\bar h_s)\operatorname{Var}(\bar j_s)} = \sqrt{\operatorname{Var}(\bar h_s)/\operatorname{Var}(\bar j_s)}$ , which is $\rho_{\text{sys}}(n)$ . The item-level formula is the same computation with $n = 1$ , the correlation over all system-item cells pooled. As $n \to \infty$ the terms with $1/n$ vanish and $\rho_{\text{sys}}^2 \to \sigma_\mu^2/(\sigma_\mu^2 + \sigma_{\text{bias}}^2)$ .

The last claim needs two more facts. First, for a pair of random variables $(X, Y)$ the population Kendall tau is $\tau = \mathbb{P}(\text{concordant}) - \mathbb{P}(\text{discordant})$ for two independent draws $(X_1, Y_1)$ and $(X_2, Y_2)$ , where concordant means $(X_1 - X_2)(Y_1 - Y_2) > 0$ . For continuous variables ties have probability zero, so $\tau = 2\,\mathbb{P}(\text{concordant}) - 1$ . The sample tau over $m$ systems averages the sign of $(x_s - x_t)(y_s - y_t)$ over all $\binom{m}{2}$ pairs. Each term has expectation $\tau$ when the systems are independent draws, so the average does too.

Second, the differences $D_X = X_1 - X_2$ and $D_Y = Y_1 - Y_2$ are jointly normal with mean zero, variances $2\operatorname{Var}X$ and $2\operatorname{Var}Y$ , and covariance $2\operatorname{Cov}(X, Y)$ , so their correlation is the same $\rho$ . The orthant formula of Sheppard (1899), also used in §16.1, says that for such a pair $\mathbb{P}(D_X > 0, D_Y > 0) = \tfrac14 + \tfrac{1}{2\pi}\arcsin\rho$ . To see it, standardize and write $D_X = Z_1$ and $D_Y = \rho Z_1 + \sqrt{1 - \rho^2}\,Z_2$ , with $Z_1, Z_2$ independent standard normals. The vector $(Z_1, Z_2)$ points in a uniformly random direction. The event $D_X > 0$ is a half-plane with normal $(1, 0)$ , and $D_Y > 0$ is a half-plane with normal $(\rho, \sqrt{1 - \rho^2})$ , at angle $\arccos\rho$ from the first. Two half-planes whose normals differ by an angle $\phi$ intersect in a wedge of angle $\pi - \phi$ . So the probability is $(\pi - \arccos\rho)/(2\pi) = \tfrac14 + \tfrac{1}{2\pi}\arcsin\rho$ , using $\arccos\rho = \tfrac{\pi}{2} - \arcsin\rho$ . By symmetry both differences are negative with the same probability, so $\mathbb{P}(\text{concordant}) = \tfrac12 + \tfrac{1}{\pi}\arcsin\rho$ and $\tau = \tfrac{2}{\pi}\arcsin\rho$ . Applied to the system means, with $\rho = \rho_{\text{sys}}$ , this is the theorem's last claim.

In plain words, averaging over $n$ items divides the judge's item noise by $n$ , so the noise disappears from system rankings. A bias that attaches to a whole system does not average away at all. Self-preference for one family, or a taste for one system's style, caps the agreement you can reach however many items you grade.

Let's check the formulas. Systems differ by $\sigma_\mu = 0.2$ and items vary by $\sigma_v = 1$ within a system, so systems differ by a fifth of the item spread, a realistic ratio. The judge's item noise is as large as the item variation, $\sigma_\eta = 1$ . The cell simulates 2,000 leaderboards of 20 systems. It draws each system's average directly from its exact distribution, since a mean of $n$ independent normals with variance $\sigma^2$ is normal with variance $\sigma^2/n$ .

Python · cell 6

So the two kinds of agreement answer different questions. If the judge ranks systems or picks checkpoints, check its system-level agreement with humans, and hunt for biases that attach to systems, such as self-preference and style. If the judge decides about single responses (filtering training data, gating a response, acting as a reward), check item-level agreement, because no averaging rescues a bad single verdict. Arena-Hard-Auto's 98.6% is a statement of the first kind and says little about the second. An interviewer who asks "is this judge good?" wants you to ask back "good for what?".

In the widget, eight systems have true means with a spread of about 0.2, and as in the cell the judge's noise on one answer is as large as the item spread, $\sigma_\eta = \sigma_v = 1$ . Under the picture, the average Kendall tau and the share with the same top system come from 200 fresh leaderboards. Press Play to grade more items per system and watch the lines between the two rankings straighten. Then give one system a bonus and play again.

Leaderboards are rankings, and coarse scores produce ties. So the natural statistic for system-level agreement is a rank correlation that handles ties.

Without ties this equals the plain $\tau_a = (n_{\text{con}} - n_{\text{dis}})/\binom{m}{2}$ . With ties, $\tau_a$ is pulled toward zero, so even two identical rankings that contain a tie score below 1, while $\tau_b$ gives them exactly 1. The uncertainty comes from the prompts, which are the sample. So a bootstrap interval resamples prompts and recomputes every system's mean. The systems are the fixed objects of interest and are never resampled. Spearman's rank correlation is the other common choice and behaves similarly.

Core

Does the judge rank systems like humans?

J and H are arrays of shape (n_prompts, n_systems), holding the judge's and the humans' scores of every system on every prompt. First, implement kendall_tau_b(x, y) from scratch, handling ties. Then implement system_agreement(J, H, n_boot, rng, level=0.95). It returns a dict with three entries.

"tau_b", the tau-b between the per-system mean scores.
"ci", a percentile bootstrap interval over prompts, with replicates drawn as idx = rng.integers(0, n_prompts, size=(n_boot, n_prompts)).
"item_spearman", the Spearman correlation over all prompt-system cells. You may use scipy.stats.spearmanr.

import numpy as np
from scipy.stats import spearmanr

def kendall_tau_b(x, y):
    raise NotImplementedError

def system_agreement(J, H, n_boot=1000, rng=None, level=0.95):
    raise NotImplementedError

Tests, hints, verified solutions, and AI help, with Pro at $7.99 a month.

Where judges fail at the frontier

Everything so far assumes the judge measures the right thing, with some noise and bias. On the hardest problems that assumption breaks in four ways.

Judges cannot grade what they cannot solve. JudgeBench (Tan et al. 2024) took hard problems whose answers can be checked, in knowledge, reasoning, math, and coding. It turned them into 350 pairs, each with one correct and one incorrect response. Both responses came from GPT-4o, so the wrong answers look plausible. Each pair is judged in both orders, and it counts as correct only if the two verdicts pick the right answer, or one picks it and the other calls a tie. A guess made once per pair would score 50%, which the authors take as chance. GPT-4o as a judge scored 50.9% with a plain prompt, and 56.6% with the Arena-Hard judge prompt. All but two of the judge models fine-tuned for grading scored below 50%. The pairs are hardest for GPT-4o itself, since it wrote both responses, and the authors note that bias. In the updated paper, reasoning models did much better, 75.4% for o1-preview and 80.9% for o3-mini at high effort. Each model's accuracy as a judge closely tracked its own accuracy at solving the problems.

Benchmarks of judges built from human preferences miss this, because crowd raters cannot check these answers either. The remedies are a reference answer, a program that checks the answer where one exists (§18.3), and a stronger or reasoning judge. When the models being graded exceed every available judge, the remaining option is the research program of scalable oversight, which studies how weaker graders can supervise stronger models.

Judges can be attacked. The fixed text from the opening worked in two steps. First, a structured response imitated the judge's own template and rewrote the comparison it was asked to make. That alone reached a 76.8% length-controlled win rate on AlpacaEval 2.0. Then a short prefix found by random search against GPT-4 raised it to 86.5%. The prefix was tuned on instructions from a public dataset, UltraFeedback, not on the benchmark's own (Zheng et al. 2024). So the attack needs no access to the benchmark's questions. It exploits the fact that the response being graded is text the judge reads, the same weakness as prompt injection against agents (§19.3).

The defenses are the ones a security engineer would expect. Delimit and clean the response. Test the judge with adversarial and constant responses before trusting a leaderboard. Flag responses whose content does not depend on the prompt. Use several judges from different families, keep the judge's instructions private, and keep humans in the loop for anything high-stakes.

Judges drift. A judge is a model version plus a prompt, and changing either moves every number. Verga et al. (2024) found GPT-4 a relatively weak judge in some settings, with high variance under small changes to the prompt. Pin the judge model and the prompt, version them with the eval, and rerun the validation set whenever either changes.

Judges can be contaminated. §22.3 was about a model that saw the test before it was graded. A judge has its own version of the problem, which Li et al. (2025) call preference leakage. Much post-training data is now written by strong models, and the same strong models are the usual judges. A judge then grades students trained on its own writing, and it likes what it recognizes.

Li et al. had GPT-4o, Gemini-1.5-flash, and Llama-3.3-70B each answer the same 30,000 prompts. They fine-tuned small models (Mistral-7B and Qwen-2.5-14B) on each set of answers, giving one "student" per teacher. Then the same three models judged the students against each other on Arena-Hard and AlpacaEval 2.0. In most pairs, each judge favored the student trained on its own outputs. Their preference leakage score measures by how much, relative to what the two judges say on average.

✏️ Preference leakage from published win rates

On AlpacaEval 2.0, two Mistral-7B students were compared, one trained on GPT-4o's answers and one on Gemini-1.5-flash's. The GPT-4o judge gave its own student a 55.1% win rate. The Gemini judge gave that same student 36.8%. The two judges' average for the GPT-4o student is $(55.1 + 36.8)/2 = 45.95\%$ . So its own teacher rated it $(55.1 - 45.95)/45.95 = 19.9\%$ higher than the average.

The other direction works the same way. The Gemini judge gave its own student 63.2%, and the GPT-4o judge gave it 44.9%. Their average is 54.05%, and the excess is $(63.2 - 54.05)/54.05 = 16.9\%$ . The preference leakage score is the mean of the two excesses, $(19.9\% + 16.9\%)/2 = 18.4\%$ , the value Li et al. report. On Arena-Hard the same pair scored 28.7%, so the average over the two benchmarks is 23.6%.

An excess of about a fifth in relative win rate is large. It also survives, at a smaller size, when the judge is only a sibling of the teacher, as the summary of Li et al.'s findings shows.

🔬 How preference leakage varies

The bias is not universal, and its size depends on how related the judge is to the teacher. One pair, GPT-4o and Llama-3.3, showed almost no effect. The bias grew with the share of synthetic data in the student's training mix. It persisted at 8.9% when the judges were only siblings from the same series, with GPT-4-turbo and Gemini-1.5-pro judging the students of GPT-4o and Gemini-1.5-flash. It fell to 2.8% for siblings from an earlier series (GPT-3.5-turbo and Gemini-1.0-pro). It was largest on subjective questions, such as writing and programming, and smallest on math with definite answers.

The judges could not recognize their students' responses. Asked to pick them out, they scored near or below chance. Yet a small BERT classifier told the three students apart 82.4% of the time. What leaks is a style and format the judge likes, not a signature it knows.

Preference leakage hides from the judge's own verdicts for the same reason self-preference does. The bias attaches to a whole system, so in the biased-judge model it only shifts that system's rating. From judge data alone, a favored student looks exactly like a better one. It is also harder to audit. The thing to record is no longer the student's model family but where its training data came from. You may know that for your own models, and you rarely know it for anyone else's. And like every bias that attaches to a system, it does not average away however many items you grade, as the theorem on system-level agreement showed.

🔬 Leakage through the benchmark itself, and what to do

A related channel runs through the benchmark. A judge may have been trained on a benchmark's reference answers, or may come from the family that wrote its synthetic items. Then it shares a source with the items and may reward resemblance to them. We know of no clean measurement of that channel, so treat it as a reason for the same defenses rather than as an effect of known size.

The defenses are a judge, or a jury, from families unrelated to whatever generated the training data of the systems being compared. Add a record of each system's data sources when you have it, and a human-labeled slice, the one grader related to none of the models. Li et al. also tried fixes, in a separate test with human-labeled preference data. The most effective adjusted the judge's verdicts with a correction fitted on a held-out set. It cut their error-bias measure from 17.8 to 7.3. That is the same idea as correcting a judge with human labels.

Two mitigations recur across all four failures. The first is a jury, which replaces one large judge with several smaller ones from different families and takes a vote. Verga et al.'s panel of three models from different families (Command R, Claude 3 Haiku, and GPT-3.5) reached a higher Cohen's kappa with human judgments than GPT-4 alone on three question-answering sets. On Natural Questions it was 0.763 against 0.627. The panel also showed less bias toward any one family, and it cost over seven times less. A vote of independent judges improves like the majority of several annotators in §16.2, but only to the extent that their errors are independent. That is why the families should differ.

The second is calibration with humans, which turns a biased judge into an unbiased estimate. Label a random slice of a few hundred items with people. Then correct the judge's mean in one of two ways. Prediction-powered inference adds the judge's average error on the labeled slice to its average on everything, and it needs no model of the judge's errors (§14.7, from Angelopoulos et al. 2023). Or estimate the judge's sensitivity and specificity and invert them (§19.4). Either way, the human slice is what makes the number defensible.

Open evaluator models give you control over all of this. Prometheus 2 (Kim et al. 2024) is an open model trained to grade against criteria you write, in both pointwise and pairwise formats. It is the kind of judge you can pin forever and audit fully.

⚠️ Judge mistakes that look like results

Judging in one order only, so that the win rate measures the slot. Comparing the judge's own-family win rate with 50% instead of with humans on the same pairs. Reporting raw agreement on a sample where 95% of answers pass. Reporting kappa from 100 items without an interval (at extreme prevalence its standard error can exceed 0.1). Quoting a system-level correlation as evidence that the judge can filter single responses. Letting the judge grade problems it cannot solve. Judging students with the model that wrote their training data. Thresholding a judge's probabilities without checking their calibration. Changing the judge's model version in the middle of a project. And validating the judge on the same items you tuned its prompt on, which overfits the prompt exactly as tuning a model on its test set overfits the model (§22.1).

A checklist for validating a judge

Before a judge's numbers drive a decision, collect a random sample of 200 to 500 items. Stratify it by the failure modes you care about, and have at least two humans label each item with the rubric fixed in advance. Then measure these six things, in order.

Agreement between the humans, with an interval. It is the ceiling a judge can be compared with.
The judge's agreement with the humans on the same items, on items not used to tune the judge prompt. Report kappa with an interval and agreement on each class, or sensitivity and specificity for each criterion.
The calibration of the judge's probabilities, if you will use them for expected scores or thresholds. Report a reliability table and an ECE against the human labels, with a temperature or Platt repair fitted on one split and checked on another.
The bias audit. Measure position from both orders, length from a fitted slope, and self-preference against the human labels. Record whether the judge's family wrote any of the compared systems' training data.
System-level agreement, if the judge will rank systems.
Adversarial behavior, with constant, empty, very long, and judge-addressing responses.

Then use the judge with a human-labeled slice on every run and a prediction-powered correction. MT-Bench's original validation is the template. Its GPT-4 pairwise judge agreed with expert humans on 85% of the votes that were not ties, while the experts agreed with each other on 81% (Zheng et al. 2023). That comparison is what justifies calling a judge "as good as a human".

The rest of the chapter builds on this. §22.5 meets judges that read an agent's whole trajectory, including judges that are themselves agents (Zhuge et al. 2024). §22.6 writes one judge per failure mode for your own product and validates each with its sensitivity and specificity. And §22.7 applies the agreement statistics to human raters.

🧭 What to remember

A judge is a grader with a calibration, a bias, and a noise level. Scores suit absolute gates, comparisons suit choosing between systems, and references and rubrics make judging reliable where they exist. A probability-weighted score separates answers that share the same most likely score, and it can be trusted once its probabilities are checked against humans and repaired by temperature or Platt scaling.
Model a judge's verdict as a Bradley-Terry log-odds difference plus a position term, a length slope, and a self-preference term, and fit them jointly. Swapping orders removes position only. Length can be estimated because it varies within systems. Self-preference can be measured only against humans on the same pairs.
Report kappa with an interval. Its large-sample variance follows from the delta method on the table of label pairs. At extreme prevalence kappa is both small and imprecise (92% agreement can be $\kappa = 0.16$ , with a 95% interval of about $\pm 0.24$ on 200 items). Report agreement on each class, or sensitivity and specificity, alongside it.
Use weighted kappa for ordered scales (it reduces to Cohen's kappa with 0 or 1 weights), Fleiss' kappa for a fixed number of ratings per item (Scott's pi when there are two), and Krippendorff's alpha for many raters with missing ratings (Fleiss' kappa up to a factor $(N-1)/N$ when none are missing).
Item noise averages out of system rankings, and bias that attaches to a system does not. A judge can rank models well and still be unfit to grade single responses, so validate the kind of agreement your use requires.
On the hardest problems, judges fail where they cannot solve the problem. They can be fooled by responses aimed at them, they favor students trained on their own outputs (preference leakage, invisible in their verdicts), and they drift with versions. The defenses are references, answer-checking programs, juries from families unrelated to the training data, pinned versions, and a human-labeled slice with a prediction-powered correction.
Next, §22.5 leaves single responses behind and evaluates agents that act over many steps, where the environment and the protocol become part of the measurement.

Zheng et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. The paper that made judges standard, with the position-consistency tables, the repeated-list test, reference-guided grading, and the judge-human agreement numbers.
Wang et al. (2023). Large Language Models are not Fair Evaluators. The order-swap experiment from the opening, and simple calibration fixes for position bias.
Zheng et al. (2024). Cheating Automatic LLM Benchmarks: Null Models Achieve High Win Rates. How a constant response tops three judge-based leaderboards, and why such benchmarks need defenses against cheating.
Li et al. (2024). From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline. A judge-based benchmark built to agree with human rankings, with separability and agreement metrics for benchmarks.
Arora et al. (2025). HealthBench: Evaluating Large Language Models Towards Improved Human Health. Rubric grading at scale, and a careful validation of the model grader against physicians.
Panickssery et al. (2024). LLM Evaluators Recognize and Favor Their Own Generations. The evidence that self-preference follows from self-recognition.
Tan et al. (2024). JudgeBench: A Benchmark for Evaluating LLM-based Judges. Judges on objectively labeled hard pairs, and the finding that judging tracks solving.
Thakur et al. (2024). Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judges. Thirteen judges against humans, why percent agreement hides disagreement, and item-level versus ranking agreement.
Li et al. (2025). Preference Leakage: A Contamination Problem in LLM-as-a-judge. Judges favor the students trained on their own synthetic data, by how much under which kinds of relatedness, and why the judges cannot see it.
Verga et al. (2024). Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Models. Panels of small judges from different families against one large judge.
Krippendorff (2011). Computing Krippendorff's Alpha-Reliability. The coincidence-matrix computation with worked examples, for any number of raters, missing data, and scale types.

Made for larger screens

LLM as a judge

What earlier sections established

Four ways to ask a judge

Scores from probabilities

Expected scores from a judge's probabilities

Checking that the probabilities mean what they say

Writing a rubric that people and judges apply the same way

A model of a biased judge

Measuring every bias at once

Ratings and position bias from a judge, fit jointly

Self-preference needs a second grader

A self-preference audit

Agreement between two graders

The kappa paradox

How precise is a kappa

Weighted kappa for ordered scales

Cohen's kappa with an interval, and weighted kappa

Many raters and missing ratings

Fleiss' kappa, and how it meets Scott and Krippendorff

Krippendorff's alpha with missing ratings

Item-level and system-level agreement

Does the judge rank systems like humans?

Where judges fail at the frontier

A checklist for validating a judge

Read more

Watch next