An arena ranks language models by people's votes. A visitor types a prompt, two anonymous models answer it side by side, and the visitor votes for the answer they like better or calls a tie. Millions of such votes become one rating per model, and those ratings are among the most watched numbers in the field. This section is about what the votes measure, how they become ratings, how precise the ratings are, and where they mislead.

Here is a concrete case. WebDev Arena is an arena in which the prompt describes a web app and each model builds it. In October 2026 it listed Claude Opus 5.5 (max) at 1820 points from 1,887 votes and GPT-6 Astra (max) at 1800 points from 2,281 votes (Epoch AI's copy of the leaderboard, retrieved October 2026). Is the first model better? The arena's own 95% intervals were 1803 to 1837 and 1784 to 1817, and they overlap. That overlap does not settle the question, because checking whether two intervals overlap is a much stricter test than it looks (§22.2). Later in this section you will run the right test by hand.

A second case shows that the process behind a rating matters as much as the arithmetic. Singh et al. (2025) found that Meta had tested 27 private variants of its models on Chatbot Arena, the original text arena, before the Llama 4 release. Providers could test variants privately and publish only the ones they liked. Google and OpenAI had each received about a fifth of all the arena's data (an estimated 19.2% and 20.4%), while 83 open-weight models, whose weights anyone can download, together had received 29.7%. If you can try 27 variants and keep the best, how much of the published rating is the model and how much is the selection? We will put a number on it.

§22.1 treated every evaluation as an estimate of a quantity you care about. Its error has several sources, among them whether the test measures the right thing, luck in which items were drawn, and mistakes by whoever grades. §22.4 studied the grader when it is a language model. Here the graders are people, and the ratings are a statistical model fitted to their votes. We will design a human evaluation, then derive the chess rating system Elo and show that it is a form of gradient descent. Then we fit ratings to all the votes at once, with honest intervals, and turn the intervals into ranks. We correct for answer length and formatting, work out how many votes a comparison needs, and end with what arenas get wrong.

What a human preference vote measures

People are the graders of last resort. Nobody can write a unit test for "a helpful reply to a worried parent" or "a landing page that looks good", and a model judge brings the biases measured in §22.4. So we ask people. But "people prefer A" means something only once we say which people, on which prompts, asked which question.

Here is the quantity a vote estimates, in plain words first. Draw a random prompt, a random rater, and one answer from each model. Record 1 if the rater prefers model A's answer, 0.5 for a tie, and 0 if they prefer B's. Average that record over endless repetitions. The average is the population's preference for A over B. The exact quantity an evaluation targets is called its estimand.

In plain words, $\theta_{AB}$ is the share of points $A$ would win against $B$ if the arena ran forever. It belongs to a triple, the prompts, the raters, and the interface. Change any one of them and you have a different number. This is the question of construct validity from §22.1, whether a test measures what you care about, asked of human raters. An arena estimates $\theta$ for many pairs of models and then summarizes them with one number per model, and that summary is where the statistics begin. An interviewer who asks "what does an arena rank measure?" wants to hear this triple named.

Rating one answer or comparing two

There are two basic formats. In an absolute (pointwise) evaluation a rater scores one response on its own. The score is either a point on a Likert scale, a numbered scale such as 1 for very poor to 7 for excellent, or a pass or fail on each yes-or-no criterion of a rubric. In a side-by-side (pairwise) evaluation the rater sees two responses to the same prompt and says which is better, or that they tie. Model judges face the same choice in §22.4, for the same reasons.

Labs mostly compare models side by side. Side-by-side comparison is also the format of the preference data used in reinforcement learning from human feedback (RLHF), where a reward model, a network trained to predict which answer people prefer, learns from such pairs (§16.2). Labs use rubrics where an absolute bar is needed. HealthBench, for example, scores medical answers against criteria written by physicians (§22.4).

Designing a human evaluation that means something

A human evaluation is an experiment, and the habits of good experiments apply. This checklist extends the short one in §14.7.

• Blind and randomized. Raters never see model names, and the left-right order is randomized for each item. People favor one position just as model judges do, and randomizing makes that effect cancel on average. The size of the side bias can then be measured as one extra term in the rating model below.
• The right raters. Match expertise to the claim. Crowd raters can judge tone and clarity. Checking a drug dose, a proof, or a legal argument needs people who can verify it.
• A written rubric with priorities. Say what to judge (correctness, completeness, instruction following, safety, style) and in what order, with worked examples of hard cases, so that "better" means the same thing to every rater.
• Allow ties and "both bad". A forced choice between two equally good or two equally broken answers is a coin flip that looks like data.
• Measure agreement first. Run a calibration round, in which all raters label the same items. Then compute their agreement with the statistics of §22.4, such as Cohen's kappa, the agreement rate corrected for agreement by chance. Rater noise also sets a ceiling on how often anything, a model judge included, can agree with one rater. That ceiling sits a little above the raters' agreement with each other, because the best possible grader matches the consensus that each rater strays from (§16.2).
• Plan the sample size. A win rate $\theta$ estimated from $n$ comparisons has standard error $\sqrt{\theta(1-\theta)/n}$. So 100 comparisons near a 50% win rate give a standard error of 5 points. Telling a 55% win rate from 50% with 80% power at the 5% level takes about 800 comparisons (§0.11). Later in this section we do the same calculation for ratings.
• Pre-register the comparison. Fix the prompts, the raters, and the main metric before looking at results. Otherwise the many small choices of an analysis, the garden of forking paths of §0.11, will find a difference for you.

People are graders with biases of their own

Human votes are the reference, not the truth. Three findings about them matter to anyone who reads a leaderboard.

Style is mistaken for substance. Gudibande et al. (2023) fine-tuned small open models to imitate ChatGPT's outputs. Crowd raters judged the imitations nearly as good as ChatGPT because the style was right, and they missed that the facts were not (§14.7). Hosking et al. (2023) found that overall preference scores under-weight factuality. Raters more often judge a confident answer to be free of factual errors, so assertiveness skews their judgments. The authors also gave preliminary evidence that training on human preferences makes outputs more assertive. That is a loop in which the model learns to please the grader rather than to be right, the reward hacking of §16.5.

Crowds are noisier than experts, but not by much on everyday prompts. Chiang et al. (2024) had experts relabel a sample of Chatbot Arena battles, checking facts against outside sources. Crowd votes agreed with the experts 72% to 83% of the time. The two experts agreed with each other 79.4% and 89.8% of the time on the two model pairs studied. The authors attribute most of the 5 to 10 point gap to crowd raters making mistakes or missing factual errors.

Raters differ in what they want. Pooling everyone's preferences into one hides real disagreement about values. §16.3 showed three groups of raters, each consistent on its own, whose pooled preferences go around in a circle, A over B, B over C, and C over A. This is called Condorcet's paradox, and no single ranking fits such votes. A leaderboard is an average over a population, and it is only as meaningful as that population is close to your users.

In practice this means a human evaluation is only as good as its design. An interviewer will ask you to design one, and the answer is the checklist above with the raters matched to the claim.

Elo, the rating that updates after every game

Votes arrive one at a time, and we want a number per model that predicts who wins. Start with the simplest model of winning. Each model has one number, its strength. The chance that $A$ beats $B$ depends only on the gap between their strengths, through the S-shaped logistic curve. Equal strengths give a coin flip, and a large gap gives near certainty. This is the Bradley-Terry model (Bradley and Terry 1952, anticipated by Zermelo in 1929). §16.3 derived it from noisy utilities and used it to train reward models. With strength $r_i$ for model $i$, measured in natural-log units,

Chess ratings use the same curve with a different unit. A gap of 400 rating points means odds of ten to one, so the stronger player wins $10/11 = 91\%$ of games. A gap of 100 points gives $1/(1 + 10^{-0.25}) = 64\%$. These units are called Elo points. A rating in Elo points is $R_i = c_{\text{Elo}}\,r_i + \text{const}$, with

so that $\sigma\big((R_i - R_j)/c_{\text{Elo}}\big) = 1/\big(1 + 10^{-(R_i - R_j)/400}\big)$. In plain words, one natural-log unit of strength is 173.7 Elo points, and 400 points is a gap of $\ln 10 = 2.30$ natural-log units. What the Elo system adds to the curve is a rule for changing the ratings after each game. That rule explains both its strengths and its troubles.

In plain words, you gain points when you do better than expected and lose them when you do worse. The points you gain are exactly the points your opponent loses, so the update is zero-sum and the total of all ratings never changes. Beating a much stronger player is a big surprise and earns many points. Beating a much weaker one earns almost none.

Elo is stochastic gradient descent

Arpad Elo designed his system for chess federations without any talk of likelihoods. Yet it is exactly an algorithm you already know. Stochastic gradient descent (SGD) improves a model by looking at one example at a time and taking a small step against the gradient of that example's loss (§0.6). Take the Bradley-Terry model as the model, one game as the example, and the log loss as the loss. Then one SGD step is one Elo update.

The cell checks the hand example and the theorem numerically. It compares a finite-difference gradient of the loss with $P_{AB} - S$.

Seeing Elo as SGD explains its behavior at once, because SGD with a fixed step size has three well-known habits.

• The result depends on the order of the games. Each step starts from wherever the last one ended, so the same games in a different order end at different ratings.
• It never settles. With a fixed learning rate, SGD does not converge to the minimum of the loss. It wanders around the minimum, with a spread set by the step size. To converge, the steps must shrink, slowly enough that their sum is infinite but fast enough that the sum of their squares is finite. These are the Robbins-Monro conditions, $\sum_t\eta_t = \infty$ and $\sum_t\eta_t^2 \lt \infty$. That is why chess federations lower $K_{\text{Elo}}$ as a player accumulates games.
• The K-factor is a learning rate, with the usual trade-off. A large $K_{\text{Elo}}$ follows recent games closely and is noisy. A small one is precise but slow to rate a newcomer and slow to notice change.

For chess the last habit is a feature, because players improve and decline. For language models it is a defect, since a released checkpoint does not change (a hosted API can, which we come back to at the end). Let's measure the first two habits. The cell plays the same 2,000 games among four models in 200 different orders and records the final ratings.

The games are identical in every run. Yet with $K_{\text{Elo}} = 32$ a model's final rating has a standard deviation of 46 to 50 points across orders, and with $K_{\text{Elo}} = 8$ about 21. The true gap between the two middle models is 30 points. Averaged over orders the ratings are sensible, and the mean gaps come close to the batch fit later in the section. But any single run in one order is a coin toss among neighbors. How large is this noise in general, and does it shrink with more games? A short calculation answers both questions.

In plain words, Elo with a fixed K-factor forgets. Its error is set by $K_{\text{Elo}}$, not by how much data you have. With the common default of 32 it is as precise as a fit to a couple of dozen games, even after a million. The next cell checks the formula with 400 independent runs between two equal models.

Simulation and formula agree within a couple of points at every $K_{\text{Elo}}$. LMSYS, the academic group that launched Chatbot Arena in 2023, saw exactly this variability. They first stabilized Elo by averaging it over 1,000 random orderings of the games. In December 2023 they switched to a batch fit of the Bradley-Terry model, for two reasons (LMSYS 2023). The full history of games is available, so no online algorithm is needed. And most models are static, so their strength should not drift. The widget shows both behaviors side by side. On the left is Elo game by game for one order of the games, and Play replays the same games in a new order. On the right, for each model, the dots are where 100 other orders end, the black bar is the batch Bradley-Terry fit with its 95% interval, and the diamond is the true rating.

An interviewer who asks about Elo wants the three-line derivation and its consequences. The K-factor is a learning rate, the order of games matters, and the noise never goes away.

import numpy as np

def elo(games, n_models, K=32, init=1500.0):
    return np.full(n_models, float(init))    # placeholder: nobody's rating ever moves

Bradley-Terry is a logistic regression

The batch alternative fits all the games at once. §16.3 did this for a table of win counts. For arenas it is more useful to write the same model as a logistic regression with one row per battle. Then intervals, ties, and extra explanatory variables all come with little extra work. Recall that logistic regression predicts a probability as the sigmoid of a weighted sum of features (§0.4).

Here is one battle as a row. Take three models, numbered 0, 1, and 2. Model 2 is shown on the left, model 0 on the right, and the left one wins. The row's features are $(-1, 0, +1)$, with $+1$ for the left model, $-1$ for the right model, and 0 for the model not in the battle. The label is 1. With strengths $\mathbf{r} = (r_0, r_1, r_2)$, the weighted sum is $r_2 - r_0$, the gap between the two players, and its sigmoid is the Bradley-Terry win probability.

In plain words, it is a logistic regression with one feature per model and no intercept. A model's feature is $+1$ when it sits on the left, $-1$ when it sits on the right, and 0 otherwise. Add an intercept and you are estimating a bonus for the left side. That bonus is the position bias $b_{\text{pos}}$, the extra log-odds of winning from the first slot, which §22.4 measured for model judges.

Why the constraint? Every design vector sums to zero, so $\mathbf{x}_t^\top(\mathbf{r} + a\mathbf{1}) = \mathbf{x}_t^\top\mathbf{r}$ for any constant $a$. The likelihood cannot tell $\mathbf{r}$ from $\mathbf{r} + a\mathbf{1}$, and fixing one rating removes that freedom. Two more conditions decide whether a finite best fit exists. First, the comparison graph, with models as nodes and an edge for every pair that has met, must be connected. Otherwise each connected piece floats freely. Second, no group of models may have won every game against all the others, or their ratings run off to infinity. This is the "never lost" case of §16.3, which Ford (1957) made exact. Counting ties as half wins helps here, because a tie gives each side a little of both.

The gradient and the matrix of second derivatives follow from the sigmoid's derivative (§0.4), applied battle by battle,

The matrix $\mathbf{I}(\mathbf{r})$ is the Fisher information, which measures how sharply the log-likelihood curves around its peak. It is positive semidefinite, so the log-likelihood is concave and has no false peaks. Newton's method climbs it by fitting a quadratic at the current point and jumping to that quadratic's peak, the step $\mathbf{r} \leftarrow \mathbf{r} + \mathbf{I}^{-1}\nabla\ell$ on the coordinates left free by the constraint. Started from zero, it typically converges in a handful of steps, as in the length-controlled fit of §16.6.

How precise is a rating

The Fisher information also gives the standard errors. In large samples the maximum-likelihood estimate is close to normal with covariance $\mathbf{I}^{-1}$ on the free coordinates. These are the model-based standard errors, correct when the model is true. The simplest case shows the scale of the numbers, and it is the one formula worth memorizing from this section.

Precision grows only as the square root of the votes, so halving the error bar takes four times as many votes. Two vote counts make that concrete.

A rating measured against many opponents behaves about the same, as long as the opponents are near its own strength and their ratings are well measured. Interviewers often ask for this formula, and "347 over root n Elo points" is the answer to give.

Intervals that survive a wrong model

The model-based standard errors are right if the Bradley-Terry model is true. It never exactly is. Preferences can be intransitive (A beats B, B beats C, and C beats A). They can depend on the prompt, and raters disagree. A tie is not really half a win. What does the fit estimate then, and how wide should its interval be?

The answer has two parts. The fit still converges to something, namely the ratings that best predict the votes on average. And its uncertainty can be measured from the data rather than from the model, by looking at how much each battle's contribution to the gradient actually scatters. The estimator that does this is called the sandwich, because the measured scatter sits between two copies of the inverse Fisher information.

In plain words, the outer pieces, $\mathbf{I}^{-1}$, say how fast the likelihood curves. The middle measures how much the per-battle gradients $\mathbf{g}_t$ actually scatter. If the data scatter more than the model predicts, the interval widens. If they scatter less, as with ties, it narrows. The usual alternative is a bootstrap over battles, which refits the model on many resamples of the battles drawn with replacement and takes the spread of the refits as the standard error (§0.11). Chiang et al. (2024) compared the two in simulations and found them often identical to the eye. The sandwich intervals were wider in small samples but more stable, and narrower in large samples, so the authors chose to deploy them.

The first line of the theorem hides the most important fact about arena ratings. When the model is wrong, the target $\mathbf{r}^\circ$ depends on which pairs were played and how often. The match schedule becomes part of the definition of a rating. That is why Chiang et al. reweight the battles. Let $A_t$ be the pair shown in battle $t$ and $P(A_t)$ the probability that the arena's sampler chose it. Weighting each battle by $1/P(A_t)$ is called inverse-probability weighting. It makes the target the fit to a uniform schedule over pairs, whatever the adaptive sampler did. The next two cells fit Bradley-Terry by Newton's method with three kinds of standard error, and then show the dependence on the schedule with a deliberately intransitive example.

The fit recovers the true ratings to within one and a half standard errors each. The model-based and sandwich standard errors agree at about 14 Elo points, since here the data really are Bradley-Terry. The bootstrap gives 12 to 14, close enough given that 100 resamples estimate a standard error only to within several percent. Compare that 14 with the spread of nearly 50 points that online Elo showed on the same games, from the order of the games alone.

Now the intransitive case. Three models beat each other in a cycle, each winning 70% of its games against the next, like rock, paper, scissors. The cell computes the target $\mathbf{r}^\circ$ exactly, by Newton's method on the expected log loss, under two schedules.

With every pair played equally often, the three models tie, the only answer one number per model can give to a cycle. Play A against B 80% of the time and A ends up 119 points above B, with nothing about any model changed. A real arena is never this extreme. But its preferences are not perfectly transitive either, models enter and leave at different times, and the sampler favors some pairs. A rating is a property of the models and the schedule together. An interviewer who asks why two arenas with the same models disagree is looking for this point.

Now we can settle the opening question with the right test. It asks whether the gap is large compared with the standard error of the gap, not whether two intervals overlap.

import numpy as np

def bt_fit(battles, n_models, ref=0, iters=30):
    raise NotImplementedError

Ties count as half a win

Arenas let raters press "tie" or "both are bad". The simplest treatment, the one LMSYS adopted in 2023, counts a tie as half a win for each side, $S_t = \tfrac12$, in the same likelihood. That is not a true likelihood, since $S_t$ is no longer a yes-or-no outcome. But the equations that define the fit, $\sum_t (S_t - P_t)\mathbf{x}_t = 0$, still make sense. They say that each model's points equal its expected points, and the sandwich gives valid intervals for the result. A fit of this kind, which keeps the estimating equations of a likelihood without the likelihood itself, is called a quasi-likelihood.

What do the half-win ratings mean? Suppose a pair ties with probability $\tau$. When the game is decisive, $A$ wins with probability $q$. Then $A$'s expected points are $\tfrac\tau2 + (1 - \tau)q$, and the half-win rating gap is the logit (log odds) of that. For small gaps, $\operatorname{logit}(\tfrac12 + \epsilon) \approx 4\epsilon$ and $\tfrac\tau2 + (1-\tau)q - \tfrac12 = (1 - \tau)(q - \tfrac12)$, so

In plain words, ties pull half-win ratings toward each other by the factor $1 - \tau$. A pool with 30% ties shows gaps about 30% smaller than the decisive games alone would. A tie between evenly matched models also says nothing about which is better, so for the decisive gap only the $(1 - \tau)n$ decisive votes count. The cell checks the shrinkage and the effect of ties on the two kinds of standard error. It uses 20,000 games between two models at a decisive gap of 0.5 natural-log units, with 30% ties.

The half-win gap is 0.343 against 0.496 from decisive games alone, and $0.7 \times 0.496 = 0.347$, as the approximation says. The sandwich standard error is smaller than the model-based one by almost exactly $\sqrt{1 - \tau}$. The reason is that a half point has no variance. A fit that skips the sandwich therefore overstates its own uncertainty when ties are common.

A model that gives ties their own probability

This part is optional on a first read. Instead of treating a tie as half a win, a three-outcome model can give ties their own probability. It keeps the scale of decisive games and adds one parameter for how often ties happen.

For example, with $\nu = 0.8$ two equal models tie $0.8/2.8 = 29\%$ of the time and each wins 36%. The other classical choice is a threshold model, in which a difference in quality below some threshold reads as a tie (Rao and Kupper 1967). Davidson's model has a property that explains why the half-win shortcut works as well as it does.

import numpy as np
from scipy.optimize import minimize

def davidson_probs(ri, rj, nu):
    return 0.5, 0.0, 0.5             # placeholder: a coin flip that never ties

def davidson_fit(W, Tm):
    raise NotImplementedError

Ranks come from intervals

A leaderboard shows ranks, and ranks are where readers look first. A rank depends on all the ratings at once, so its uncertainty is not one model's interval. Chiang et al. (2024) report a cautious rank built from intervals. Model $j$ is surely above model $i$ when $j$'s whole interval lies above $i$'s. A model's best possible rank is one plus the number of models surely above it. Its worst is the number of models minus the number surely below it.

A tiny example first. Three models have intervals A $[1790, 1830]$, B $[1780, 1810]$, and C $[1700, 1760]$. No interval lies wholly above A's or B's, so both have best rank 1. C lies wholly below both, so A and B have worst rank $3 - 1 = 2$. Both A and B lie wholly above C, so C has rank 3 to 3. A and B share first place.

The guarantee needs simultaneous intervals, a set of intervals that all hold at once with probability 95%. Separate 95% intervals do not, because on average one in twenty of them misses. Chiang et al. build the simultaneous set from the large-sample normal law of $\hat{\mathbf{r}}$. That law says the estimate lies, with probability 95%, inside an ellipsoid around the truth. The ellipsoid's size comes from the chi-squared distribution, the distribution of a sum of squared independent standard normals. Each model's interval is the ellipsoid's shadow on that model's axis.

In plain words, being honest about many models at once costs width, and the cost grows with the number of models. Taking $\mathbf{a} = \mathbf{e}_i - \mathbf{e}_j$ in the theorem gives simultaneous intervals for every pairwise gap too. Those are tighter than comparing two separate intervals, because they account for the correlation between the two ratings. The cell applies the rank rule to the top eight of WebDev Arena in October 2026, with the arena's own reported intervals.

Three models share rank 1. Claude Sonnet 5.5 (xhigh) sits 34 points below the leader and is still not surely below it. Its upper end, 1804.5, clears the leader's lower end, 1803.4, by one point. Claude Opus 5 (max), with 12,087 votes, has the tightest interval, about $\pm 7.5$, and the standard errors track $347/\sqrt{\text{votes}}$ closely throughout. That match also shows that these are separate intervals, 1.96 standard errors wide, not simultaneous ones. Simultaneous intervals for eight models would be wider by the factor $\sqrt{\chi^2_{0.95,7}}/1.96 = 1.91$, which would widen every rank range. When an interviewer asks whether the top model is "really" first, the answer is a rank range, not a yes or no.

import numpy as np

def rank_intervals(lo, hi):
    n = len(lo)
    return np.ones(n, dtype=int), np.ones(n, dtype=int)     # placeholder: everyone is first

Style control rates models at equal style

People prefer longer, better-formatted answers, sometimes because those answers are better and sometimes regardless. An arena rating mixes both effects. In August 2024 LMSYS added style control, which puts measures of style into the regression as extra variables so that each model is rated at equal style (LMSYS 2024). It is the same move as the length-controlled win rate of §16.6, which fits a judge's verdicts on the length difference and reads off the win rate at equal length, extended to a whole leaderboard.

Each style feature compares the two answers relative to their size. If the left answer has 500 tokens and the right 520, the length feature is $(500 - 520)/(500 + 520) = -0.02$, nearly equal. If they have 20 and 40 tokens, it is $(20 - 40)/60 = -0.33$, a real difference.

In plain words, each style feature gets its own coefficient, and the models are rated on whatever the style does not explain. In LMSYS's fit with both length and markdown controlled, the length coefficient was 0.249 and the three markdown coefficients 0.019 to 0.031, in the regression's units. Length dominated. The ranks moved. GPT-4o-mini fell from 6th to 11th overall, Grok-2-mini from 6th to 18th, and Claude 3 Opus rose from 16th to 10th. These ranks are on the arena's interval-based scale, where several models share a rank. The cell plants the situation in a simulation with six models. One of them is mid-pack in substance but writes about twice as much.

The verbose model tops the raw leaderboard at 221, above the genuinely best model at 177, while its substance is worth 52. Style control puts it back at 49 and recovers the length coefficient (0.78 against a planted 0.8). It also removes a smaller inflation of the other models' gaps, which came from better models in this simulation also writing a little more. Here the extra length was worthless by construction, so removing its effect is right. In real data the same regression cannot tell worthless length from a derivation that makes an answer better, as the pitfall explains.

import numpy as np

def bt_style_fit(battles, Z, n_models, ref=0, iters=30):
    raise NotImplementedError

How many votes a comparison needs

Arena maintainers decide where the next votes go, and model developers decide how long to wait before trusting a rating. Both are sample-size questions, and the head-to-head standard error answers them. §0.11 answered the same questions for accuracies. There, the sample size for a test came from asking that the true difference sit a set number of standard errors from zero. Here the standard error is $1/\sqrt{nv}$ in natural units, with $v = P_{AB}(1 - P_{AB})$. The power of a test is the chance that it detects a gap that is really there.

In plain words, precision costs the square of what it buys, ties waste votes, and comparing two models through third parties costs four times as many votes as letting them meet. That last fact is why an arena's sampler does well to put close competitors against each other. The cell computes the headline numbers. Then it prices private testing with the selection result of §22.1. Publishing the best of $k$ equally good tries overstates their common score, in expectation, by the expected maximum of $k$ standard normals times one try's standard error.

The last lines of the cell are the arithmetic of private testing, the question from the opening.

Singh et al. (2025) ran simulations of this kind. They showed that submitting many variants and keeping the best lets a family of weaker models outrank a stronger one that submits once.

The widget puts these formulas in your hands. Two models a set Elo gap apart collect votes, either all against a pool of well-measured opponents or all head to head. The upper plot shows how often the better model ranks first and how often a test at the 5% level finds the gap. Below it, five models spaced by the same gap form a leaderboard, and Collect votes runs a fresh arena so you can watch the intervals shrink. Head to head, the five play a round robin, each splitting its votes evenly over the other four.

An interviewer will often hand you two ratings and a vote count and ask whether the gap is real, or how many more votes it needs. The answer is these formulas, with the fourfold cost of indirect comparison and the $1/(1 - \tau)$ cost of ties.

import numpy as np
from scipy.stats import norm

def votes_for_halfwidth(h_elo, p=0.5, z=1.96):
    raise NotImplementedError

def votes_to_detect(gap_elo, alpha=0.05, power=0.8):
    raise NotImplementedError

Where arenas mislead

Everything so far treats the arena as a well-behaved statistical experiment. It is also a public scoreboard with commercial stakes. Goodhart's law says that a measure pursued as a target stops measuring well, and §22.1 showed how it acts on any shared test that everyone optimizes. It acts on arenas too. The critiques fall into three groups, what is measured, who controls the data, and who votes.

What is measured. The estimand is the preference of the arena's users on the arena's prompts through the arena's interface. Chiang et al. (2024) found the prompts diverse, with over 100 languages in the first 240,000 votes and 77% of them in English. But they are what people choose to type into a free comparison site, which is neither your product's traffic nor expert work. One number per model also averages over very different tasks. Category leaderboards (coding, hard prompts) help. Prompt-to-Leaderboard (Frick et al. 2025) goes further. It trains a language model that reads a prompt and outputs a Bradley-Terry rating for every model on that prompt, so a ranking can be specific to a task. And style is part of the estimand unless it is controlled, with a control that is only observational.

Who controls the data. Singh et al. (2025) documented the imbalances behind this section's opening. Undisclosed private testing let a few providers test many variants and retract the scores they did not like. That inflates published ratings by the best-of-27 arithmetic above. Proprietary models were sampled more often and retired less often, so 61.4% of all data went to proprietary providers. Of 243 public models, 205 had been silently deprecated (their sampling rate cut to nearly zero without notice), against 47 officially listed as deprecated. Removing models changes the pool of opponents that later ratings are measured against, which matters when preferences are not perfectly transitive (the schedule cell above).

Access to arena data is itself a training advantage. Arena-Hard is a benchmark drawn from the arena's prompts. In Singh et al.'s controlled experiment, raising the share of arena data in fine-tuning from 0% to 70% raised the win rate on Arena-Hard from 23.5% to 49.9%, a 112% relative gain, with limited benefit on other tasks. And hosted models are not static. Once LMSYS separated the two API versions of GPT-4, they found GPT-4-0314 at 1201 and GPT-4-0613 at 1152 (LMSYS 2023).

Who votes. Votes can be rigged. Min et al. (2025) showed that because every vote moves the joint fit, an attacker can raise a target model's rank by voting in battles that do not involve it. In replays on about 1.7 million historical votes, a few hundred rigged votes moved rankings. Huang et al. (2025) showed that an attacker can tell which model wrote a response with over 95% accuracy. In a simulated arena, about a thousand such votes could move a model's ranking. The arena already had bot protection, detection of malicious users, and rate limits. Working with the authors, it began adding reCAPTCHA and login. Chiang et al. had already built a test that flags IP addresses whose votes are improbable under the fitted model.

None of this makes arenas useless. They measure something real that benchmarks miss, the taste of a large population on open-ended prompts. The project has grown into a family of arenas (text, WebDev, vision, text-to-image, video, search, and agents), run today as Arena at arena.ai. But a rating needs its interval, its category, its style setting, and its sampling history before it means anything. Here is what careful labs and arena maintainers do.

• Report intervals and shared ranks, never a bare ordering, and use simultaneous intervals when claiming a rank.
• Fit a batch model with sandwich or bootstrap intervals and inverse-probability weights for the sampler, not online Elo.
• Publish the raw and the style-controlled leaderboards, and the categories.
• Disclose private testing, cap the variants per provider, publish all results of tested models, and retire models by a stated rule.
• Pin model versions, and rate a hosted model separately when its API changes.
• Defend against rigging with authentication, rate limits, and anomaly detection.
• For high-stakes claims, run your own blinded expert evaluation on your own prompts, with a rubric and an agreement check. Use automatic stand-ins such as Arena-Hard-Auto, in which a model judge compares answers on hard arena prompts (§22.4), only after checking that they rank systems the way the human evaluation they replace does.

The table compares the rating methods of this section.

The style-controlled fit in PyTorch

Here is the same style-controlled fit with automatic differentiation. The model-based standard errors come from the exact matrix of second derivatives, which autograd computes. This is the shape of a production fit, except that real pipelines also add the inverse-probability weights and the sandwich.

Read more

• Chiang et al. (2024). Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference. The arena's statistical method, Bradley-Terry with sandwich intervals, rank intervals, active sampling, anomalous-user detection, and the crowd-expert agreement study.
• LMSYS (2023). Chatbot Arena: New models & Elo system update. Why the arena left online Elo for a batch Bradley-Terry fit, and the GPT-4 version split.
• LMSYS (2024). Does style matter? Disentangling style and substance in Chatbot Arena. Style-controlled Bradley-Terry, the coefficients, the rank changes, and an honest statement of its limits.
• Singh et al. (2025). The Leaderboard Illusion. Private testing, unequal sampling and deprecation, and the gains from training on arena data, measured.
• Huang et al. (2025). Exploring and Mitigating Adversarial Manipulation of Voting-Based Leaderboards. Identifying models from their responses and moving rankings with about a thousand votes, and the defenses the arena added.
• Min et al. (2025). Improving Your Model Ranking on Chatbot Arena by Vote Rigging. Why a joint fit lets votes in unrelated battles move a target model.
• Hosking et al. (2023). Human Feedback is not Gold Standard. Assertiveness biases human judgments of factuality, and preference scores under-weight it.
• Davidson (1970). On Extending the Bradley-Terry Model to Accommodate Ties in Paired Comparison Experiments. The tie model of this section, with the original paired-comparison model in Bradley and Terry (1952), Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons.

Watch next

▶

▶

▶

▶