After §16.2 you have a pile of triples $(x, y_w, y_l)$, a prompt with a preferred and a rejected response. What you need for RL is a function that takes any prompt and any response, including responses nobody has ever labeled, and returns a single number, the reward. The labels never mention numbers. They only say which of two responses won.

So two questions have to be answered. What number should a response get, given only who beat whom? And how should a difference of two numbers turn into the probability that an annotator prefers one response? The answer to both is a model of choice that is almost a century old, and it is the most important equation in this chapter.

From noisy utilities to Bradley-Terry

Imagine that when an annotator reads response $y$ to prompt $x$, they form an internal sense of its worth, a utility $U(y)$. Part of it is shared by everyone, a true reward $r^\star(x, y)$. Part of it is idiosyncratic, a random term $g_y$ that depends on the annotator's mood, what they read before, and what they happen to care about today. The annotator prefers $y_1$ over $y_2$ when $U(y_1) > U(y_2)$, so

where we abbreviate $r^\star_i = r^\star(x, y_i)$ and write $\succ$ for "is preferred to". This is a random utility model. Notice what it already says. The preference probability depends on the rewards only through their difference, $r^\star_1 - r^\star_2$, and its shape is the cumulative distribution of the noise difference $g_2 - g_1$.

Which noise? If the $g$ terms are independent standard Gumbel variables, with CDF $F(g) = \exp(-e^{-g})$, the difference of two of them has a logistic distribution, and

with $\sigma(z) = 1/(1 + e^{-z})$ the logistic sigmoid. That is the Bradley-Terry model (Bradley and Terry 1952, and earlier Zermelo 1929, who invented it to rank chess players). In plain words, each response has a strength, and the chance that one beats another is the sigmoid of the gap in strength.

If instead the noise is Gaussian, $g \sim \mathcal{N}(0, s^2)$, the difference is $\mathcal{N}(0, 2s^2)$ and $\mathbb{P}(y_1 \succ y_2) = \Phi\big(\Delta/(s\sqrt2)\big)$, with $\Phi$ the standard normal CDF. That is Thurstone's 1927 model, the probit version. The two curves are nearly indistinguishable once scaled, since $\sigma(z)$ and $\Phi(z/1.702)$ never differ by more than 0.0095. Everyone uses the logistic one because its log-likelihood is simpler.

What a reward is, and what it is not

Three consequences of the model shape everything that follows.

Only differences are identified. Add any constant $c(x)$ to the reward of every response to prompt $x$, and every preference probability for that prompt is unchanged, because $\sigma\big((r_1 + c) - (r_2 + c)\big) = \sigma(r_1 - r_2)$. No amount of comparison data can pin down the level of a reward, only gaps between responses to the same prompt. So a reward of 2.3 means nothing alone. This is why InstructGPT shifted its reward model so that labeler demonstrations score 0 on average before RL, and why Stiennon et al. did the same with reference summaries. It is also why rewards for different prompts are not directly comparable, a problem RL handles with a value baseline and whitening (§16.4, §16.6). And it is the cancellation that makes DPO possible, because an unknown per-prompt normalizer drops out of the Bradley-Terry probability (§17.2).

The scale is the noise. If the noise has scale $\tau$ instead of 1, the model becomes $\sigma\big((r^\star_1 - r^\star_2)/\tau\big)$, so a reward model learns $r^\star/\tau$, the true reward measured in units of annotator noise. Consistent annotators (small $\tau$) produce large learned reward gaps, noisy annotators small ones. The magnitude of the learned reward is therefore a statement about confidence, and it sets how hard RL will push (§16.4).

Elo is Bradley-Terry. Chess Elo ratings use $\mathbb{P}(\text{A beats B}) = 1/(1 + 10^{-(R_A - R_B)/400})$, which is $\sigma\big((R_A - R_B)\ln 10/400\big)$. So one Bradley-Terry reward unit is $400/\ln 10 \approx 174$ Elo points, and a 400-point gap is a reward gap of $\ln 10 \approx 2.30$, ten-to-one odds. Leaderboards that rank chat models by crowd votes, such as Chatbot Arena, fit exactly this model to pairwise battles (human evaluation and arenas).

The reward model loss and its gradient

Now fit it. A reward model $r_\phi(x, y)$ with parameters $\phi$ predicts $\mathbb{P}(y_w \succ y_l) = \sigma\big(r_\phi(x, y_w) - r_\phi(x, y_l)\big)$, and maximum likelihood on the dataset $\mathcal{D}$ minimizes the negative log-likelihood of the observed choices,

In plain words, make the chosen response score higher than the rejected one, with a penalty that grows the more confidently the model gets the order wrong. It is binary cross-entropy with the label "the first one won" and the logit equal to the reward gap.

Write $\Delta = r_\phi(x, y_w) - r_\phi(x, y_l)$ for the margin of one pair. The loss of that pair is $\ell(\Delta) = -\log\sigma(\Delta) = \log(1 + e^{-\Delta})$, the softplus of $-\Delta$. Its derivative uses $\sigma'(z) = \sigma(z)(1 - \sigma(z))$ and $1 - \sigma(z) = \sigma(-z)$,

By the chain rule through $\Delta = r_w - r_l$, writing $r_w = r_\phi(x, y_w)$ and $r_l = r_\phi(x, y_l)$,

In plain words, gradient descent pushes the chosen reward up and the rejected one down by the same amount, and that amount, $\sigma(-\Delta)$, is the model's predicted probability that the label is wrong. A pair the model already orders confidently ($\Delta \gg 0$) contributes almost nothing. A pair it orders backward contributes almost a full unit. This is logistic regression's "learn from your mistakes" weighting, and it is the same weighting DPO inherits (§17.2).

Four properties are worth knowing cold.

• Convex in the margin. $\ell''(\Delta) = \sigma(\Delta)\sigma(-\Delta) > 0$. If the reward is linear in fixed features, $r_\phi(x,y) = \mathbf{w}^\top\mathbf{f}(x,y)$, the loss is convex in $\mathbf{w}$, and fitting is logistic regression on feature differences $\mathbf{f}(x,y_w) - \mathbf{f}(x,y_l)$, with no intercept (any intercept cancels in the difference, the shift invariance again).
• It never reaches zero. $\ell(\Delta) \to 0$ only as $\Delta \to \infty$. If a set of pairs can be ordered perfectly, the loss keeps decreasing as rewards are scaled up, so the learned rewards grow without bound. Early stopping, one epoch, and weight decay are what keep the scale sane.
• Compute it stably. $\log(1 + e^{-\Delta})$ overflows for very negative $\Delta$ in float32 and loses precision in the naive form. Use np.logaddexp(0, -delta) or torch.nn.functional.logsigmoid(delta), and $\sigma(-\Delta)$ for the gradient.
• A worked number. For $\Delta = 0.5$ the predicted win probability is $\sigma(0.5) = 0.622$, the loss is $\log(1 + e^{-0.5}) = 0.474$, and the gradient on $r_w$ is $-\sigma(-0.5) = -0.378$. At $\Delta = 0$, before training, every pair has loss $\log 2 = 0.693$, which is the number your first logged reward model loss should show.

import numpy as np

def bt_prob(r_a, r_b, tau=1.0):
    raise NotImplementedError

def rm_loss_and_grad(r_w, r_l):
    raise NotImplementedError

One score per response, and what Bradley-Terry assumes

The simplest reward model has no features at all, just one free score $r_i$ per item. That is how Chatbot Arena turns crowd votes into a leaderboard of models, and it is a good place to see the likelihood at work. Collect the results in a win table, with $W_{ij}$ the number of times item $i$ beat item $j$ and $n_{ij} = W_{ij} + W_{ji}$ the number of games between them. The log-likelihood of the scores $\mathbf{r} = (r_1, \dots, r_K)$ is $\ell(\mathbf{r}) = \sum_{j \ne k} W_{jk}\log\sigma(r_j - r_k)$. Item $i$ appears as the winner in the terms $W_{ij}\log\sigma(r_i - r_j)$, whose derivative is $W_{ij}\big(1 - \sigma(r_i - r_j)\big)$, and as the loser in $W_{ji}\log\sigma(r_j - r_i)$, whose derivative is $-W_{ji}\,\sigma(r_i - r_j)$, by the same $d\ell/d\Delta$ as above. Adding them,

In plain words, at the maximum every item's observed wins equal its expected wins. The equation also says what the fit cannot do. An item that never lost has a positive gradient at every finite score, so its score runs off to infinity, the "never reaches zero" property again, and a small L2 penalty or prior is what keeps it finite (the PyTorch fit in §16.1 used one). Zermelo's 1929 algorithm solves the equations by a fixed-point iteration on strengths $p_i = e^{r_i}$, $p_i \leftarrow \big(\sum_j W_{ij}\big)\big/\sum_{j\ne i}\frac{n_{ij}}{p_i + p_j}$, which Hunter (2004) showed is a minorize-maximize algorithm, so every step increases the likelihood.

import numpy as np

def bt_scores(wins, iters=500):
    W = np.asarray(wins, dtype=float)                 # placeholder: the log-odds of each item's win fraction
    frac = W.sum(1) / (W + W.T).sum(1)
    return np.log(frac / (1 - frac))

def to_elo(r, base=1000.0):

The random utility derivation made assumptions, and each can fail on real preference data.

• One number per response implies transitivity. If $r_a \ge r_b$ and $r_b \ge r_c$ then $r_a - r_c$ is at least as large as either gap, so $\mathbb{P}(a \succ c) \ge \max\big(\mathbb{P}(a \succ b), \mathbb{P}(b \succ c)\big)$ whenever both are at least one half, a property called strong stochastic transitivity. No set of scores can represent a cycle.
• The odds of $a$ over $b$ do not depend on what else is on the table. This is Luce's choice axiom, the independence of irrelevant alternatives, and Plackett-Luce below needs it as well.
• All annotators share one $r^\star$. Real annotators differ in values, not only in noise, and the model sees one pooled population.

The last one is the most consequential, and a three-voter example shows why. Suppose a third of annotators rank $a \succ b \succ c$, a third $b \succ c \succ a$, and a third $c \succ a \succ b$ (Condorcet's paradox). Every pairwise preference is then two to one, and they go around in a circle.

The best Bradley-Terry fit gives all three responses the same reward and predicts a coin flip for every pair, when every pair is in fact decided two to one. More generally, Siththaranjan et al. (2023) proved that when preferences depend on context the data does not record, such as who the annotator is, standard preference learning aggregates them by the Borda count, a voting rule that can rank first an answer most annotators place second. Practitioners mostly accept these limits. The alternatives are a pairwise preference model that reads both responses at once and outputs $\mathbb{P}(y_1 \succ y_2)$ directly, which can represent cycles (PairRM, Jiang et al. 2023), objectives built on such a model that look for a policy no other policy beats more than half the time (Nash learning from human feedback, Munos et al. 2023), and separate reward models for goals that annotators weigh differently, such as Llama 2's helpfulness and safety models.

Margins and noisy labels

Two small changes to the loss show up in published recipes.

Margins for graded preferences. Llama 2 used the strength label to demand a bigger gap for clearer preferences, $\ell = -\log\sigma\big(\Delta - m\big)$, where the margin $m$ depends on the rating. Their "large" margins were 3, 2, 1, and 0 for "significantly better", "better", "slightly better", and "negligibly better / unsure", and a "small" set used 1, 2/3, 1/3, and 0 (Touvron et al. 2023). The margin helped most on the clearly separable pairs. Llama 3 later dropped it, reporting diminishing improvements once the data was scaled up (Dubey et al. 2024).

A floor for annotator mistakes. Christiano et al. (2017) assumed a 10% chance that a person answers at random. That replaces the model by $\mathbb{P}(y_w \succ y_l) = 0.9\,\sigma(\Delta) + 0.05$, which can never be more confident than 95%. The loss on a badly misordered pair is then at most $-\log 0.05 \approx 3.0$ instead of growing linearly in $-\Delta$, and its gradient vanishes there, so a few mislabeled pairs cannot drag the model around. A related fix for label noise, label smoothing, reappears as conservative DPO in §17.4. It trains toward a target of $1 - \varepsilon$ instead of 1 for an assumed flip rate $\varepsilon$, which gives the loss a finite optimal margin, $\log\frac{1-\varepsilon}{\varepsilon}$, but unlike the floor it still charges a badly misordered pair a loss that grows linearly in $-\Delta$.

A reward model from scratch

Let's fit one end to end. A "response" is a point $\mathbf{y} \in \R^2$, the true reward is $r^\star(\mathbf{y}) = \sin(1.5y_1) + 0.5y_2 - 0.3y_2^2$, and annotators label random pairs with Bradley-Terry noise. The reward model is linear in seven polynomial features $\mathbf{f}(\mathbf{y})$, $r_\phi(\mathbf{y}) = \mathbf{w}^\top\mathbf{f}(\mathbf{y})$, and we train it by gradient descent using exactly the gradient above, $\nabla_{\mathbf{w}}\ell = -\sigma(-\Delta)\big(\mathbf{f}(\mathbf{y}_w) - \mathbf{f}(\mathbf{y}_l)\big)$.

Three numbers tell the story. On held-out pairs the model matches about 71% of the labels, which sounds mediocre until you see that the true reward matches only 72%, because the labels are noisy. That is the ceiling of §16.2 in action. Against the true ordering the model is right more than 91% of the time, and its rewards correlate with $r^\star$ at 0.88 on fresh points. The loss starts at $\log 2$ and stops far above zero, as it must with noisy labels. The widget below runs the same experiment live.

Rankings, the K-wise loss, and Plackett-Luce

InstructGPT's labelers ranked $K$ responses at once. With responses ordered best first, $y_{(1)} \succ y_{(2)} \succ \dots \succ y_{(K)}$, the loss averages the Bradley-Terry loss over all $\binom{K}{2}$ implied pairs,

In plain words, every response should outscore every response ranked below it, and each prompt counts once no matter how many pairs it produces. The $1/\binom{K}{2}$ is what makes a prompt count once. Without it, a prompt with $K = 9$ would weigh 6 times more than one with $K = 4$. InstructGPT computed all of a prompt's pairs from a single forward pass over its $K$ responses, as one batch element, and found that this fixed the overfitting that came from scattering the correlated pairs across the dataset.

In code, the pairwise structure is a $K \times K$ matrix of reward differences, masked to the pairs where $i$ is ranked above $j$. With $K$ varying across prompts, you pad to the largest $K$ and keep a validity mask.

import numpy as np

def kwise_loss(rewards, ranks, valid):
    raise NotImplementedError

Averaging pairwise losses is not the only way to use a ranking. The Plackett-Luce model (Luce 1959, Plackett 1975) generalizes Bradley-Terry to whole orders by thinking of the ranking as a sequence of choices. The annotator first picks the best of all $K$ with softmax probabilities, then the best of the remaining $K - 1$, and so on,

In plain words, each response in turn has to beat everything ranked below it, in a softmax over the responses still left. For $K = 2$ the product has one factor, $e^{r_1}/(e^{r_1} + e^{r_2}) = \sigma(r_1 - r_2)$, and Plackett-Luce is Bradley-Terry. It is also what the Gumbel derivation gives when $K$ noisy utilities are sorted. Zhu, Jiao, and Jordan (2023) proved that both estimators (the exact Plackett-Luce likelihood and InstructGPT's average over pairs) converge to the true reward, and that the Plackett-Luce maximum likelihood estimate is asymptotically more efficient. The pairwise version is what most code uses anyway, because it is simpler and tolerates partial rankings with ties. DPO can be trained on rankings with the same Plackett-Luce likelihood, with its implicit reward in place of $r_\phi$ (§17.2).

import numpy as np

def plackett_luce_nll(rewards, order):
    raise NotImplementedError

Building a reward model from a language model

A reward model for a chat assistant has to read a whole conversation and understand it as well as the policy does, so it is built from a language model. The recipe, from InstructGPT on, is short.

1. Start from a pretrained or SFT checkpoint, usually the SFT model itself or a sibling of it.
2. Remove the unembedding $\mathbf{W}_U \in \R^{d\times V}$ that maps the final hidden state to $V$ vocabulary logits (§5.6), and put a scalar head in its place, a vector $\mathbf{w}_r \in \R^d$ and a bias $b_r$.
3. Feed the prompt and response concatenated, $(x, y)$, in the chat template, ending with the end-of-sequence token, and read the head at the last real token,

In plain words, the reward is a linear readout of the network's summary of the whole conversation, taken at the one position that, under causal attention, has seen all of it. Earlier positions have not read the end of the response, so their hidden states cannot judge it.

Why initialize from the SFT model? Because judging a response requires the same knowledge as writing it. Llama 2's authors initialized their reward models from the chat model checkpoints so that "the reward model knows what the chat model knows", which avoids a gap in which the policy could exploit things the judge does not understand (Touvron et al. 2023). InstructGPT used a 6B reward model even for the 175B policy, because it saved a great deal of compute and 175B reward model training could be unstable (Ouyang et al. 2022). The head is small and freshly initialized. Huang et al. (2024), reproducing OpenAI's summarization setup, draw its weights with standard deviation $1/\sqrt{d + 1}$, matching OpenAI's original code, so that a hidden state with entries of order 1 gives an initial reward of order 1.

Training is short. InstructGPT trained for a single epoch at learning rate $9\times10^{-6}$ with 64 prompts per batch, because multiple epochs overfit quickly. Huang et al. also used one epoch, at $3\times10^{-6}$. After training, shift the bias $b_r$ so that a reference set of responses (labeler demonstrations in InstructGPT, the dataset's reference summaries in Stiennon et al. and Huang et al.) averages zero reward. Shift invariance means this changes no preference prediction, but it gives RL a meaningful zero.

The last-token readout is the most common source of reward model bugs, because padded batches put the last real token in a different column for every row. With right padding it sits at index (number of real tokens) minus 1. With left padding, the convention for generation, it is always the final column.

import numpy as np

def last_token_reward(H, mask, w, b):
    return H[:, -1] @ w + b          # placeholder: always reads the last column

In PyTorch

Here is a complete small reward model, a two-layer causal Transformer with a scalar head read at the last token, trained on synthetic preferences. A "response" is 12 tokens, its true reward is the number of 7s minus the number of 3s, and labels follow Bradley-Terry. Chosen and rejected sequences go through the network in one stacked batch, the trick from §16.2.

One epoch over 4,096 pairs is enough for the model to learn to count, and its held-out accuracy lands near what the true reward itself achieves on the noisy labels.

Accuracy and calibration

The first number anyone reports for a reward model is accuracy, the fraction of held-out pairs on which it scores the chosen response higher, $\frac1N\sum_n \mathbb{1}\big[r_\phi(x_n, y_{w,n}) > r_\phi(x_n, y_{l,n})\big]$, with a tie counted as one half. Published numbers are modest. InstructGPT's 6B reward models reached 72.4 ± 0.4% on held-out comparisons from the labelers they were trained on, and 69.6 ± 0.9% on labelers held out entirely. Stiennon et al. (2020) found accuracy rose by about 1.1% per doubling of the data and about 1.8% per doubling of the model. Llama 2's helpfulness model averaged 63.2% on its own test set, from 80.7% on "significantly better" pairs down to 54.7% on near-ties. By §16.2's ceiling argument, most of these models are not far from what a single extra annotator would achieve.

Every one of these numbers is a proportion measured on a finite set of pairs, so it needs an error bar, and the intervals of §0.11 apply directly. At 72% on 2,000 held-out pairs the standard error is $\sqrt{0.72 \times 0.28/2000} \approx 1.0$ point, so the 95% interval is about $\pm 2$ points, and two reward models one point apart on such a set cannot be told apart by their separate intervals. Compare them on the same pairs with a paired test, McNemar's test of §0.11, which looks only at the pairs where the two models disagree and is far more sensitive.

Accuracy only checks the sign of the margin. Calibration checks its size, in the sense of §0.11. A reward model is calibrated if, among all pairs where it predicts $\sigma(\Delta) = 0.8$, the predicted winner really wins 80% of the time. Calibration matters for RL because the margin is what RL optimizes. An overconfident reward model reports large reward differences between responses that annotators barely distinguish, and the policy will spend its KL budget chasing them.

The only new step for a reward model is what "confidence" means. A pairwise prediction is a binary one, which of the two responses wins, so the confidence is the probability of the favored side, $c = \max\big(\sigma(\Delta), 1 - \sigma(\Delta)\big) \in [0.5, 1]$, and a prediction is correct when the favored side won. With that, the reliability table and the expected calibration error of §0.11 carry over unchanged. Bin the pairs by $c$, and with $n_b$ pairs in bin $b$, average confidence $\operatorname{conf}_b$, and accuracy $\operatorname{acc}_b$ out of $N$ pairs,

The cautions of §0.11 carry over too. ECE is biased upward on small evaluation sets and depends on the bins, so compare reward models on the same pairs with the same bins, and report the held-out Bradley-Terry loss next to it. That loss is the reward model's own training objective, and as a log loss it is a proper scoring rule, which rewards calibration and sharpness together and cannot be gamed the way ECE can.

Bai et al. (2022) found preference models trained on helpfulness data alone very well calibrated, and models trained on the helpful-and-harmless mixture slightly under-confident. Huang et al. (2024) found the accuracy of their summarization reward models rising with the margin, as it should, but still off the calibrated curve $\sigma(\Delta)$, which they attributed partly to a diverse validation set.

import numpy as np

def rm_accuracy(r_w, r_l):
    return float(np.mean(r_w >= r_l))          # placeholder: ties count as wins

def rm_ece(r_a, r_b, a_won, n_bins=5):
    raise NotImplementedError

Accuracy where the policy goes

Held-out accuracy and calibration are measured on pairs like the training pairs. RL does not stay there. §16.2 built a 2D world whose true reward $r^\star(\mathbf{y}) = 2y_1 - 0.6y_1^2 + 0.5y_2$ peaks at $y_1 = 5/3$, and found that comparisons collected around the starting policy say "more $y_1$ is better" while comparisons collected at $y_1 = 3$ say the opposite. Now fit a reward model to the start-up pairs and see what that does to it. A linear reward $r_\phi(\mathbf{y}) = \mathbf{w}^\top\mathbf{y}$ is logistic regression on response differences with no intercept, as the loss section showed, so scikit-learn can fit it (each pair is entered in both orders, so the classes balance).

Near where it was trained, the reward model orders 96% of pairs the way $r^\star$ does. It has learned "more $y_1$ is better", with weight 1.69 on $y_1$, which is true there. A policy following that reward walks to larger $y_1$, and by $y_1 = 2$ the same reward model is at chance, and at $y_1 = 3$ it orders 78% of pairs backward. Its held-out accuracy, the number a team would report, never moved, because the held-out pairs came from the start like the training pairs. Fresh comparisons of the new policy's outputs fix it at once. The refit puts a negative weight on $y_1$ and orders 78% of the new pairs correctly (the true reward is flatter there, so pairs are harder). This is the reason RLHF collects preferences in rounds, and the reason the next tool exists, a way to notice that the policy has left the data.

Ensembles and uncertainty

A single reward model gives one number with no error bar. Train several, from different random seeds, data orders, or pretrained checkpoints, and their disagreement is an error bar. Where the training data was dense they agree. Where it was sparse, which is exactly where a policy under RL goes, they diverge. Let's see that with five reward models fit to bootstrap resamples of the polynomial-feature data from the from-scratch fit, queried near the data and far from it.

Most training points lie within a radius of about 2. Inside it the five models agree closely and are close to the truth (up to the unidentifiable shift). At radius 4.5 both the disagreement and the error are tens of times larger. The disagreement is a useful alarm, and it grows where the error grows. But look at the sizes. Far out, the members disagree by about 1.3 while the ensemble is wrong by about 11. All five share the same cubic features and the same kind of data, so they extrapolate in the same wrong direction, and their spread understates the error nearly tenfold. Disagreement tells you that you have left the data. It does not tell you how wrong you are, a small version of the shared-blind-spot problem below.

That observation is the basis of several mitigations. Christiano et al. (2017) trained an ensemble of three and asked people about the clips where its members disagreed most. Coste et al. (2023) optimized conservative objectives over an ensemble of five, the ensemble's minimum or its mean minus a multiple of its variance across members, and found they nearly eliminated overoptimization in best-of-n, with gains of up to 70%. Eisenstein et al. (2023) added a sober caveat. Ensembles whose members differ in pretraining seed generalize better than those that differ only in fine-tuning seed, but even they do not eliminate reward hacking, because all the members share similar blind spots. And WARM averages the weights of several fine-tuned reward models instead of their outputs, which costs one model at inference, and its policies beat those trained on a single reward model in 79.4% of comparisons (Ramé et al. 2024). §16.5 puts these to work against overoptimization.

Length bias, RewardBench, and other kinds of reward model

Whatever confounds sit in the preference data, the reward model learns them, and the most consistent one is length. §16.2 built a synthetic dataset in which annotators like quality and also like length a little for its own sake, and better responses are somewhat longer. Here is a reward model trained on exactly those pairs. It sees length exactly but quality only through a noisy feature, as a real reward model sees a response's length perfectly and its correctness imperfectly.

The reward model puts nearly four times more weight on log length than on its quality feature, because length is the cleaner signal. The annotators' taste for length has weight 1 in how this data was made, and the model's 1.27 exceeds it, because length also stands in for some of the quality the model cannot see well. Balancing the data so that the chosen response is longer exactly half the time cuts the length weight from 1.27 to 0.33 and leaves the quality weight unchanged, at the price of 40% of the pairs, and it does not remove the bias completely. A policy optimized against the unbalanced model will find that the cheapest way to raise its reward is to write more, which is the length exploitation of §16.5.

A quick check is the correlation between reward and response length over the policy's samples, and the length-baseline accuracy on held-out pairs. One architectural fix is ODIN (Chen et al. 2024), which gives the reward model two heads on shared features, trains one to track length and the other to be decorrelated from it, and throws the length head away for RL.

To compare reward models, the community uses RewardBench (Lambert et al. 2024). It contains 2,985 prompt-chosen-rejected trios in four sections, Chat (358), Chat Hard (456), Safety (740), and Reasoning (1,431), each chosen so that the right answer is unambiguous, for example a subtle bug or a wrong fact in the rejected response, and in several subsets the chosen response is no longer than the rejected one, so that "pick the longer one" does not work. The metric is accuracy. Chat Hard is the discriminating section, where even the best models in the paper stayed near 80% while most scored far lower. DPO-trained policies can be scored too, through their implicit reward, the scaled log-ratio to the reference model (§17.2), which only works if you use the right reference model. Treat benchmark accuracy as necessary, not sufficient. Ivison et al. (2024) found that larger, better-scoring reward models improved the final policy by up to 5% on math but only marginally in other categories. What RL needs is a reward model that stays right on the policy's own, shifting distribution, which no fixed benchmark measures.

Read more

• Bradley and Terry (1952). Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons. The original paper of the model every reward model fits.
• Ouyang et al. (2022). Training language models to follow instructions with human feedback. Section 3.5 and Appendix C give the K-wise loss, the batching trick, and the reward model training details.
• Zhu, Jiao, and Jordan (2023). Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisons. Consistency and efficiency of the Bradley-Terry and Plackett-Luce estimators, and why pessimism helps.
• Touvron et al. (2023). Llama 2: Open Foundation and Fine-Tuned Chat Models. Separate helpfulness and safety reward models, margins, and accuracy by preference strength.
• Lambert et al. (2024). RewardBench: Evaluating Reward Models for Language Modeling. How reward models are compared, and where they fail.
• Siththaranjan, Laidlaw, and Hadfield-Menell (2023). Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHF. What Bradley-Terry does with annotators who disagree, and the Borda count result.
• Coste et al. (2023). Reward Model Ensembles Help Mitigate Overoptimization and Eisenstein et al. (2023), Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hacking. What ensembles fix and what they cannot.
• Huang et al. (2024). The N+ Implementation Details of RLHF with PPO: A Case Study on TL;DR Summarization. Reward model details that matter, from the head initialization to the padding token.

Watch next

▶

▶