A reward model can only be as good as the comparisons it learns from, and comparisons are made by people (or models) who are tired, rushed, opinionated, and fond of long answers. When Ivison et al. (2024) took apart the ingredients of preference learning, the choice of preference data mattered more than the choice of algorithm, the reward model, or the prompts used for RL. If you remember one thing from this section, remember that the data is the product.

This section covers what a preference record looks like, how rankings become pairs, which responses are worth comparing, what annotator guidelines say, how often annotators agree and the hard ceiling that agreement puts on any reward model, how to use AI judges instead of people without inheriting their biases, why the data has to come from the model you are training, and the confounds, above all length, that leak into every dataset.

What a preference record contains

The atomic unit is a prompt $x$ with two responses and a label saying which one was preferred. Following InstructGPT, we write the preferred (chosen) response $y_w$ and the other (rejected) one $y_l$, for "winner" and "loser", and a dataset of such triples $\mathcal{D} = \{(x, y_w, y_l)\}$. Real records carry more than that, and the metadata turns out to matter.

The prompt is a conversation in the chat format of §14.3, so multi-turn comparisons (pick the better next turn of an ongoing chat) fit the same schema, as in Anthropic's HH data where crowdworkers chose between two responses at every turn (Bai et al. 2022). The strength field records how sure the annotator was. The model fields record where each response came from, which you need to know whether the data is on-policy (later in this section). And the annotator field lets you measure agreement and remove a single bad annotator's work.

Pairs, rankings, and strengths

There are three common formats, and they convert into each other.

• Pairwise. Two responses, one choice. The simplest and the most common.
• Best of $K$. Ziegler et al. (2019) showed labelers four samples and asked for the best one. That is one winner and $K-1$ losers, so it yields $K-1$ pairs.
• Full ranking. InstructGPT showed $K$ responses, with $K$ between 4 and 9, and had labelers rank them all (Ouyang et al. 2022). A ranking of $K$ responses implies $\binom{K}{2} = K(K-1)/2$ pairwise comparisons, from 6 pairs at $K = 4$ to 36 at $K = 9$, for the price of reading $K$ responses once. Ranking is the cheapest way to get many comparisons per minute of annotator time.

Strengths come on top of any format. Llama 2 asked annotators to choose between two responses and then label the degree as "significantly better", "better", "slightly better", or "negligibly better / unsure" (Touvron et al. 2023). §16.3 uses that label as a margin in the loss.

That last column is a trap. The $\binom{K}{2}$ pairs from one ranking are not independent examples, since each response appears in $K-1$ of them. InstructGPT found that shuffling the pairs into the dataset as separate examples made the reward model overfit within one epoch, and fixed it by putting all pairs from one prompt into the same batch element. That needs only $K$ forward passes instead of one per pair, and it treats the ranking as one example (§16.3 implements the loss).

import numpy as np

def ranking_to_pairs(ranks):
    # placeholder: adjacent pairs only, so a ranking of K gives K - 1 pairs
    order = np.argsort(ranks, kind="stable")
    return np.array([(order[k], order[k + 1]) for k in range(len(order) - 1)], dtype=np.int64).reshape(-1, 2)

Which responses to compare

A comparison is only as informative as the contrast between its two responses. Two samples from the same model at the same temperature are often nearly the same answer, and the annotator's choice between them is close to a coin flip. A pair of one excellent answer and one broken one is easy to label and teaches little that anyone did not already know. The useful pairs sit in between, different enough that a careful person can tell them apart.

Labs build that contrast on purpose. Llama 2 sampled the two responses to each prompt from two different model variants and varied the sampling temperature, to keep the pairs diverse (Touvron et al. 2023). InstructGPT's rankings of $K$ samples put several levels of quality side by side in one judgment. Christiano et al. (2017) chose what to ask about actively, sending people the clips on which an ensemble of reward predictors disagreed most, so each label went where the current model was least sure. And the prompts matter as much as the responses. InstructGPT's came from what customers actually sent the API, so the comparisons covered the tasks the model would face, and a prompt set that leaves out a kind of request leaves the reward model guessing about it.

Guidelines decide what "better" means

Annotators do not discover what good means. They are told. The guidelines are where a lab's values enter the data, and they are as much a part of the model as the architecture.

InstructGPT's labelers judged outputs on being helpful (follow the user's intention), truthful (do not make things up), and harmless (do not cause harm). The three conflict, and the guidelines resolved the conflict differently at different times. "During training we prioritize helpfulness to the user", while "in our final evaluations we asked labelers to prioritize truthfulness and harmlessness" (Ouyang et al. 2022). The labelers were about 40 contractors hired through Upwork and ScaleAI, screened for their judgment on sensitive content. Anthropic split the problem in two. For helpfulness, crowdworkers picked the more helpful and honest response. For harmlessness, red-teamers tried to make the model say something harmful and picked the more harmful response, which becomes a rejected example (Bai et al. 2022). Bai et al. also found the tension directly. Models trained only for helpfulness were preferred on helpfulness to models trained on both, and the authors noted that their data rarely showed a model how to respond helpfully to a harmful request, which left avoidance as the easy lesson. Llama 2 kept the two goals in separate datasets too, and its annotators also sorted each safety pair into one of three bins (the chosen response is safe and the other is not, both are safe, both are unsafe, 18%, 47%, and 35% of the data), and dropped every pair in which the chosen response was unsafe and the rejected one safe, the one kind of label that would teach the reward model that harm wins (Touvron et al. 2023).

Good guidelines share a few habits. They give worked examples of hard cases, they say how to break ties between criteria, they let annotators mark "unsure" or "both bad" rather than forcing a coin flip, and they are revised when agreement data shows annotators reading them differently. Stiennon et al. (2020) worked closely with a small group of labelers, measured their agreement with the researchers continuously, and gave feedback, and the paper stresses how much that care mattered.

How often annotators agree, and what that caps

Two careful annotators shown the same pair often disagree. The published numbers are sobering.

With binary choices a coin flip agrees 50% of the time, so 73% agreement is far from perfect. What does that number imply for a reward model?

A flip-rate model. Suppose each pair has a "consensus" answer, and each annotator independently reports the opposite with probability $\epsilon$ (the flip rate). Two annotators agree when both are right or both are wrong,

Solving the quadratic $2\epsilon^2 - 2\epsilon + (1 - A) = 0$ for the root below one half,

In plain words, the agreement rate between annotators tells you how often a single annotator departs from the consensus.

Now the ceiling. Even a perfect reward model, one that always predicts the consensus, agrees with a single annotator's label only $1 - \epsilon$ of the time, because that is how often the annotator agrees with the consensus. So reward model accuracy measured against single labels cannot exceed $1-\epsilon$. For $A = 0.726$, $\epsilon \approx 0.164$ and the ceiling is about 84%. InstructGPT's reward model reached 72.4% on held-out comparisons from the same labeler groups, about the rate at which two labelers agree with each other. A reward model that agrees with a labeler as often as another labeler does is already doing about as well as one human, and the remaining gap to the ceiling is what a perfect "average labeler" would add.

The usual remedy is to collect several labels per pair and take the majority. If $m$ annotators (odd) flip independently with rate $\epsilon$, the majority is wrong when more than half of them flip,

which for $\epsilon = 0.2$ falls from 0.20 with one label to 0.104 with three and 0.058 with five. Each extra label costs as much as a new pair, so labs usually prefer more pairs with one label each, and use multiple labels on a subset to measure $\epsilon$.

A chance-corrected version of agreement, Cohen's $\kappa = (A - A_e)/(1 - A_e)$ from §14.7, subtracts the agreement $A_e$ that two annotators labeling independently, with their own label frequencies, would reach by chance. For balanced binary labels $A_e = 0.5$, so $A = 0.726$ is $\kappa \approx 0.45$, what the statistics literature calls "moderate" agreement.

The widget models difficulty properly. Each pair has a true quality gap, and an annotator sides with the better response with a probability that rises smoothly with the gap, the sigmoid of the gap divided by a noise scale $\tau$ (so a big gap is easy and a small gap is a coin flip). That choice rule is the Bradley-Terry model, which §16.3 derives. You control how spread out the gaps are, how consistent the annotators are, and how many labels each pair gets.

import numpy as np
from math import comb

def flip_rate(A):
    return 1 - A                     # placeholder: confuses disagreement with the flip rate

def accuracy_ceiling(A):
    return A                         # placeholder: the agreement itself

Real agreement data is messier than two annotators per pair. A lab labels most pairs once and a random subset several times, by whichever annotators happened to be available, so the labels form a sparse matrix with one row per pair and one column per annotator. Agreement is then estimated from every pair of labels that landed on the same item. An item with $n$ labels, $k$ of which prefer the first response, contributes $\binom{n}{2}$ label pairs, of which $\binom{k}{2} + \binom{n-k}{2}$ agree. Pooling those counts over items gives the probability that two random labels of one item agree, which is exactly the $A$ of the flip-rate model.

import numpy as np

def agreement_stats(labels):
    # placeholder: treats a missing label as a vote for the second response, compares two columns
    L = np.nan_to_num(np.asarray(labels, dtype=float), nan=0.0)
    agree = float(np.mean(L[:, 0] == L[:, -1]))
    return {"agreement": agree, "majority": (L.mean(1) > 0.5).astype(float), "flip_rate": 1 - agree}

AI feedback

Human comparisons cost money and time, so the obvious move is to ask a strong model to make them. RL from AI feedback (RLAIF) does exactly that. Lee et al. (2023) labeled preferences with PaLM 2 and trained the same pipeline. Humans preferred the RLAIF summaries to the SFT baseline 71% of the time and the RLHF summaries 73% of the time, and in a head-to-head between RLAIF and RLHF the split was 50%. On harmless dialogue the AI-feedback policy was rated harmless 88% of the time, against 76% for RLHF and 64% for SFT. The AI labeler agreed with human labels 78% of the time on summarization with their best prompt, about as often as humans agree with each other. Anthropic's Constitutional AI pushed the same idea further, generating harmlessness preferences from a written list of principles instead of human red-team labels (Bai et al. 2022b), the subject of §19.2.

AI feedback also made open preference datasets possible. UltraFeedback (Cui et al. 2023) took 63,967 prompts from six public datasets, sampled four completions for each from a pool of 17 models, and had GPT-4 score each completion from 1 to 5 on instruction following, truthfulness, honesty, and helpfulness. Pairs are formed from the scores, and it became a default dataset for open DPO and reward-model training.

A model judge has biases of its own, and they are systematic, which is worse than random. §14.7 went through them for evaluation, following Zheng et al. (2023). Position bias favors whichever response is shown first (or second), verbosity bias favors longer answers, and self-enhancement bias favors answers from the judge's own model family, while GPT-4's agreement with humans was over 80%, as high as humans with each other. For labeling preference data the biases matter more than for evaluation, because a biased label is not just a wrong score on a report. It is a wrong lesson that the reward model learns and RL then amplifies. Lee et al. measured the position effect in their labelers. The largest picked the same position regardless of order in 18% of cases and the smallest in 56%.

The fix from §14.7 carries over unchanged. Ask in both orders and average. If $P(a, b)$ is the judge's probability that the first-shown response $a$ is better, the debiased preference is

If the judge adds a fixed push $b_{\text{pos}}$ toward the first slot to a fair log-odds $\Delta$, so that $P(a, b) = \sigma(\Delta + b_{\text{pos}})$ with $\sigma$ the logistic sigmoid, the average is $\tfrac12\big(\sigma(\Delta + b_{\text{pos}}) + \sigma(\Delta - b_{\text{pos}})\big)$, which lies on the same side of one half as $\Delta$. The bias no longer decides any comparison. Two consequences are specific to preference data. The averaged probability is a soft label, a target between 0 and 1 rather than a hard choice, and training a reward model on soft labels is part of §19.2. And verbosity bias is the length confound at the end of this section, arriving through the judge instead of through people, so the length checks there apply to AI labels too.

On-policy and off-policy data

Where do the two responses in each pair come from? If they come from the model you are about to train, the data is on-policy. If they come from other models, say the 17 behind UltraFeedback, it is off-policy. The difference matters more than it first appears.

A comparison answers a narrow question, which of these two particular responses is better. The lesson a dataset of comparisons teaches is a broad one, such as "longer answers are better" or "more detail is better", and the broad lesson can be true in one region of response space and false in another. RL moves the policy, and it moves it in exactly the direction the data said was good. Here is that failure in two dimensions, with no reward model at all, only labels.

A response is a point $\mathbf{y} \in \R^2$, and people's preferences follow a hidden score $r^\star(\mathbf{y}) = 2y_1 - 0.6y_1^2 + 0.5y_2$, which rises in $y_1$ up to $y_1 = 5/3$ and then falls. An annotator shown $\mathbf{y}_a$ and $\mathbf{y}_b$ picks $\mathbf{y}_a$ with probability $\sigma\big(r^\star(\mathbf{y}_a) - r^\star(\mathbf{y}_b)\big)$, so a big gap is easy and a small one is a coin flip (the same rule as the widget above). We collect pairs from policies centered at different points along $y_1$ and ask the data one question. How often did the response with the larger $y_1$ win?

In the data collected around the starting policy, the response with more $y_1$ wins 79% of its comparisons. The lesson is clear and, for that region, correct. A policy trained to produce what this data rewards walks toward larger $y_1$. Collect fresh pairs at $y_1 = 2$ and the same question gets slightly worse than a coin flip, 41%, and at $y_1 = 3$ the response with less $y_1$ wins 76% of the time. The annotators have not changed their minds. The responses they are shown have moved, into a region the start-up data barely touched, since only 2.3% of its responses have $y_1 > 2$. Any reward model fit to the old pairs inherits the old lesson, and §16.3 fits one and watches its accuracy collapse as the policy moves. Fresh comparisons of the new policy's samples carry the new lesson.

This is why every serious RLHF effort iterates. Ziegler et al. (2019) found that online collection mattered for summarization, where it gave clearly better summaries than a reward model trained once. Bai et al. (2022) ran "iterated online" RLHF, roughly weekly over about five weeks, with each round's best policy generating the next round's comparisons. Llama 2 collected preference data in weekly batches from its latest models and retrained its reward models as it went (Touvron et al. 2023). And Christiano et al. (2017) had learned the same lesson on Atari. Off-policy data still has a place, because it is cheap and diverse, which is why Tulu 3 mixed on-policy pairs from its own models with off-policy ones (Lambert et al. 2024). The trade-offs between the two for direct preference optimization are the subject of §17.5.

What leaks into the labels

A comparison records everything that made one response win, not only the thing you meant to measure. Several confounds are well documented.

Length. Annotators, human and AI, tend to prefer longer answers, and longer answers often are more thorough, so the preference is half earned. Singhal et al. (2023) found that much of the reward gain in RLHF across three settings came from making responses longer, and that a reward based purely on length reproduced most of RLHF's improvement over SFT. They traced the bias mainly to reward models that pick up length from the preference data.

Style and format. Bullet lists, headers, bold keywords, confident tone, and polite openings look like effort and tend to win comparisons even when the content is the same.

Agreement with the user. Sharma et al. (2023) found that responses matching a user's stated views were more likely to be preferred in existing human preference data, and that both people and preference models sometimes preferred convincing sycophantic answers to correct ones. §16.5 follows this through to the trained policy.

The annotator pool. A few hundred contractors are not "humans". Their language, culture, and expertise shape what "better" means. InstructGPT's authors were explicit that their model was aligned to the preferences of their labelers and researchers, not to people in general.

Let's measure the length effect in a synthetic dataset where annotators like quality, and also like length a little for its own sake, and where better responses also tend to be somewhat longer, so the two are tangled together the way they are in real data.

In this dataset the chosen response is longer 70% of the time, so the rule "pick the longer one" already scores 70%, close to the 75% at which annotators pick the truly better response. From the labels alone you cannot say how much of that 70% is earned. When the chosen response is the longer one it is truly better 81% of the time, and when it is the shorter one only 62%. Length and quality point the same way in most pairs, which makes those labels easy, and in the pairs where they disagree label noise decides far more often. Balancing the data so that the chosen response is longer exactly half the time makes the length rule worthless by construction, and the truly better response still wins 72% of the balanced pairs, so most of the quality signal survives. The price is data, 40% of the pairs here. What a model does with the unbalanced version is the real test, and §16.3 trains a reward model on exactly these pairs to find out. §16.5 adds remedies on the reward model and policy side.

import numpy as np

def length_baseline(len_chosen, len_rejected):
    return float(np.mean(len_chosen > len_rejected))      # placeholder: ties count as losses

def length_balanced(len_chosen, len_rejected):
    return np.arange(len(len_chosen))                       # placeholder: keeps everything

Datasets you will meet

A record format for training is the last practical step. Reward model trainers usually concatenate each prompt with its chosen and its rejected response, stack the chosen sequences and the rejected sequences into one batch so a single forward pass scores both, and remember where each sequence ends, since the reward is read at the last real token (§16.3).

Read more

• Ouyang et al. (2022). Training language models to follow instructions with human feedback. Section 3 and the appendices describe labeler selection, guidelines, ranking interfaces, and agreement in detail.
• Stiennon et al. (2020). Learning to summarize from human feedback. The care taken with labelers, and agreement numbers for researchers and labelers.
• Lee et al. (2023). RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback. AI feedback against human feedback on three tasks, with position-bias measurements.
• Cui et al. (2023). UltraFeedback: Boosting Language Models with Scaled AI Feedback. How the most used open AI-feedback dataset was built.
• Bai et al. (2022b). Constitutional AI: Harmlessness from AI Feedback. Preferences generated from written principles, the template for AI feedback on harmlessness.
• Zheng et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. Position, verbosity, and self-enhancement bias in model judges.
• Singhal et al. (2023). A Long Way to Go: Investigating Length Correlations in RLHF. How much of RLHF's gain is length, and where the bias comes from.
• Ivison et al. (2024). Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedback. Evidence that preference data quality matters most.

Watch next

▶

▶