How a model learns what people prefer when nobody can write the reward down. Comparisons, the Bradley-Terry reward model, PPO with four models in memory, and the ways an optimizer games a learned reward, with every loss, gradient, and number checked.

Try to write the reward function for "a good answer". A user asks how to fix a memory leak in a Python service. The best reply is correct, which you could check if you knew the answer. It is also the right length for this user, admits what it does not know, does not lecture, uses code where code helps, and is polite without fawning. Each of those is a judgment, several of them conflict, and the weights change from one prompt to the next. Nobody has ever written that function, and nobody will.

Yet anyone can look at two replies and say which one is better. That asymmetry, between how hard it is to specify what we want and how easy it is to recognize it, is the whole reason this chapter exists. Reinforcement learning from human feedback (RLHF) turns a pile of "this one is better than that one" judgments into a reward function, then uses reinforcement learning to make a language model produce what that reward function likes.

This section explains why we would go to that trouble when supervised fine-tuning already works, why the judgments should be comparisons rather than scores, where the method came from, and the three-stage pipeline the rest of the chapter takes apart piece by piece.

What demonstrations cannot teach

Supervised fine-tuning (Chapter 14) trains a model on prompts $x$ paired with written demonstrations $y$, by maximizing $\log \pi_\theta(y \mid x)$, the log-probability that the model $\pi_\theta$ with parameters $\theta$ assigns to the demonstration. (We write the model as $\pi_\theta$, a policy, because from §16.4 on we will treat it as one, in the sense of §15.1.) SFT is cheap, stable, and the right first step. It also has four limits that no amount of extra demonstration data removes.

It can only copy. The best an imitator can do is match the distribution of its demonstrators, flaws included. If the people writing demonstrations produce answers of uneven quality, the model learns the average of that unevenness. A demonstration says "write like this". It never says "this part was the good part".

It never sees its own mistakes. Every training token in SFT comes from someone else. The model is never shown one of its own outputs and told it was bad, so it gets no signal about the errors it actually makes, which are not the errors a human makes. This is the exposure problem of teacher forcing (§3.5) in a new form.

It teaches the model to claim what the demonstrator knew. When a labeler writes a confident, correct answer to a question the model cannot answer, SFT trains the model to produce confident answers to such questions, whether or not the knowledge is there. John Schulman argued, in a 2023 talk at Berkeley, that this is one source of hallucination. Imitation copies the form of knowledgeable answers without the knowledge behind them. A training signal that judges the model's own answers can instead reward "I am not sure" when the model is unsure.

Writing is expensive, judging is cheap. An excellent demonstration of a 400-word legal summary can take an expert half an hour. Choosing the better of two summaries takes a few minutes, and it can be done by people who could not have written either one. For the hardest tasks we care about, the skills to judge an answer are more common than the skills to produce it.

The last point is the deepest one. It is the same asymmetry that makes checking a proof easier than finding it. If a judge's signal can be turned into training signal, a model can be pushed past its demonstrations, toward answers that judges prefer but nobody wrote down.

How much a noisy judge is worth

Let's make that claim quantitative with the simplest possible model. Suppose each answer a model can write has a true quality $q \sim \mathcal{N}(0, 1)$, so the average answer has quality 0. A judge does not see $q$. It sees a noisy score $s = q + e$ with independent noise $e \sim \mathcal{N}(0, \sigma_e^2)$. We sample $n$ answers and keep the one the judge scores highest. How good is the answer we keep?

The pair $(q, s)$ is jointly Gaussian, with $\operatorname{Var}(s) = 1 + \sigma_e^2$ and $\operatorname{Cov}(q, s) = 1$. For jointly Gaussian variables the conditional mean is linear, $\mathbb{E}[q \mid s] = \frac{\operatorname{Cov}(q,s)}{\operatorname{Var}(s)}\,s = \frac{s}{1+\sigma_e^2}$. Choosing by $s$ only tells us about $q$ through $s$, so the expected quality of the winner is $\mathbb{E}[\max_i s_i]/(1+\sigma_e^2)$. The winning score is the largest of $n$ Gaussians with standard deviation $\sqrt{1+\sigma_e^2}$, so $\mathbb{E}[\max_i s_i] = \sqrt{1+\sigma_e^2}\, m_n$, where $m_n$ is the expected maximum of $n$ standard normals. Putting it together,

In plain words, picking the best of $n$ by a noisy judge buys you $m_n$ standard deviations of quality, shrunk by a factor that depends only on how noisy the judge is.

Even a judge whose noise is as large as the whole spread of quality ($\sigma_e = 1$) picks, from 16 samples, an answer 1.25 standard deviations above the model's average, better than about 89% of what the model writes ($\Phi(1.25) \approx 0.89$). A sharper judge ($\sigma_e = 0.5$) gets 1.58. This is best-of-n sampling, and it already beats imitation without any training. RLHF goes one step further and trains the model so that it writes such answers on the first try, instead of paying for $n$ samples at every query. We will meet best-of-n again in §16.5, where the same arithmetic explains why optimizing a judge too hard backfires.

import numpy as np
from scipy import integrate, stats

def expected_max_normal(n):
    raise NotImplementedError

def pick_quality(n, sigma):
    raise NotImplementedError

Comparisons are easier than ratings

If judgments are the signal, what form should they take? The obvious design asks annotators to rate each answer, say from 1 to 7. The design every lab converged on asks them to compare answers instead. There are three reasons, one statistical and two human.

Comparisons cancel each annotator's private scale. Annotator Ana is generous and gives most answers a 6. Annotator Ben is harsh and gives most a 3. If Ana happens to rate the answers to one prompt and Ben the answers to another, averaging raw scores says the first prompt's answers are better, when it only means Ana rated them. Model annotator $a$'s rating of an answer with quality $q$ as $s = c_a q + b_a + \text{noise}$, with a personal leniency $b_a$ and a personal spread $c_a > 0$. The offset $b_a$ and the scale $c_a$ shift and stretch everything Ana says, but they never change her order. Whenever she says $s_1 > s_2$, the sign of $c_a(q_1 - q_2)$, and so the order of the true qualities (up to noise), is the same for her as for anyone. A comparison keeps exactly the information that survives every annotator's private monotone rescaling, and throws away exactly the part that does not.

People are more consistent at comparing. Ask someone whether a reply deserves a 5 or a 6 and the answer depends on what they read just before, how tired they are, and what they think a 6 means. Ask which of two replies is better and most of that context drops out. This is an old finding of psychophysics, formalized as Thurstone's law of comparative judgment (1927), which modeled each stimulus as producing a noisy internal value and each judgment as a comparison of two such values. Christiano et al. (2017) reported the same thing for RL, that they "found it much easier for humans to provide consistent comparisons than consistent absolute scores".

Comparisons calibrate themselves as the model improves. An absolute scale has a ceiling. Once most answers deserve a 7, ratings stop distinguishing anything. A comparison between two good answers is still informative, so the signal stays useful as the model gets better, which is exactly when we need it most.

The cost is information. A 7-point rating carries up to $\log_2 7 \approx 2.8$ bits, a binary comparison at most 1 bit, and a comparison only tells you about a difference. Let's measure the trade-off. In the simulation below, 60 annotators each rate 8 of 40 answers on a 1 to 7 scale, with personal offsets of varying spread. We estimate each answer's quality two ways, by averaging its raw ratings, and by turning each annotator's ratings into within-annotator comparisons (ties dropped) and counting the fraction it wins. Then we score each estimate by the fraction of answer pairs it puts in the right order.

Read the table honestly. When every annotator uses the scale the same way (offset spread 0), averaging ratings is slightly better, because ratings carry more bits and our win-fraction estimate is crude. As soon as annotators differ, the average of raw ratings degrades steadily, while the comparison-based estimate barely moves. Real annotator pools always differ, and the comparison-based estimate can be improved further by fitting a proper model to the comparisons instead of counting wins. That model is Bradley-Terry, and §16.3 derives it. The widget lets you vary the annotators and watch both estimates.

Ratings are not useless, and modern pipelines often collect both. Llama 2's annotators picked the better of two responses and also said by how much, on a four-level scale from "significantly better" to "negligibly better or unsure" (Touvron et al. 2023). UltraFeedback asked GPT-4 for 1 to 5 scores on four aspects and derived comparisons from them (Cui et al. 2023). And Wang et al. (2024) pointed out that there was no evidence either way on whether reward models trained on ratings or on comparisons are better, because no public dataset had both for the same responses. They built one and proposed a reward model that uses both. The comparison is the backbone, and extra detail rides along.

import numpy as np

def within_annotator_pairs(annotator, item, rating):
    # placeholder: compares every pair of ratings, even across annotators
    rows = []
    for i in range(len(item)):
        for j in range(i + 1, len(item)):
            if rating[i] != rating[j]:

Where RLHF came from

The idea that an agent can learn a reward from people is older than language models. TAMER had humans press buttons to reward or punish an agent's recent actions and learned a model of that signal (Knox and Stone 2009). Preference-based policy learning asked people to compare whole trajectories in small domains with hand-coded features (Akrour et al. 2011), and Wirth et al. (2017) surveyed a whole literature of preference-based RL. These methods worked on small problems. The step that made the idea scale was deep learning on both sides, a neural reward model and a neural policy.

Language came next, at OpenAI, in three papers that each added one piece.

Ziegler et al. (2019) fine-tuned the 774M GPT-2 with about 5,000 human comparisons for stylistic continuation (positive sentiment, vivid description) and 60,000 for summarization. Labelers picked the best of four samples. The reward was the learned score minus a penalty $\beta\log\big(\pi_\theta(y\mid x)/\pi_{\text{ref}}(y\mid x)\big)$, with a coefficient $\beta > 0$, for drifting from the pretrained model $\pi_{\text{ref}}$, the KL penalty that every later system kept (§16.4). Two findings were prophetic. The summarizers learned to be "smart copiers", lifting whole sentences from the source, which labelers rewarded because copied sentences are accurate. And a refactoring bug flipped the sign of the reward, so the model optimized for the worst text it could write while staying fluent, which the authors only noticed after training ended. Optimizers find whatever the reward actually says.

Stiennon et al. (2020) scaled this to Reddit TL;DR summarization with 1.3B and 6.7B models and 64,832 comparisons collected by carefully trained labelers. Their 1.3B RLHF model's summaries were preferred to the human reference summaries 61% of the time, and it outperformed a supervised model ten times its size. They also showed, for the first time clearly, that pushing the policy against the reward model eventually made summaries worse by human judgment while the reward model's score kept climbing. That is overoptimization, the subject of §16.5.

Ouyang et al. (2022), the InstructGPT paper, applied the recipe to general instruction following on prompts that customers sent to the OpenAI API. Labelers preferred outputs of the 1.3B InstructGPT model to those of the 175B GPT-3, a model more than 100 times larger. The 175B InstructGPT was preferred to the 175B GPT-3 85 ± 3% of the time. ChatGPT, released that November, used the same methods with differences in data collection, and RLHF became the default way to turn a pretrained model into an assistant.

In parallel, Anthropic trained helpful and harmless assistants with RLHF and released their comparison data (Bai et al. 2022), and found that online, iterated rounds of fresh comparisons of the latest model mattered, the same lesson Christiano et al. learned on Atari. DeepMind, Meta, and others followed. Llama 2 (Touvron et al. 2023) described the most detailed open recipe of that era, with over 1.4 million comparisons and five rounds of reward modeling and RL.

The three-stage pipeline

InstructGPT's recipe has three stages, and every later variant is a modification of it. Here they are with InstructGPT's numbers.

Stage 3 maximizes, over prompts $x$ from a dataset $\mathcal{D}$ and responses $y$ that the policy samples itself,

with the reference policy $\pi_{\text{ref}} = \pi_{\text{SFT}}$ frozen and a penalty strength $\beta > 0$ (InstructGPT used $\beta = 0.02$). In plain words, write responses the reward model likes, but stay close to the model you started from, because the reward model was trained on that model's kind of output and cannot be trusted far from it.

Three properties of this design are worth fixing in your mind now, because each gets its own section.

• The reward model is a learned proxy. It was fit to a few tens of thousands of human comparisons, so it is wrong in places, and RL is an optimizer that searches for those places (§16.5).
• The policy is trained on its own samples. That is what separates stage 3 from SFT. The model gets feedback on the errors it actually makes, which is the cure for the second and third limits of imitation above (§16.4).
• Stage 2 does not need the model to write well, only to be compared. The comparisons are of the model's own samples, so the data must be refreshed as the model changes, the on-policy problem of §16.2.

The last row is where the field went next. When an answer can be checked by a program, as for math with a known answer or code with tests, the reward model can be replaced by the checker. That is RL with verifiable rewards, the engine of reasoning models (Chapter 18). And when the RL stage is replaced by a classification loss on the comparisons themselves, you get direct preference optimization (Chapter 17), which starts from the very objective $J(\theta)$ above. Preferences remain the signal for everything a program cannot check, which is most of what people ask for.

import numpy as np

def win_rate(judgments):
    # placeholder: ignores ties and returns no error bar
    a = sum(j == "A" for j in judgments); b = sum(j == "B" for j in judgments)
    return a / max(a + b, 1), 0.0

The real way to turn comparisons into scores

Counting wins, as our simulation did, ignores whom each answer beat. Beating a strong answer should count for more than beating a weak one. The principled fix gives every answer a score $r_i$, models the probability that $i$ beats $j$ as $\sigma(r_i - r_j)$, and fits the scores by maximum likelihood. That is the Bradley-Terry model, the foundation of §16.3. Here it is fit with autograd to the same kind of within-annotator comparisons, and compared with counting.

The fit orders 87.4% of the pairs correctly against 86.0% for counting wins. The gain is small here because every answer met a random sample of opponents. It grows when the schedule is lopsided and some answers only ever meet strong ones, which is the normal situation when the responses being compared come from models of different quality. The tiny L2 term pins down the scores, which the likelihood alone leaves free to shift by any constant, a fact that will matter for reward models.

Read more

• Christiano et al. (2017). Deep reinforcement learning from human preferences. The paper that made learning rewards from comparisons work with deep networks, with the ensemble, the noise model, and the online-feedback lesson.
• Ziegler et al. (2019). Fine-Tuning Language Models from Human Preferences. The first RLHF on a language model, with the KL penalty, the adaptive KL controller, and the sign-flip bug story.
• Stiennon et al. (2020). Learning to summarize from human feedback. Careful labeling at scale, and the first clear picture of a learned reward being overoptimized.
• Ouyang et al. (2022). Training language models to follow instructions with human feedback. InstructGPT, the three-stage pipeline with every number, and a frank list of what still goes wrong.
• Bai et al. (2022). Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback. Anthropic's HH data, iterated online RLHF, and the tension between helpfulness and harmlessness.
• Lambert (2025). Reinforcement Learning from Human Feedback. The standard book on the subject, broad and current, and a good companion to this whole chapter.

Watch next

▶

▶

▶