The RLHF pipeline of Chapter 16 has two learning stages. First a reward model is fit to human comparisons with the Bradley-Terry loss (§16.3). Then a policy is trained with PPO to maximize that reward minus a KL penalty (§16.4). The second stage is where nearly all of the cost and instability lives.

Section 17.1 showed that the second stage has an exact answer, $\pi^\star(y\mid x) \propto \pi_{\text{ref}}(y\mid x)\,e^{r(x,y)/\beta}$, which we cannot use because of its normalizer $Z(x)$. Rafailov, Sharma, Mitchell, Ermon, Manning, and Finn (2023) noticed that this is backwards. Instead of fitting a reward and then searching for its optimal policy, choose the form of the reward model so that its optimal policy is known in advance. If the reward model is written as $\beta\log\frac{\pi_\theta(y\mid x)}{\pi_{\text{ref}}(y\mid x)}$ for a language model $\pi_\theta$, then the optimal policy for that reward is $\pi_\theta$ itself. Fitting the reward model by maximum likelihood is training the policy, the RL stage disappears, and the normalizer cancels out of the loss along the way. The title of their paper says it in one line, your language model is secretly a reward model.

This section derives Direct Preference Optimization (DPO) step by step, proves that the change of variables loses nothing, computes the loss and its gradient by hand, and reads the gradient carefully, because what it weights and what it ignores explain most of DPO's behavior in practice, good and bad.

Preferences, one more time

A preference dataset $\mathcal{D}$ contains triples $(x, y_w, y_l)$, a prompt and two responses to it, where a labeler preferred $y_w$ (the "winner", also called chosen) over $y_l$ (the "loser", or rejected). The Bradley-Terry model of §16.3 assumes the labels come from a latent reward $r^\star$,

with $\sigma(z) = 1/(1+e^{-z})$ the logistic sigmoid. A reward model $r_\phi$ is fit by maximizing the likelihood of the observed labels, which is the binary cross-entropy of §0.4,

In plain words, the probability that $y_w$ wins is a logistic function of how much more reward it has, and we fit the rewards so that observed winners get higher scores.

Two facts about this model matter here. It only ever sees reward differences between two responses to the same prompt, so adding any function $f(x)$ of the prompt to $r$ changes no prediction. And it says nothing about how to generate a good response, which is why Chapter 16 needed a second stage.

Three steps to DPO

Step 1, solve for the reward. Section 17.1 showed that for any reward function $r$, the optimal policy $\pi_r$ of the KL-regularized problem satisfies $\pi_r(y\mid x) = \pi_{\text{ref}}(y\mid x)e^{r(x,y)/\beta}/Z(x)$. Take the log of both sides and solve for $r$,

This holds for every reward, so in particular for the true reward $r^\star$ and its optimal policy $\pi^\star$. In plain words, a reward is the scaled log-ratio between its optimal policy and the reference, plus a constant for each prompt.

Step 2, substitute into Bradley-Terry. The preference probability depends only on the difference $r^\star(x,y_w) - r^\star(x,y_l)$. Write both rewards with Step 1,

Both responses answer the same prompt $x$, so they share the same $Z(x)$, and it cancels,

In plain words, under Bradley-Terry, the probability that a human prefers one response to another can be written using only the optimal policy and the reference policy. The intractable sum over all responses has vanished, because preferences compare two responses to the same prompt.

Step 3, fit by maximum likelihood. We do not know $\pi^\star$, so replace it with a parameterized policy $\pi_\theta$ and maximize the likelihood of the observed preferences, exactly as for a reward model. The result is the DPO loss,

In plain words, train the policy so that, relative to the reference, it has raised the probability of the chosen response more than the probability of the rejected one, and score how well it has done with a logistic loss. Everything in the loss is computable. It needs the log-probability of two given responses under two models, one of them frozen, and nothing else. No sampling, no reward model, no value function, no $Z$.

It helps to give the quantity inside the sigmoid a name, because it appears in every formula from here on. The log-ratio difference of a pair is

measured in nats, and the DPO loss is $-\log\sigma(\beta h_\theta)$ averaged over pairs. When the context is clear we write $h$. (This $h$ is unrelated to the number of attention heads $h$ of Chapters 4 to 6. In this chapter it always means a pair's log-ratio difference.) Notice where $\beta$ sits. It multiplies $h$, so the sigmoid sees the difference of two scaled log-ratios, while $h$ itself stays in nats.

Why this is not a trick

It is worth stating plainly why this procedure is legitimate, because the first reaction of many people is that something must have been swept under the rug. Define a reward model by

The DPO loss is exactly the Bradley-Terry reward-model loss $\mathcal{L}_R$ evaluated at this $r_\theta$. So DPO is reward modeling with an unusual architecture. Now ask what the optimal KL-regularized policy is for this reward. Plug it into the closed form,

The normalizer is a sum of probabilities, which is 1. So whatever reward DPO fits, the policy that RLHF would extract from it is the network we are already holding. Reward modeling and policy optimization have become a single maximum-likelihood problem. The cell checks Steps 1 and 2 with numbers. It picks a random reward on six responses, computes $\pi^\star$, recovers the reward from the policy, and compares Bradley-Terry probabilities computed both ways.

Every response's reward differs from the policy-derived reward by the same constant, $\beta\log Z = 0.7628$, and the 36 preference probabilities agree to machine precision.

Does the change of variables lose anything?

DPO restricts reward models to the form $\beta\log(\pi/\pi_{\text{ref}})$. A natural worry is that some rewards cannot be written this way, so DPO could be fitting a smaller family than a reward model can. The answer is no, and the proof is short enough to give in full. It is Theorem 1 of the DPO paper.

Call two rewards equivalent if they differ by a function of the prompt alone, $r'(x,y) = r(x,y) + f(x)$. Two facts follow from what we already know.

Lemma 1. Equivalent rewards give identical preferences. Bradley-Terry depends on $r(x,y_w) - r(x,y_l)$, and $f(x)$ cancels in the difference.

Lemma 2. Equivalent rewards give the same optimal policy. Adding $f(x)$ multiplies every weight $\pi_{\text{ref}}e^{r/\beta}$ by $e^{f(x)/\beta}$, and the normalizer absorbs it (§17.1).

So the data can only ever identify a reward up to its equivalence class, and the policy only depends on the class. It is therefore enough to show that every class contains a reward of DPO's form.

Theorem. Assume $\pi_{\text{ref}}(y\mid x) > 0$ for all $x$ and $y$. Every equivalence class contains exactly one reward that can be written as $\beta\log\frac{\pi(y\mid x)}{\pi_{\text{ref}}(y\mid x)}$ for some policy $\pi$.

Existence. Given any $r$, subtract its own log-partition function, $r'(x,y) = r(x,y) - \beta\log Z_r(x)$. This is in the same class, since we subtracted a function of $x$, and by Step 1 it equals $\beta\log\frac{\pi_r(y\mid x)}{\pi_{\text{ref}}(y\mid x)}$, which has the required form with $\pi = \pi_r$.

Uniqueness. Suppose two members of a class have the form, $\beta\log\frac{\pi}{\pi_{\text{ref}}} + f(x) = \beta\log\frac{\pi'}{\pi_{\text{ref}}}$. Exponentiating gives $\pi(y\mid x)\,e^{f(x)/\beta} = \pi'(y\mid x)$. Summing over $y$, both sides being distributions, gives $e^{f(x)/\beta} = 1$, so $f = 0$ and the two rewards are the same.

In plain words, among all rewards that make the same predictions, DPO's form picks out the single one whose partition function is 1, which is the one that is "already normalized". Nothing representable by a reward model is lost, as long as the policy network itself is expressive enough. That last condition is not trivial. A reward model and a policy model of the same size have different strengths, and DPO bets that the policy's representation is a good one for judging responses.

The loss, by hand

Two worked examples fix the scale of the numbers.

Example 1, early in training. Take $\beta = 0.1$ and one pair. The policy assigns the chosen response a sequence log-probability of $-12.0$ (the reference gave $-12.5$) and the rejected response $-15.0$ (the reference gave $-14.2$). The policy has raised the chosen response by 0.5 nats and lowered the rejected one by 0.8, so

At initialization, when $\pi_\theta = \pi_{\text{ref}}$, every pair has $h = 0$ and loss $\log 2 = 0.6931$. This is the first number to check in any DPO run, and §17.3 lists the bugs that make it come out differently. After moving 1.3 nats the loss has fallen only to 0.630, because $\beta = 0.1$ shrinks the logit.

Example 2, a pair the model has learned. To bring this pair's loss down to 0.018 we need $\beta h = 4$, which means $h = 40$ nats. With $\beta = 0.1$, "learning" a preference means changing the relative log-probability of the two responses by tens of nats. Spread over two responses of a few hundred tokens each, that is only about a tenth of a nat per token, small enough to look harmless locally. But it is forty nats of relative probability, a factor of $e^{40}$, and it is the first hint that DPO with a small $\beta$ can wander far from the reference on the pairs it trains on.

import numpy as np

def dpo_loss(policy_chosen_logps, policy_rejected_logps, ref_chosen_logps, ref_rejected_logps, beta):
    u = beta * (policy_chosen_logps - policy_rejected_logps)       # forgot the reference
    return float(np.mean(-np.log(1 / (1 + np.exp(-u)))))

The implicit reward

The quantity inside the loss deserves a name. The implicit reward of a response is

so that $\beta h_\theta = \hat r_\theta(x, y_w) - \hat r_\theta(x, y_l)$, the implicit reward margin. In Example 1 the implicit rewards are $+0.05$ for the chosen response and $-0.08$ for the rejected one, with margin 0.13. Every DPO training log reports their averages over a batch as the chosen reward, the rejected reward, and the margin, along with the reward accuracy, the fraction of pairs with a positive margin.

Three properties are worth knowing before reading such a log.

• It starts at zero. At initialization every implicit reward is exactly 0. A positive implicit reward means the policy has made that response more likely than the reference did, and a negative one means less likely. Nothing more.
• It is only meaningful within a prompt. The data identify the reward only up to a per-prompt constant, and $\hat r_\theta$ is the particular member of the class whose partition function is 1. Saying that one prompt's response has a higher implicit reward than another prompt's response compares two numbers that the model was never trained to make comparable.
• It can serve as a reward model. Since $\hat r_\theta$ is a Bradley-Terry reward model, a DPO-trained policy can score responses, rank candidates for best-of-$n$, or label new preference pairs. Benchmarks of reward models, such as RewardBench (Lambert et al. 2024), evaluate DPO models this way, by comparing the policy's and the reference's log-probabilities.

import numpy as np

def implied_preferences(pi, pi_ref, beta):
    ratio = pi / pi_ref
    return ratio[:, None] / (ratio[:, None] + ratio[None, :])     # forgot beta

The gradient, and what it weights

Now the analysis that every interviewer asks for. Write $u = \beta h_\theta$ for one pair, so the per-pair loss is $\ell(u) = -\log\sigma(u)$. Two facts about the sigmoid do all the work, $\sigma'(u) = \sigma(u)(1 - \sigma(u))$ and $1 - \sigma(u) = \sigma(-u)$. So

The reference log-probabilities do not depend on $\theta$, so $\nabla_\theta u = \beta\big(\nabla_\theta\log\pi_\theta(y_w\mid x) - \nabla_\theta\log\pi_\theta(y_l\mid x)\big)$. By the chain rule,

where we used $\sigma(-u) = \sigma(\hat r_\theta(y_l) - \hat r_\theta(y_w))$. In plain words, a gradient descent step makes the chosen response more likely and the rejected one less likely, and weights each pair by the probability that the implicit reward model currently ranks the pair the wrong way. It is exactly the weight $\sigma(-\Delta)$ that §16.3 found in the gradient of an ordinary reward model, with the implicit margin in place of $\Delta$, as it must be, since DPO's loss is a reward-model loss. A pair the model gets badly wrong gets a weight near 1. A pair it already ranks correctly by a wide margin gets a weight near 0.

The weight is not a detail. Rafailov et al. report that a naive version of the method without it, which raises $\log\pi_\theta(y_w)$ and lowers $\log\pi_\theta(y_l)$ with equal force on every pair (essentially an unlikelihood objective), makes the language model degenerate (their Appendix Table 3). The reason is that the unweighted objective $\log\pi_\theta(y_w) - \log\pi_\theta(y_l)$ is unbounded. It can always be improved by pushing the rejected responses further toward probability zero, and in a network with shared parameters the cheapest way to do that is to damage the model broadly. The weight acts as a brake. Once a pair is ranked correctly by a margin of a few units of $\beta h$, it stops pulling, and the gradient budget goes to the pairs that are still wrong. This is the KL regularization at work, disguised, because $\beta h$ is large exactly when the policy has moved far from the reference on that pair.

The brake never fully engages, though. The weight $\sigma(-\beta h)$ is positive for every finite $h$, so DPO keeps pushing on every pair forever. How fast does a learned pair drift? For one pair and a tabular policy (one free logit per response), only the chosen and rejected logits move, by equal and opposite amounts, and a gradient step of size $\eta$ updates the log-ratio difference by $h \leftarrow h + 2\eta\beta\,\sigma(-\beta h)$. When $\beta h$ is large, $\sigma(-\beta h)\approx e^{-\beta h}$, so $e^{\beta h}$ grows by about $2\eta\beta^2$ per step, and after $t$ steps

In plain words, on a pair the model already ranks correctly, the margin keeps growing, but only logarithmically, like the weights of logistic regression on separable data. Without the weight it grows linearly. The cell runs both recurrences.

After 10,000 steps DPO's margin is 45.7 nats, matching the logarithmic prediction of 46.1, while the unweighted version has reached 500. (The approximation assumes $\beta h$ is already large, which is why it says nothing useful at step 100.) Both are pushing $\pi_\theta(y_l)$ toward zero, which on a single deterministic pair is what maximum likelihood asks for, and §17.4 returns to why that is a problem when preferences are noisy or rare.

What the gradient looks like inside a softmax

For a policy that is a single softmax over $K$ responses with logits $\mathbf{z} \in \R^K$ (think of one-token responses, or the tabular policy above), the score of a response is $\nabla_{\mathbf{z}}\log\pi(y) = \mathbf{e}_y - \boldsymbol\pi$ (§15.3). The difference of two scores is

The policy's own probabilities cancel. A DPO step on one pair changes exactly two logits, by equal and opposite amounts, and leaves every other logit alone (the other probabilities still change, through the normalization). Contrast this with a language model, where the parameters are shared. There $\nabla_\theta\log\pi_\theta(y_w)$ and $\nabla_\theta\log\pi_\theta(y_l)$ are large vectors that overlap heavily when the two responses are similar, so lowering one drags the other down with it. That overlap is the mechanism behind likelihood displacement, where both log-probabilities fall during training, which §17.5 studies in detail.

There is one more cancellation that matters in practice. A sequence log-probability is a sum over tokens, $\log\pi_\theta(y\mid x) = \sum_t\log\pi_\theta(y_t\mid x, y_{<t})$. If $y_w$ and $y_l$ begin with the same tokens, those tokens have identical contexts in both responses, so their terms in $\log\pi_\theta(y_w) - \log\pi_\theta(y_l)$ cancel exactly, in the loss and in the gradient. DPO learns only from the first point where the responses differ onward. Tokens after the divergence do not cancel even when they match, because their contexts differ. Pal et al. (2024) show that for pairs differing in only a few tokens, this structure can lower the probability of the chosen response's later tokens, one of the first documented forms of likelihood displacement.

import numpy as np

def dpo_loss_and_grads(pc, pr, rc, rr, beta):
    u = beta * ((pc - rc) - (pr - rr))
    loss = float(np.mean(np.logaddexp(0.0, -u)))
    w = 1 / (1 + np.exp(u))
    return loss, -w, w                      # missing beta and the 1/B of the mean

import numpy as np

def dpo_logit_grad(z, z_ref, pairs, beta):
    w, l = pairs[:, 0], pairs[:, 1]
    p = np.exp(z) / np.exp(z).sum()
    h = np.log(p[w]) - np.log(p[l])                      # forgot the reference
    g = np.zeros_like(z)
    g[w] -= beta / 2; g[l] += beta / 2                   # constant weight, and overwrites repeats

What beta does in the loss

The loss depends on $\beta$ and $h$ only through their product. Suppose we want a pair's loss to fall to some level, say a weight of 0.05, which needs $\beta h \approx 3$. With $\beta = 0.5$ that takes $h = 6$ nats of change relative to the reference. With $\beta = 0.1$ it takes 30. Smaller $\beta$ means the policy must move further from the reference to satisfy the same data, which is the same direction as in the closed form, where small $\beta$ meant a sharper tilt. Typical values are 0.01 to 0.5, with 0.1 the default in the DPO paper (which used 0.5 for summarization) and in many later recipes, including Zephyr and Llama 3.

There is a second, less obvious effect. $\beta$ also multiplies the gradient, so at initialization the step is $\beta/2$ times the difference of scores. With plain SGD a smaller $\beta$ therefore also means smaller steps. With Adam, which divides by a running estimate of the gradient's scale, a constant factor in the gradient mostly cancels, so $\beta$ acts mainly through where the weight $\sigma(-\beta h)$ saturates. This is one reason DPO learning rates are tuned together with $\beta$, never separately, and §17.3 gives typical values.

Finally, notice what $\beta$ in DPO does not do. In RLHF with PPO, $\beta$ penalizes the KL on the policy's own samples at every step. DPO never samples, so nothing directly measures or limits the KL during training. The KL constraint is only present through the parameterization, and it holds exactly only at the optimum of a loss with complete data. On responses that are not in the dataset, nothing in the loss holds the policy near the reference at all. That gap is the subject of §17.5.

From pairs to rankings

Sometimes labelers rank $K$ responses rather than compare two. The Plackett-Luce model generalizes Bradley-Terry to rankings. The top choice is drawn with probability proportional to $e^{r}$ among all $K$, the second among the remaining $K-1$, and so on. For a ranking $\tau$ (so $y_{\tau(1)}$ is the best),

Each factor is a softmax over a subset of responses to the same prompt, so substituting $r = \beta\log(\pi/\pi_{\text{ref}}) + \beta\log Z(x)$ multiplies the numerator and every denominator term by the same factor $Z(x)^\beta$, which cancels again. The DPO loss for rankings is the negative log of this product with the implicit rewards $\hat r_\theta$ in place of $r^\star$. With $K = 2$ it is exactly the pairwise loss. Exercise 17.2e implements it.

import numpy as np

def plackett_luce_dpo_loss(logratios, beta):
    s = beta * logratios
    return float(np.mean(-(s[:, 0] - np.log(np.exp(s).sum(1)))))   # only the top choice

DPO and RLHF, side by side

The row that says "the same" is true and important, and it is also the most misunderstood line in the table. Both methods target the same policy when the reward model is fit to the same data and the optimization succeeds over all responses. In practice the two differ in what happens off the data. PPO's policy is judged by a reward model on its own samples, so it can be pulled back from (or pushed into) responses the data never covered. DPO's policy is judged only on the responses in the dataset. Whether that makes DPO better (no reward model to exploit) or worse (nothing controls the unseen responses) is an empirical question, and §17.5 reviews the evidence.

In PyTorch

The DPO loss is a few lines in any framework. Below we write it the way the paper's reference code does, as a function of four vectors of sequence log-probabilities, then check the hand-derived gradient on a small policy with shared parameters, a linear softmax over response features, where the gradient of $\log\pi_\theta(y)$ is no longer a one-hot vector.

Read more

• Rafailov, Sharma, Mitchell, Ermon, Manning & Finn (2023). Direct Preference Optimization: Your Language Model is Secretly a Reward Model. The derivation, Theorem 1 with its lemmas, the gradient analysis, and the Plackett-Luce extension in Appendix A.3.
• Bradley & Terry (1952). Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons. The original paired-comparison model that both reward modeling and DPO rest on.
• Plackett (1975). The Analysis of Permutations. The ranking model that generalizes Bradley-Terry to $K$ responses.
• Pal, Karkhanis, Dooley, Roberts, Naidu & White (2024). Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive. Shows, with the token-level structure of the gradient, how DPO can lower the probability of the chosen response when pairs differ in few tokens.
• Lambert et al. (2024). RewardBench: Evaluating Reward Models for Language Modeling. Evaluates DPO-trained policies as reward models through their implicit reward.
• Soudry, Hoffer, Nacson, Gunasekar & Srebro (2018). The Implicit Bias of Gradient Descent on Separable Data. Why logistic losses on separable data grow their margins logarithmically, the same behavior as DPO on pairs it has learned.

Watch next

▶

▶

▶