The derivation in §17.2 is exact. DPO and RLHF with PPO aim at the same optimal policy, $\pi^\star \propto \pi_{\text{ref}}\,e^{r/\beta}$, and DPO gets there with one supervised loss instead of four models and a sampling loop. So why did Xu et al. (2024) find that a well-tuned PPO beat DPO on every benchmark they tried? Why do Llama 3 and Tülu 3, both of which chose DPO, go to the trouble of generating fresh preference data from their own latest models? And why, in the healthy toy run of §17.3, did the chosen responses end up less likely than under the reference?

The answer to all three is one sentence of the derivation that is easy to read past. DPO recovers $\pi^\star$ if the loss is minimized over all responses. A real dataset covers a vanishing fraction of them, and DPO never looks at anything else, in particular not at the responses its own policy would generate. This section is about what that costs, how to see it coming, and how online and iterative DPO buy most of it back by sampling from the model being trained. It ends with the evidence on DPO versus PPO, which is the question every post-training interview eventually asks.

Throughout, $\pi_\theta$ is the policy, $\pi_{\text{ref}}$ the frozen reference, $\beta > 0$ the KL strength, a dataset $\mathcal{D}$ holds triples $(x, y_w, y_l)$ of a prompt with chosen and rejected responses, $\hat r_\theta(x,y) = \beta\log\frac{\pi_\theta(y\mid x)}{\pi_{\text{ref}}(y\mid x)}$ is the implicit reward, $h_\theta$ is a pair's log-ratio difference (chosen log-ratio minus rejected log-ratio, in nats, so that the DPO loss is $-\log\sigma(\beta h_\theta)$), and $\mu$ denotes the distribution that produced the responses in the dataset, often called the behavior distribution.

What the offline loss can see

Look at the DPO loss as a function of the policy. It depends on $\pi_\theta$ only through the probabilities it assigns to the responses that appear in $\mathcal{D}$. Two policies that agree on those responses have the same loss, however differently they behave everywhere else. For an expressive model trained on a finite dataset, the set of minimizers is enormous, and which one gradient descent finds depends on how the network generalizes, not on the preferences.

Xu et al. (2024) make this concrete with a three-response example and prove that every policy PPO can reach with a reward model fit to the data also minimizes the DPO loss, but not the other way around. Here is a version of their argument with numbers. Take a uniform reference over three responses and a dataset that only ever compares $y_1$ with $y_2$, always preferring $y_1$. The DPO loss is $-\log\sigma\big(\beta\log\frac{\pi(y_1)}{\pi(y_2)}\big)$, which goes to zero as $\pi(y_2)\to0$ regardless of $\pi(y_1)$. The policy $(0.1, 0, 0.9)$ minimizes it exactly as well as $(1, 0, 0)$ does. Nothing in the data mentions $y_3$, so nothing in the loss stops the policy from putting 90% of its mass there. PPO, by contrast, optimizes a reward model's opinion of $y_3$ under a KL penalty to the reference, and its optimum gives $y_3$ exactly what that reward and the reference justify.

Whether the policy actually drifts to $y_3$ depends on the parameterization. In a tabular softmax, raising $y_1$ and lowering $y_2$ leaves $y_3$'s logit alone (§17.2). In a real network, the parameters are shared, and the update that separates $y_1$ from $y_2$ also moves every response that shares their features. A small model shows what can go wrong. Four responses are described by two features, helpfulness and length, and the policy is a softmax over a linear function of the features. The true reward is helpfulness only. But in the offline dataset, as in many real ones, the preferred response is also the longer one.

Offline DPO puts essentially all its mass on the unhelpful, very long response, a response that appears nowhere in the data, and the true reward falls from 1.5 to zero. The mechanism is the shared parameterization. The only direction the data push is $\phi(y_w) - \phi(y_l) = (1, 1)$, "more helpful and longer", and the response with the most of that direction is the long one, whose length feature of 2.5 outweighs the helpful one's combined 2. This is a cartoon of a real phenomenon. Park et al. (2024) found that DPO exploits the length bias of preference data, producing answers far longer than the reference, and tie it to exactly this kind of out-of-distribution extrapolation.

Notice that the loss cannot detect the problem. On the dataset, the policy ranks the pair perfectly and the loss goes to zero. Only samples from the policy itself would reveal that it now writes something nobody preferred.

Distribution shift

The example has no distribution shift, since it is already broken by coverage alone, but real offline data add a second problem. Preference datasets are usually built from responses written by other models, for example UltraFeedback's responses from a pool of open and closed models rated by GPT-4. Those responses come from a behavior distribution $\mu$ that differs from the reference $\pi_{\text{ref}}$ the loss measures against. On pairs that the reference finds unlikely, the log-ratios start in an unusual place, the implicit rewards of the data are not centered where the policy actually generates, and the policy is trained on text it would never write.

The DPO paper's own recommendation targets this. When the dataset was not sampled from the SFT model, first fine-tune the reference on the chosen responses, which pulls $\pi_{\text{ref}}$ toward $\mu$. Xu et al. measured the effect directly. On the SafeRLHF dataset, DPO's results improved substantially when the base model was first fine-tuned on data from the same distribution, and when preference labels were regenerated on the model's own outputs.

There is a quantitative consequence worth deriving, because it explains a curve every DPO practitioner has seen. Take the expected implicit reward of responses drawn from the reference itself,

In plain words, averaged over what the reference would say, the implicit reward can only go down as the policy moves away from the reference, and it is zero only if nothing has changed. Rafailov, Hejna, Park, and Finn (2024) point out the consequence. If the data are sampled from the reference, as DPO recommends, the average implicit reward of the data must become negative during training. And if the reference was fine-tuned on the chosen responses, it is the chosen responses whose implicit reward must fall on average. The falling chosen reward of §17.3 is not a bug. It is arithmetic.

Likelihood displacement

The falling chosen likelihood has a name, likelihood displacement, and it is sometimes harmless and sometimes not. Razin, Malladi, Bhaskar, Chen, Arora, and Hanin (2024) define it as the situation in which the margin $\log\pi_\theta(y_w\mid x) - \log\pi_\theta(y_l\mid x)$ grows while both log-probabilities fall, as reported by Pal et al. (2024), Rafailov et al. (2024), Tajwar et al. (2024), and others. Probability is conserved, so if the chosen and rejected responses both lose mass, something else gains it. The question is what.

Their analysis is clean enough to reproduce. Take a single prompt and single-token responses. The model produces a hidden vector $\mathbf{h}\in\R^d$ for the prompt, and the next-token distribution is $\operatorname{softmax}(\mathbf{W}\mathbf{h})$, with $\mathbf{w}_z \in \R^d$ the unembedding row of token $z$ (§6.2). Treat $\mathbf{h}$ as the trainable quantity and hold $\mathbf{W}$ fixed for now. The score of a token with respect to $\mathbf{h}$ is

the token's unembedding minus the probability-weighted average. For a DPO pair $(y^+, y^-)$, the averages cancel in the difference of scores, so the DPO gradient (§17.2) moves the hidden vector along the difference of the two unembeddings, $\dot{\mathbf{h}} = c\,(\mathbf{w}_{y^+} - \mathbf{w}_{y^-})$ with $c = \beta\sigma(-\beta h_\theta) > 0$. Keep the two $h$'s apart. The bold $\mathbf{h}$ is the hidden vector, and the italic $h_\theta$ is the scalar margin of the pair. Now ask how fast each token's log-probability changes. By the chain rule,

In plain words, every token's log-probability changes in proportion to how well its unembedding aligns with the direction $\mathbf{w}_{y^+} - \mathbf{w}_{y^-}$, relative to the average token. Two consequences follow.

• The preferred token can fall. For $z = y^+$ the rate is $c\big(\lVert\mathbf{w}_{y^+}\rVert^2 - \langle\mathbf{w}_{y^+}, \mathbf{w}_{y^-}\rangle - \langle\bar{\mathbf{w}}, \mathbf{w}_{y^+} - \mathbf{w}_{y^-}\rangle\big)$. When $y^+$ and $y^-$ are similar, the first two terms nearly cancel, and the third, which grows as aligned tokens gain probability, can make the whole rate negative.
• The winner may be a third token. The token whose log-probability rises fastest is the one most aligned with $\mathbf{w}_{y^+} - \mathbf{w}_{y^-}$. That direction can have a large component orthogonal to $\mathbf{w}_{y^+}$, contributed by $-\mathbf{w}_{y^-}$, so the winner can be unrelated, or opposite in meaning, to $y^+$.

Razin et al. observed exactly this in real models. Training to prefer No over Never often sharply increased the probability of Yes. Their analysis also lets $\mathbf{W}$ train, which adds the unembedding similarity $\langle\mathbf{w}_{y^+}, \mathbf{w}_{y^-}\rangle$ to the conditions for displacement (the PyTorch block below shows that in a small toy, trainable unembeddings can be enough to prevent it, which real models are not). The cell builds a two-dimensional version in which No and Never point in nearly the same direction, starting from a model that, after SFT, gives them equal probability.

For the first hundred steps everything looks right. No rises from 0.49 to 0.72 and Never falls. Then Yes, whose unembedding is the most aligned with $\mathbf{w}_{No} - \mathbf{w}_{Never}$, overtakes, and by step 400 the model answers Yes 89% of the time, while the DPO margin, the only thing the loss measures, has grown every step.

For real sequences, Razin et al. turn the analysis into a score that can be computed for every pair in a dataset. With $\mathbf{h}_{x, y_{<k}}$ the model's hidden state after the prompt and the first $k-1$ tokens of a response, the centered hidden embedding similarity is

the inner product of the summed hidden states of the two responses minus the squared norm of the preferred one's. It plays the role for whole sequences that $\langle\mathbf{w}_{y^+}, \mathbf{w}_{y^-}\rangle - \lVert\mathbf{w}_{y^+}\rVert^2$ plays for single tokens (minus the first two terms of the preferred token's rate above), measured on the model's hidden states, which is where two similar responses overlap in a real network. A higher score means the two responses look more alike to the model and the pair is more likely to displace likelihood. On UltraFeedback and AlpacaFarm, the score identified the samples that contributed most to displacement, where simpler similarity measures did not.

How bad can it get? In their most striking experiment, training Llama-3-8B-Instruct with DPO to refuse unsafe prompts lowered its refusal rate on SORRY-Bench from 74.4% to 33.4%, because probability mass moved from the preferred refusals to compliant, harmful responses. Filtering out the pairs with the highest CHES scores prevented this, more effectively than adding an SFT term to the loss. Displacement is benign when the mass moves to responses as good as the chosen one (as in the toy of §17.3, where it went to short, high-reward answers) and catastrophic when it moves to responses that are worse. The training log cannot tell the two apart, so the only reliable check is to sample.

Three things help, listed in rough order of how directly they attack the cause.

• Distinct pairs. Pairs whose responses differ in meaning, not just in a few tokens, displace less. Llama 3's safety team reports a related practice, crafting response pairs that are nearly orthogonal in an embedding space. CHES-based filtering does this systematically.
• On-policy pairs. Pairs sampled from the current model show it its own likely outputs, so displaced mass shows up in the next round's data and gets corrected (next part).
• A likelihood term on the chosen response. An NLL term (Llama 3's weight 0.2, Exercise 17.3c) or DPO-Positive's penalty (§17.4) keeps the chosen likelihood from falling, though Razin et al. found it a weaker fix than data filtering for their unalignment case.

import numpy as np

def logprob_velocity(W, h, h_ref, pos, neg, beta):
    hdot = W[pos] - W[neg]                       # right direction, but the scale and the centering are missing
    return W @ hdot

Overoptimization without a reward model

One hope for DPO was that, with no explicit reward model, there would be nothing to hack, and the overoptimization curves of §16.5 would not apply. They do. DPO still fits a reward model, its implicit one, on a finite dataset, and that reward is accurate only near the data. As training moves the policy further from the reference, its own samples leave the region where the implicit reward was fit, which is the same situation as a PPO policy outrunning its reward model.

Rafailov, Chittepu, Park, Sikchi, Hejna, Knox, Finn, and Niekum (2024) measured it. Training direct alignment algorithms across a range of KL budgets and model sizes, and judging the results with a stronger model, they found the same hump that Gao et al. found for RLHF. Quality rises with KL from the reference and then falls at higher budgets, and the decline often starts before a single epoch over the dataset is complete. In practice this is one more reason to choose $\beta$ and the number of steps by evaluating samples from checkpoints, not by the training loss, which keeps falling through the decline.

Online and iterative DPO

Every failure so far has the same root. The policy is never shown its own outputs. The fix is to show them. Instead of a fixed dataset, sample responses from the current policy, label them, and train DPO on those pairs.

Online DPO does this at every step. Guo et al. (2024) call their version online AI feedback (OAIF). At each training step they sample two responses per prompt from the current policy, ask an LLM annotator which is better, and take a DPO (or IPO, or SLiC) step on the result. In human evaluations, the online versions of the three methods beat their offline versions about 66% of the time on average, and on TL;DR summarization human raters preferred online DPO over the SFT baseline, RLHF, and RLAIF 58% of the time in a four-way comparison.

Iterative DPO does it in rounds, which is cheaper to engineer because generation and training alternate instead of interleaving. A typical round, following the recipe of Xiong et al. (2024) and Dong et al. (2024), looks like this.

1. Take a batch of prompts and sample $n$ responses to each from the current policy $\pi_t$.
2. Score every response with a reward model, an LLM judge, or a verifier.
3. Pair the best response with the worst, dropping prompts where they tie or nearly tie. (Dong et al. describe $n = 8$, half sampled at temperature 1.0 and half at 0.7, and generate 16 responses per prompt in their experiments.)
4. Train with DPO on the new pairs (often together with earlier rounds' data, as Dong et al. do), initialized from $\pi_t$, to get $\pi_{t+1}$.
5. Repeat for a few rounds (three in Dong et al.).

Which reference should round $t$ use? Dong et al. keep the original SFT model $\pi_0$ as the reference in every round while initializing from the latest policy, so every round optimizes toward the same target $\pi^\star \propto \pi_0 e^{r/\beta}$. The alternative is to reset the reference to $\pi_t$ each round, which makes each round a small trust-region step from the previous policy and lets the total distance from $\pi_0$ accumulate, a different and more aggressive objective. A middle ground updates the reference during training, as in TR-DPO (Gorbatovski et al. 2024). Llama 3 follows the iterative pattern at scale, with six rounds, each training DPO mostly on the newest preference data collected with the best models of the previous rounds, so that, in the report's words, the training data conform better to the distribution of the policy being optimized. Tülu 3 found that adding on-policy pairs, sampled from its own SFT model, improved DPO over off-policy data alone.

Back to the four-response example. Replace the fixed dataset by pairs sampled from the current policy and labeled by Bradley-Terry with the true reward.

Online DPO puts its mass on the two helpful responses, splitting it almost evenly, which is the optimal policy $\pi^\star$ for this reward and $\beta$. As soon as the policy started to favor the long, unhelpful response, it began to sample it, the labels compared it with helpful responses and rejected it, and the length feature was unlearned. Coverage now follows the policy wherever it goes.

Why does sampling help so much? Tajwar et al. (2024) compared supervised, contrastive, and RL fine-tuning methods across didactic bandits and full-scale LLM problems and identified two ingredients that matter. One is on-policy sampling, and the other is a negative gradient, pushing down the likelihood of specific bad responses, which plain maximum likelihood on good responses lacks. Methods with either ingredient, which they unify as mode-seeking, moved probability mass between regions of the output space much faster than maximum likelihood. Online DPO has both. In a separate study from the GPO paper of §17.4, Tang et al. (2024), working at Google DeepMind with matched budgets, found that the advantage of online over offline methods could not be explained by data coverage or data quality alone, persisted across contrastive and non-contrastive losses, and did not go away with larger policies. They observed an interesting split. Offline-trained policies became good at the discriminative task, ranking pairs, but worse at generating, while online-trained policies were better generators and worse rankers.

There is a deeper reason, too. Online DPO with a reward model is no longer reward-model-free. It is a policy improvement method that uses the reward model on the policy's own samples, in pairs, which makes it close kin to the RL methods of Chapter 16 and to the group-relative methods of §18.4, where several responses to the same prompt serve as each other's baseline. The line between "DPO" and "RL" blurs once you sample.

The loop can even close on itself. A model can act as its own judge, labeling its own samples with an LLM-as-a-judge prompt, and train on the resulting pairs with iterative DPO. Yuan et al. (2024) did this for three iterations starting from Llama 2 70B, and the final model outperformed several strong systems on the AlpacaEval 2.0 leaderboard, including Claude 2, Gemini Pro, and GPT-4 0613. Whether such loops keep improving or collapse onto their own biases is one of the central questions of self-rewarding models and synthetic data pipelines, which are built on exactly this iteration.

import numpy as np

def best_worst_pairs(rewards, min_gap):
    chosen = np.argmax(rewards, axis=1)
    return chosen, chosen, np.ones(len(rewards), bool)        # rejected is wrong and nothing is filtered

A toy that recovers the optimum

With full coverage the derivation's promise is exact, and it is worth seeing it kept. On a categorical policy over $K$ responses with labels drawn from Bradley-Terry on a reward $r$, the population DPO loss, the expectation over labels of each compared pair, is

where $h_{ij}$ is the log-ratio difference of responses $i$ and $j$. For each pair it is minimized when $\sigma(\beta h_{ij}) = p_{ij}$, which is $\beta h_{ij} = r_i - r_j$. If the compared pairs connect all $K$ responses, those differences pin down every log-ratio up to one shared constant, which normalization removes, so the minimizer is $\pi_\theta = \pi^\star = \pi_{\text{ref}}e^{r/\beta}/Z$. Exercise 17.5c asks you to train it and check.

import numpy as np

def fit_dpo_categorical(pi_ref, r, beta, pairs, steps=4000):
    return np.asarray(pi_ref, dtype=float)       # no training: the policy stays at the reference

DPO versus PPO, the evidence

Here is what controlled comparisons have found, in roughly chronological order. Read the settings carefully, because the conclusions depend on them.

Put together, the evidence supports a fairly clear picture. With stale, off-policy preference data, online RL tends to win, because it sees and corrects its own outputs. With fresh, on-policy pairs, regenerated every round, DPO comes close to PPO at a fraction of the cost and complexity, which is why the frontier recipes that use DPO all iterate. For tasks with checkable answers, the field has moved to online RL with verifiable rewards, where there is no preference data to go stale at all (§18.3). And everywhere, data quality matters more than the choice of algorithm.

In PyTorch

The cell above moved only the hidden vector. Here the same toy is trained with autograd and an optimizer, once with the unembeddings frozen, as in the analysis, and once with them trainable too, and with an NLL term of weight $\alpha$ on the chosen token added to the loss.

With the unembeddings frozen and no NLL term, the optimizer reproduces the displacement, and Yes ends at 0.89. An NLL weight of 0.01 already keeps No on top at 0.65, and 0.1 holds it at 0.96, which is why recipes like Llama 3's add such a term. When the unembeddings are trainable, the chosen token's own unembedding moves toward the hidden vector and raises its logit directly, and in this two-dimensional toy that is enough to prevent displacement altogether. Real models are less forgiving. Razin et al. trained OLMo-1B, Gemma-2B, and Llama-3-8B with all parameters free, on a single prompt with single-token responses, and the preferred token's probability fell by between 0.21 and 0.96 in every run. Toys show the mechanism, not its size, and the only reliable check in a real run is to sample from the model.

Read more

• Xu, Fu, Gao, Ye, Liu, Mei, Wang, Yu & Wu (2024). Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study. The theorem that PPO's solutions are a proper subset of DPO's, the synthetic out-of-distribution example, and the benchmark comparison.
• Razin, Malladi, Bhaskar, Chen, Arora & Hanin (2024). Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimization. The theory of likelihood displacement, the CHES score, and the refusal experiment.
• Ivison, Wang, Liu, Wu, Pyatkin, Lambert, Smith, Choi & Hajishirzi (2024). Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedback. A controlled comparison showing data quality matters most, and what PPO buys over DPO.
• Guo et al. (2024). Direct Language Model Alignment from Online AI Feedback. Online DPO, IPO, and SLiC with an LLM annotator, and their human evaluations against offline versions.
• Dong, Xiong, Pang, Wang, Zhao, Zhou, Jiang, Sahoo, Xiong & Zhang (2024). RLHF Workflow: From Reward Modeling to Online RLHF. A complete, reproducible recipe for iterative DPO, with the theory of why online exploration helps.
• Tajwar et al. (2024). Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Data. On-policy sampling and negative gradients as the two ingredients that matter.
• Tang et al. (2024). Understanding the performance gap between online and offline alignment algorithms. Careful ablations of why online methods win.
• Rafailov, Chittepu, Park, Sikchi, Hejna, Knox, Finn & Niekum (2024). Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithms. Overoptimization curves for DPO and its relatives, with degradation often inside the first epoch.
• Yuan, Pang, Cho, Li, Sukhbaatar, Xu & Weston (2024). Self-Rewarding Language Models. Iterative DPO with the model as its own judge.

Watch next

▶

▶

▶