DPO was published in May 2023. Within a year there were dozens of variants, usually named with a letter or two in front of "PO", and a reader of the literature could be forgiven for thinking each one is a new idea. Most are not. Line them up and nearly every one changes one of four things about the DPO loss.

1. The function of the margin. DPO applies the logistic loss $-\log\sigma(\cdot)$ to $\beta h$. Others use a squared loss (IPO), a hinge (SLiC), or a smoothed logistic (conservative DPO). Tang et al. (2024) make this precise, writing a whole family as $\mathbb{E}[f(\beta h)]$ for a convex $f$.
2. What goes inside the margin. The log-ratio to a reference (DPO), another function of that ratio that comes from a different divergence penalty ($f$-DPO, §17.1), the average log-probability with no reference at all (SimPO), an odds ratio (ORPO), with or without a target margin or a length term.
3. The shape of the data. Pairs (DPO), full rankings (Plackett-Luce, §17.2), or single responses labeled good or bad (KTO).
4. Extra terms. Most often a supervised likelihood term on the chosen response (ORPO, DPO-Positive, Llama 3's recipe).

This section covers the variants an interviewer expects you to know, IPO, conservative DPO, KTO, ORPO, and SimPO, and for each one the problem it was invented to fix. Throughout, $\pi_\theta$ is the policy, $\pi_{\text{ref}}$ the frozen reference, $\beta > 0$ the regularization strength, $(x, y_w, y_l)$ a prompt with chosen and rejected responses, and

the log-ratio difference of §17.2 in nats (not the head count $h$ of Chapter 4), so that DPO's loss is $-\log\sigma(\beta h_\theta)$.

Why DPO overfits deterministic preferences

Start with the problem that motivated one of the earliest and most influential variants. Take a prompt with two possible responses, a uniform reference, and a dataset in which $y_1$ is always preferred to $y_2$. Bradley-Terry can only explain a preference probability of 1 with an infinite reward gap, $r(y_1) - r(y_2) \to \infty$, and plugging an infinite gap into the closed form of §17.1 gives $\pi^\star(y_2) = 0$ whatever $\beta$ is. The same is visible directly in the DPO loss. $-\log\sigma(\beta h)$ is strictly decreasing in $h$, so its only infimum is at $h \to \infty$, and gradient descent keeps pushing $\pi_\theta(y_2)$ toward zero forever, slowly, at the logarithmic rate we derived in §17.2. The regularization strength changes how fast it gets there, not where it ends up.

That might sound like an edge case. It is the normal case. Azar et al. (2023) point out that with only a few labels per pair, the empirical preference is easily 0 or 1 even when the true one is, say, 0.8. Real datasets are more extreme still, because each pair is usually labeled once, so its empirical preference is exactly 0 or 1. On real data, almost every preference DPO sees is deterministic, and the KL regularization that was supposed to keep the policy near the reference becomes weaker the more deterministic the preferences are. They also give a persuasive account of why RLHF with an explicit reward model suffers less from this. Reward models are trained with regularization and early stopping, so they underfit, and their finite rewards keep the PPO policy at a finite distance from the reference. DPO fits the reward implicitly and loses that accidental protection.

IPO, regressing to a target margin

Azar et al. generalize the RLHF objective to $\Psi$PO, which maximizes

where $\mu$ is the distribution the comparison responses come from and $\Psi$ is a non-decreasing function of the preference probability. With $\Psi(q) = \log\frac{q}{1-q}$ and Bradley-Terry preferences, $\Psi(p^\star(y\succ y')) = r(y) - r(y')$ and the objective is the usual RLHF one, which is what DPO optimizes. That $\Psi$ is unbounded, and its blow-up at $q = 1$ is the problem above. Identity preference optimization (IPO) takes the simplest bounded choice, $\Psi(q) = q$, maximizing the probability of beating $\mu$ directly. The closed form of §17.1 applies with the reward replaced by $p^\star(y\succ\mu) = \mathbb{E}_{y'\sim\mu}[p^\star(y\succ y')]$, a number between 0 and 1.

The result is strikingly simple,

In plain words, IPO regresses each pair's log-ratio difference onto a fixed target of $1/(2\beta)$ nats, instead of pushing it toward infinity. (Azar et al. call the regularization strength $\tau$. We keep $\beta$ for consistency, as TRL does.) Its gradient is $2(h_\theta - \tfrac{1}{2\beta})(\nabla_\theta\log\pi_\theta(y_w) - \nabla_\theta\log\pi_\theta(y_l))$, which changes sign once a pair overshoots its target. A pair that has been learned too well is pulled back. Nothing in DPO does that.

Two details matter when you implement it. First, $\beta$ does not multiply $h$ inside the loss. It only sets the target, so $\beta = 0.1$ means a target of 5 nats and $\beta = 0.01$ a target of 50. Second, with $h$ computed from summed log-probabilities, a fixed target in nats means very different things for 20-token and 2,000-token responses. TRL's implementation divides each sequence's log-ratio by its number of tokens before forming $h$, and a comment in its source says the IPO authors confirmed that the paper's results correspond to this normalized form, although the paper itself does not discuss it.

The cell runs the two-response example. With a uniform reference and tabular logits, $h$ is the difference of the two logits and $\pi(y_1) = \sigma(h)$. Azar et al. derive the IPO optimum in closed form for this case. $p^\star(y_1\succ\mu) = \tfrac12\cdot1 + \tfrac12\cdot\tfrac12 = \tfrac34$ and $p^\star(y_2\succ\mu) = \tfrac14$, so $\pi^\star(y_1) = \sigma(\tfrac{1}{2\beta})$.

DPO drives $\pi(y_1)$ to essentially 1 for every $\beta$, and its margin is still growing when we stop. IPO converges to $h = 1/(2\beta)$, giving $\pi(y_1) = 0.993$ at $\beta = 0.1$ but only 0.622 at $\beta = 1$. Now $\beta$ controls the destination, which is what a regularization strength is supposed to do.

import numpy as np

def ipo_loss(pc, pr, rc, rr, beta, lengths_c=None, lengths_r=None):
    h = (pc - rc) - (pr - rr)
    return float(np.mean((beta * h - 1) ** 2))           # GPO's form, not IPO's

Noisy labels, conservative DPO and robust DPO

Human labels disagree with each other often. In the DPO paper's own human study on summarization, two humans judging the same comparison agreed only 65% of the time in one condition (87% in another). If a fraction $\varepsilon$ of the labels in a dataset are flipped, DPO's loss treats every one of them as certain, and because its gradient weight $\sigma(-\beta h)$ is largest on pairs the model ranks the "wrong" way, a flipped label that the model correctly disbelieves keeps receiving the strongest push.

Mitchell (2023) proposed the obvious fix, in a two-page note. Replace the hard target "$y_w$ wins with probability 1" with a soft target $1 - \varepsilon$. The binary cross-entropy against that target is

which is exactly label smoothing (§0.5) applied to the DPO classifier. Mitchell calls it conservative DPO (cDPO). Its derivative with respect to $h$ is $\beta\big(\sigma(\beta h) - (1 - \varepsilon)\big)$, which is zero when the model's implied preference probability equals the target,

In plain words, cDPO stops pushing once the implicit reward model is $1-\varepsilon$ confident, and pushes back beyond that. With $\beta = 0.1$ and $\varepsilon = 0.1$ the target is $10\log 9 = 22.0$ nats, and with $\varepsilon = 0.2$ it is 13.9. Like IPO it has a finite optimum. The two differ in what they fix. IPO targets a fixed log-ratio gap, while cDPO targets a fixed confidence. Mitchell's summary puts it well. cDPO trains until the implicit model assigns the desired probability to the observed preferences, and IPO trains until it reaches a desired reward gap. Older TRL versions implemented cDPO as the default sigmoid loss with label_smoothing set to $\varepsilon$. Recent versions keep label_smoothing only for the robust loss below and for EXO, so check which one your version computes.

Notice what cDPO does not change. On a pair the model ranks the wrong way by a wide margin, its push is $\beta(1-\varepsilon)$, barely less than DPO's $\beta$, so a flipped label that the model correctly disbelieves still gets nearly the full force. Christiano et al.'s floor model of §16.3 is a different fix for the same worry. It mixes the Bradley-Terry probability with a coin flip, $p = (1 - 2\varepsilon)\,\sigma(\beta h) + \varepsilon$, so the loss on a badly misranked pair is capped at $-\log\varepsilon$ and its gradient vanishes there. cDPO bounds overconfidence on the right side of the margin, and the floor model ignores hopeless pairs on the wrong side.

There is a third philosophy for the same problem. If labels are flipped with a known probability $\varepsilon$, one can build an unbiased estimate of the clean loss from the noisy one. Chowdhury, Kini, and Natarajan (2024) do this in robust DPO (rDPO),

Its expectation over the label noise is the clean DPO loss, so it corrects the bias that noise introduces, but it inherits DPO's unbounded optimum (as $h\to\infty$ the per-pair loss goes to $-\infty$). cDPO instead accepts a biased objective in exchange for a bounded one. TRL calls rDPO loss_type="robust", with label_smoothing as $\varepsilon$.

import numpy as np

def cdpo_loss(pc, pr, rc, rr, beta, eps):
    u = beta * ((pc - rc) - (pr - rr))
    return float(np.mean((1 - eps) * np.logaddexp(0, -u)))   # forgot the flipped-label term

def cdpo_target(beta, eps):
    return np.inf

KTO, learning from thumbs up and thumbs down

Much of the feedback a deployed model collects is not a pair. A user clicks thumbs up or thumbs down on one response. A safety reviewer marks a single output as unacceptable. A grader says pass or fail. DPO cannot use this data directly, and the reason is precisely the cancellation it is built on. The partition function $Z(x)$ cancels only when two responses to the same prompt are compared. A single response's log-ratio $\log\frac{\pi_\theta(y\mid x)}{\pi_{\text{ref}}(y\mid x)}$ is its reward minus $\beta\log Z(x)$, an unknown per-prompt offset, so "this response is good" does not tell the loss which way to move it without some baseline.

Ethayarajh, Xu, Muennighoff, Jurafsky, and Kiela (2024) supply the baseline from an unexpected place, prospect theory, the model of human decisions under risk of Kahneman and Tversky. Its central observation is that people judge an outcome relative to a reference point, not in absolute terms, and that losses loom larger than gains. Tversky and Kahneman (1992) fit a value function

with median parameters $\alpha = 0.88$ (diminishing sensitivity) and $\lambda = 2.25$ (a loss hurts about twice as much as an equal gain). The KTO authors argue that DPO and PPO's clipped objective already behave like value functions of this kind, call losses of this type human-aware losses (HALOs), and design one for unpaired data.

Kahneman-Tversky optimization (KTO) works with the log-ratio $g_\theta(x,y) = \log\frac{\pi_\theta(y\mid x)}{\pi_{\text{ref}}(y\mid x)}$ (the paper calls it $r_\theta$, and note there is no $\beta$ inside it) and a reference point equal to the policy's current KL from the reference,

the average log-ratio over the policy's own outputs (the expectation is over $y' \sim \pi_\theta(\cdot\mid x)$). The value of an example is a sigmoid of how far its log-ratio sits above or below that reference point,

where $\lambda_y$ is $\lambda_D$ or $\lambda_U$ for the example's class (subtracting it only makes the loss nonnegative). In plain words, push the log-ratio of good examples above the policy's average log-ratio and the log-ratio of bad examples below it, with a sigmoid that saturates in both directions.

The reference point is what makes unpaired learning work. Without it, the cheapest way to raise $g_\theta$ on a good example is to raise the probability of everything the example has in common with other text, which also raises the KL. With $z_0$ subtracted, a blunt increase moves both terms and gains nothing. The model has to raise this example's probability more than it raises responses in general, so it must learn what makes the example desirable. In this sense $z_0$ plays the role that the rejected response plays in DPO, a baseline that cancels whatever is common.

Three practical details come straight from the paper.

• Estimating $z_0$. Sampling from the policy at every step would be slow, so KTO estimates the reference point from the batch itself. It shifts the responses by one to form mismatched pairs $(x_i, y_{i+1})$, averages their log-ratios, clamps at zero, and shares the result across the microbatch, $\hat z_0 = \max\big(0, \frac1m\sum_i\log\frac{\pi_\theta(y_{i+1}\mid x_i)}{\pi_{\text{ref}}(y_{i+1}\mid x_i)}\big)$. No gradient flows through $\hat z_0$. It exists only to set where the sigmoid saturates. Notice that this is a crude estimate, since the $y_{i+1}$ are dataset responses rather than samples from $\pi_\theta$. This needs a microbatch of at least two, and the authors recommend batch sizes between 8 and 128.
• Hyperparameters. $\beta = 0.1$ is the default. $\lambda_D$ and $\lambda_U$ (both 1 by default) rebalance classes, and the paper recommends choosing them so that $\frac{\lambda_D n_D}{\lambda_U n_U}$ lies between 1 and $\frac32$, where $n_D$ and $n_U$ count desirable and undesirable examples (an earlier version of the paper, and TRL's documentation, give $\frac43$ as the upper end). With one desirable example per ten undesirable ones, that means $\lambda_U = 1$ and $\lambda_D$ between 10 and 15. The paper finds KTO's best learning rate is usually 2 to 10 times DPO's, and recommends $5\times10^{-6}$ with AdamW as a default.
• Results. Converting each preference pair into one desirable and one undesirable example, KTO matched or exceeded DPO at scales from 1B to 30B parameters in the paper's experiments, and matched DPO while discarding up to 90% of the desirable examples.

The gradient explains a qualitative difference from DPO. Differentiating $\lambda_D\sigma(\beta(g_\theta - z_0))$ gives a weight $\lambda_D\beta\,\sigma(\cdot)(1 - \sigma(\cdot))$, a bell curve in $g_\theta$ that peaks at the reference point and vanishes in both tails. KTO learns most from examples near the boundary and almost nothing from examples that are already far on the right side, or hopelessly far on the wrong side. The paper shows that this makes KTO implicitly ignore examples the model finds too hard, which it argues is often a blessing with noisy real-world feedback. DPO does the opposite, giving its largest weight to the pairs it ranks most wrongly.

At $g_\theta = -30$, an example the model is very sure is not good, KTO's weight has fallen to 0.0045, a sixth of its peak, while DPO's weight on an equally misranked pair is 0.095, close to its maximum of $\beta = 0.1$.

import numpy as np

def kto_loss(pol, ref, desirable, mm_pol, mm_ref, beta, lam_d=1.0, lam_u=1.0):
    r = pol - ref
    v = np.where(desirable, lam_d / (1 + np.exp(-beta * r)), lam_u / (1 + np.exp(beta * r)))   # no reference point
    return float(np.mean(np.where(desirable, lam_d, lam_u) - v)), 0.0

ORPO, one stage and no reference

Every method so far needs a reference model and an SFT stage before it. Hong, Lee, and Thorne (2024) ask whether both can go. Their starting observation is about SFT itself. Fine-tuning OPT-350M on only the chosen responses of the Anthropic HH dataset raised the log-probabilities of the rejected responses along with the chosen ones, because the two share style and domain. SFT teaches "write like this" without "and not like that". Odds ratio preference optimization (ORPO) adds the "not like that" to the SFT loss, in a single stage.

ORPO works with the per-token geometric-mean probability of a response, $P_\theta(y\mid x) = \exp\big(\frac{1}{|y|}\sum_t\log\pi_\theta(y_t\mid x, y_{<t})\big)$, and its odds, $\operatorname{odds}_\theta(y\mid x) = \frac{P_\theta(y\mid x)}{1 - P_\theta(y\mid x)}$. The loss is the SFT negative log-likelihood of the chosen response plus a logistic loss on the log odds ratio,

In plain words, imitate the chosen response, and at the same time make its odds of being generated higher than the rejected response's. The weight $\lambda$ balances the two terms. The paper's main models use $\lambda = 0.1$ (Mistral-7B), 0.2 (Llama-2 7B), and 0.25 (Phi-2), and its ablation compares 0.1, 0.5, and 1.0. A worked example sets the scale. A chosen response with average token log-probability $-0.5$ has $P = 0.607$ and odds 1.54. A rejected one at $-2.0$ has $P = 0.135$ and odds 0.157. The log odds ratio is $\log(1.54/0.157) = 2.29$, and $\mathcal{L}_{\text{OR}} = -\log\sigma(2.29) = 0.097$.

Why odds and not the probability ratio that DPO-style losses use? The authors argue that the probability ratio is too aggressive when preference learning is folded into SFT from the first step. Driving the log-sigmoid of a log probability ratio toward zero demands an extreme contrast, which over-suppresses the rejected tokens before the model has even adapted to the domain. The log odds ratio is distributed less sharply, so the same loss asks for a milder contrast, and their ablations favor it in this setting. The price of removing the reference is the anchor. Nothing in ORPO measures distance from a starting model, and only the SFT term keeps the policy on the data's distribution. Computing the log odds stably needs care, because $1 - P$ underflows as $P \to 1$. With $a = \frac{1}{|y|}\log\pi_\theta(y\mid x) \le 0$, use $\log\operatorname{odds} = a - \log(-\operatorname{expm1}(a))$.

import numpy as np

def orpo_loss(chosen_logps, chosen_len, rejected_logps, rejected_len, lam):
    Pc, Pr = np.exp(chosen_logps), np.exp(rejected_logps)          # forgot the per-token average
    lor = np.log(Pc / (1 - Pc)) - np.log(Pr / (1 - Pr))
    l_or = -np.log(1 / (1 + np.exp(-lor)))
    sft = -chosen_logps
    return float(np.mean(sft + lam * l_or)), float(np.mean(sft)), float(np.mean(l_or))

SimPO, a length-normalized margin with no reference

Meng, Xia, and Chen (2024) start from a mismatch. A model generates text by (approximately) maximizing likelihood, and when several candidates are compared, as in beam search or multiple-choice scoring, the usual criterion is the average log-probability per token (§6.6). DPO's implicit reward is something else, a summed log-ratio to a reference. The authors measured that after DPO training, on only about half of the training triples did the ranking by DPO's implicit reward agree with the ranking by average log-likelihood, so the criterion the model generates by barely agreed with the reward it was trained on. (Chen et al. (2024) report a related finding across many released preference-tuned models, whose ranking accuracy on common preference datasets is mostly below 60%.)

Simple preference optimization (SimPO) makes the reward the generation criterion itself, scaled, and adds a target margin $\gamma$ to Bradley-Terry,

In plain words, make the chosen response's average token log-probability exceed the rejected one's by more than $\gamma/\beta$, and stop caring much beyond that. Three design choices are packed into it.

• Length normalization. A summed log-probability is lower for longer responses, so a summed reward forces the model to inflate the probability of long chosen responses unnaturally, and the authors found that removing the normalization made the model produce longer, lower-quality responses. Averaging removes the systematic effect of length.
• No reference model. The reward needs only the policy, which saves the memory and compute of the reference, and removes the KL anchor along with it.
• A target margin. The term $\gamma$ shifts the sigmoid so that a pair contributes a substantial gradient until its reward gap exceeds $\gamma$. Pairs that are "barely right" keep being pushed.

The paper's guidance is $\beta$ between 2.0 and 2.5 and $\gamma$ between 0.5 and 1.5 (both on the scale of averaged log-probabilities, hence much larger $\beta$ than DPO's), and it stresses that careful tuning mattered for every method it compared. In its experiments SimPO beat DPO by up to 6.4 points of length-controlled win rate on AlpacaEval 2 and up to 7.5 points on Arena-Hard, and its strongest model, Gemma-2-9B-it trained with SimPO, reached a 72.4% length-controlled win rate on AlpacaEval 2 and 59.1% on Arena-Hard.

Two close relatives are worth knowing. Length-normalized DPO, used by Tülu 3, keeps the reference but averages its log-ratios, $\frac{\beta}{|y_w|}\log\frac{\pi_\theta(y_w\mid x)}{\pi_{\text{ref}}(y_w\mid x)} - \frac{\beta}{|y_l|}\log\frac{\pi_\theta(y_l\mid x)}{\pi_{\text{ref}}(y_l\mid x)}$ inside the sigmoid, and in Tülu 3's own ablation it was the only variant (of DPO, SimPO, and length-normalized DPO) that beat their starting checkpoint overall, with $\beta = 5$. Length-regularized DPO (R-DPO, Park et al. 2024) keeps sums but adds $\alpha(|y_w| - |y_l|)$ inside the sigmoid, derived by putting a length penalty $-\alpha|y|$ in the reward, so a longer chosen response needs a smaller log-ratio gap to be explained. The disagreement between SimPO's and Tülu 3's results is itself instructive. Which variant wins depends on the model, the data, and how much each was tuned.

The rejected answer has a far higher summed log-probability only because it is a quarter of the length. Per token, the chosen answer is more likely, and SimPO's reward gap is $2.0 \times 0.3 = 0.6$, still short of the target margin of 1.0, so the pair keeps pulling.

import numpy as np

def simpo_loss(token_logps_c, mask_c, token_logps_r, mask_r, beta, gamma):
    s_c, s_r = token_logps_c.sum(1), token_logps_r.sum(1)           # sums over all positions, no normalization
    return float(np.mean(np.logaddexp(0, -beta * (s_c - s_r))))      # and no margin

Keeping the chosen response likely

Section 17.3 showed the chosen response's log-probability falling during an ordinary DPO run. A family of fixes adds a term that resists it. DPO-Positive (DPOP, Pal et al. 2024) subtracts $\lambda\max\big(0, \log\frac{\pi_{\text{ref}}(y_w\mid x)}{\pi_\theta(y_w\mid x)}\big)$ from $h_\theta$ inside the sigmoid (so the sigmoid sees $\beta$ times the penalized margin), a penalty that switches on only when the chosen response has become less likely than under the reference. Simpler and more common is a plain negative log-likelihood term on the chosen response, added to the DPO loss with a small weight. Iterative reasoning preference optimization (Pang et al. 2024) found that term crucial for math reasoning, and Llama 3 uses it with weight 0.2. Exercise 17.3c implemented exactly this objective. Section 17.5 asks when the fall in chosen likelihood is harmless and when it is dangerous, which decides whether such a term is needed.

The family at a glance

The widget plots each loss as a function of a pair's log-ratio difference $h$ (or, for SimPO, the gap in average log-probability), together with its gradient. The shapes tell the story of the section. DPO's loss falls forever. IPO is a parabola with its bottom at $1/(2\beta)$. cDPO has a bottom at $\frac1\beta\log\frac{1-\varepsilon}{\varepsilon}$ and rises beyond it. The hinge of SLiC (Zhao et al. 2023) is flat once the margin is met. SimPO is DPO's curve shifted right by its margin.

How to choose, in practice? Start with DPO, or its length-normalized version if your data reward length, at the published defaults, and watch the metrics of §17.3. If the chosen log-probability collapses or generations degrade, add a small NLL term. If training accuracy races toward 100% and margins grow without bound, the pairs are being memorized, so try IPO or cDPO, or just fewer epochs. If your feedback is single thumbs-up or thumbs-down, or one class is rare, use KTO rather than inventing pairs. ORPO and SimPO save a reference model and, for ORPO, a whole SFT stage, at the cost of the KL anchor. Above all, hold your expectations loosely. Practitioners widely believe, and Ivison et al. (2024) found in a controlled study, that the quality of the preference data matters more than the choice of loss, and published comparisons of these variants frequently disagree once each method is tuned.

In PyTorch

All of these losses are a few lines on top of the same four (or two) vectors of log-probabilities. One function covers the paired ones, and autograd confirms the gradients we derived by hand, including the sign change of IPO and cDPO past their targets.

Below its target, each loss pushes the chosen log-probability up (negative gradient, so gradient descent increases it). Above the target, the sign flips and it pulls the pair back, the behavior that bounds these methods and that plain DPO lacks.

Read more

• Azar, Rowland, Piot, Guo, Calandriello, Valko & Munos (2023). A General Theoretical Paradigm to Understand Learning from Human Preferences. $\Psi$PO, the overfitting argument for DPO, and the derivation of IPO with its bandit examples.
• Mitchell (2023). A note on DPO with noisy preferences & relationship to IPO. Two pages that define conservative DPO and compare its gradient with IPO's.
• Ethayarajh, Xu, Muennighoff, Jurafsky & Kiela (2024). KTO: Model Alignment as Prospect Theoretic Optimization. Human-aware losses, the KTO loss and its reference point, and practical hyperparameter advice.
• Hong, Lee & Thorne (2024). ORPO: Monolithic Preference Optimization without Reference Model. The single-stage odds-ratio objective and its gradient analysis.
• Meng, Xia & Chen (2024). SimPO: Simple Preference Optimization with a Reference-Free Reward. Length-normalized rewards, the target margin, and careful comparisons with tuned baselines.
• Tang, Guo, Zheng, Calandriello, Munos, Rowland, Richemond, Valko, Ávila Pires & Piot (2024). Generalized Preference Optimization: A Unified Approach to Offline Alignment. DPO, IPO, and SLiC as convex losses of the same margin, and how the choice of loss sets the regularization.
• Chen, Malladi, Zhang, Chen, Zhang, Ranganath & Cho (2024). Preference Learning Algorithms Do Not Learn Preference Rankings. Measures how rarely preference-tuned models rank the chosen response above the rejected one by likelihood, and explains why DPO struggles to flip a ranking the reference gets wrong.
• Chowdhury, Kini & Natarajan (2024). Provably Robust DPO: Aligning Language Models with Noisy Feedback. The unbiased robust DPO loss for labels flipped at a known rate.

Watch next

▶

▶