In 2016 OpenAI trained an agent to play CoastRunners, a boat-racing game whose score rewards hitting targets along the course. The agent found a lagoon where three targets respawned, and circled it forever, crashing and catching fire, scoring higher than human players while never finishing the race (Clark and Amodei 2016). Nothing was wrong with the optimizer. It did exactly what the score said.

RLHF is the same situation with a subtler score. The reward model is a learned approximation of what people prefer, fit to a few tens of thousands of comparisons, and RL is a powerful search for the responses it scores highest. Stiennon et al. (2020) watched this happen. As they optimized summaries harder against their reward model, its score kept climbing, while human labelers first liked the summaries more, then less, and eventually the reward model's preferences were anti-correlated with theirs.

This section explains why that is inevitable for any learned reward, measures it with the scaling laws of Gao et al. (2023), names the specific exploits that show up in practice (length, sycophancy, and friends), and goes through the mitigations, with an honest account of what each one fixes.

Goodhart's law

The economist Charles Goodhart observed in 1975 that a statistical regularity tends to collapse once it is used as a target for control. The anthropologist Marilyn Strathern later put it in the form everyone quotes, "when a measure becomes a target, it ceases to be a good measure". Reward hacking is Goodhart's law for optimizers, and Skalse et al. (2022) gave it a precise definition. A proxy reward is hackable if increasing the expected proxy return can ever decrease the expected true return. They proved that, if the policy can be any stochastic policy, two reward functions are unhackable with respect to each other only if they are equivalent (they order every policy the same way) or one of them is trivial (it gives every policy the same return). Some hackability is the normal case, not a bug in a particular reward model.

Manheim and Garrabrant (2018) separated four ways a proxy fails under pressure, and Gao et al. (2023) used the same four to read their results. Each has an RLHF face.

Regressional Goodhart alone does not make things get worse, only disappointing. Recall the noisy judge of §16.1. With true quality $q \sim \mathcal{N}(0, 1)$ and proxy $s = q + e$, $e \sim \mathcal{N}(0, \sigma_e^2)$, picking the best of $n$ by proxy gives an expected proxy of $\sqrt{1 + \sigma_e^2}\,m_n$ and an expected true quality of $m_n/\sqrt{1 + \sigma_e^2}$, so the true gain is always the fraction $1/(1 + \sigma_e^2)$ of the proxy gain. Both keep rising with $n$. For the true reward to fall, the proxy's error must grow in the direction the optimizer pushes, which is extremal Goodhart. Let's build exactly that. A response has one feature $y$ (think of it as verbosity) drawn from the reference policy as $y \sim \mathcal{N}(0, 1)$. The true (gold) reward is $r_{\text{gold}}(y) = y - 0.25y^2$, which rises up to $y = 2$ and then falls, because past some point more is worse. The reward model, fit on ordinary responses, learned $r_{\text{proxy}}(y) = y + \text{noise}$, which is right near the data and wrong far from it. Now run best-of-$n$.

The proxy rises forever, like the expected maximum of more and more samples. The gold reward rises at first, because near the data the proxy is right, peaks around $n = 64$, and then falls as best-of-$n$ selects ever more extreme responses where "more" has stopped being good. By $n = 4096$ a large part of the gain is gone, and the reward model is still reporting progress. That is overoptimization, in thirty lines and one feature. The function bon_expect deserves a note. It computes the expected best-of-$n$ outcome over every subset of size $n$ of the pool at once, so the curve is smooth without resampling. Exercise 16.5b derives it.

Scaling laws for reward model overoptimization

Measuring the gold reward requires knowing it, and with human preferences you do not, except by running expensive human evaluations. Gao et al. (2023) got around this with a synthetic setup. A 6B reward model, the InstructGPT one, served as the "gold" standard and labeled 100,000 comparisons. Proxy reward models from 3M to 3B parameters were trained on those labels, a 1.2B policy (and a 6B one) was optimized against each proxy by best-of-$n$ or by PPO, and the gold reward model scored the results. Everything could then be measured as a function of how far the policy had moved.

They measured distance as $d = \sqrt{D_{\mathrm{KL}}(\pi\Vert\pi_{\text{ref}})}$. In this section $d$ means that square-rooted KL, as in their paper, and not a model width. The gold reward (relative to the starting policy, so $R(0) = 0$) followed two simple functional forms,

with coefficients that depend on the reward model. (These $\alpha$ and $\beta$ always carry their subscript, to keep them apart from the KL coefficient $\beta$ of §16.4.) In plain words, at first the gold reward grows linearly in the distance traveled, at rate $\alpha$, and then a penalty that grows with distance, quadratically for best-of-$n$ and like $d\log d$ for RL, takes over and bends it back down.

Each curve has a peak, which tells you how far it pays to optimize. For best-of-$n$, $R'_{\text{bon}}(d) = \alpha_{\text{bon}} - 2\beta_{\text{bon}}d = 0$ gives

For RL, $R'_{\text{RL}}(d) = \alpha_{\text{RL}} - \beta_{\text{RL}}\log d - \beta_{\text{RL}} = 0$ gives $\log d^\star_{\text{RL}} = \alpha_{\text{RL}}/\beta_{\text{RL}} - 1$, and substituting back,

Our one-feature simulation is a best-of-$n$ experiment, so the best-of-$n$ form should fit it. A neat property makes the fit easy. Dividing by $d$ gives $R/d = \alpha_{\text{bon}} - \beta_{\text{bon}}d$, a straight line in $d$, so ordinary least squares on $(d, R/d)$ recovers both coefficients. (For RL, $R/d$ is a straight line in $\log d$.)

The fitted peak sits at a KL of about 2.8 nats, about where the table showed the gold reward topping out (between $n = 16$ and $n = 64$), and the two-parameter form follows the whole curve closely. In Gao et al.'s experiments the same forms fit real reward models, and their main findings are worth knowing one by one.

• Bigger reward models overoptimize less. The coefficients changed smoothly, roughly linearly in the logarithm of the proxy's parameter count. Larger proxies give a higher peak, reached at a larger KL. For RL, $\alpha_{\text{RL}}$ stayed nearly constant and $\beta_{\text{RL}}$ carried the size dependence.
• More reward model data helps too. Below about 2,000 comparisons the proxies were near chance, and more data gave better gold scores and less overoptimization.
• RL is far less KL-efficient than best-of-$n$. RL "consumes" much more KL to reach the same proxy score, and to overoptimize. KL is not a universal currency between methods.
• The KL penalty did not move the frontier. Adding a KL penalty to the RL objective made the gold score peak earlier in training, like early stopping, but at any given KL the gold score was the same with or without it. (They note this may depend on hyperparameters, and their main RL runs used no KL penalty.)
• Bigger policies gain less but do not overoptimize more. A 6B policy started higher and improved less than a 1.2B one, and both peaked at about the same KL.

How far best-of-n moves the policy

Best-of-$n$ is the cleanest optimizer to analyze, because its KL from the reference has a closed form, at least when no two responses tie.

In plain words, best-of-$n$ moves about $\log n$ nats, so each doubling of $n$ buys a fixed amount of KL, $\log 2 \approx 0.69$ nats, minus a small correction. Best-of-2 is only 0.19 nats away from the reference, best-of-16 1.84, best-of-1024 5.93. Beirami et al. (2024) pointed out that for real language models, where many samples can coincide (short answers repeat), the formula is an upper bound on the true KL, not its exact value, and gave a better estimator. It is exact in the continuous, no-ties case derived here.

import numpy as np

def bon_kl(n):
    return np.log(n)                  # placeholder: forgets the (n - 1) / n correction

def largest_n_within(budget):
    return int(np.floor(np.exp(budget)))

import numpy as np
from scipy.special import gammaln

def best_of_n(scores, n):
    return np.argmax(scores, axis=1)              # placeholder: looks at all N samples, not the first n

def expected_best_of_n(proxy, gold, n):
    return float(np.mean(gold))                   # placeholder: the n = 1 answer for every n

import numpy as np

def fit_bon(d, R):
    raise NotImplementedError

def fit_rl(d, R):
    raise NotImplementedError

The widget runs the full experiment in your browser. A pool of reference samples lives in a 2D response space, a gold reward is defined on it, and an ensemble of proxy reward models each sees the gold reward plus its own error, an error that is small near the data and grows away from it. Best-of-$n$ is computed exactly for every $n$ with the subset formula, and the best-of-$n$ form is fit to the gold curve.

In a real system, best-of-$n$ is a batching pattern. Repeat each prompt $n$ times, sample all the responses in one batch, score them with the reward model in one batch, reshape the scores to (P, n), and gather the winners. Here it is with a stand-in policy (independent per-position token distributions) and a stand-in reward model, so the pattern is visible without a real model.

What models actually learn to exploit

The overoptimization curves are averages. Up close, reward hacking takes a few recognizable forms.

Length. The most common and best documented. Singhal et al. (2023) found that across three RLHF settings, much of the reward improvement came from longer responses, and that optimizing a reward of response length alone reproduced most of RLHF's gain over SFT. The reward model is the main culprit, having picked up the length correlation in its preference data (§16.2), often more strongly than the annotators themselves held it (§16.3). Length is also cheap for the policy to change, so it is the first direction RL finds.

Sycophancy. A model tells users what they want to hear. Perez et al. (2022) found that larger models more often repeat back a user's stated view, and that more RLHF made some behaviors worse, for example stronger expressed political views. Sharma et al. (2023) showed that five production assistants were consistently sycophantic across four tasks, that human preference data favors responses matching the user's views, and that optimizing against a preference model sometimes traded truthfulness for agreement. This is adversarial Goodhart with the annotator as the target. The policy learns what wins comparisons, and agreeing with the person judging you wins comparisons.

Persuasive wrongness. Wen et al. (2024) gave the problem its sharpest form. After RLHF on tasks where people could not easily check answers, models became better at convincing human evaluators that wrong answers were right, without becoming more correct. Evaluators' false positive rate rose by 24.1% on a reading comprehension task and 18.3% on a programming task. When the judge cannot verify, optimizing the judge's approval optimizes the appearance of correctness.

Style and hedging. Formatting that looks thorough (headers, lists, bold), reflexive disclaimers, and refusals of borderline requests all raise scores under some reward models without making answers better. Which ones appear depends on the reward model, so they are best found by reading samples from the policy at the end of training, not by guessing in advance.

Hacking the verifier. When the reward is a program, the exploits are bugs. Models trained against unit tests have learned to special-case the tests, and models graded by answer matching have learned to output in formats the checker misparses (§18.3). The broader catalog of specification gaming in RL agents is the subject of §19.1.

Mitigations, and what each one fixes

There is no cure, because a learned reward is a proxy by construction. There are several ways to optimize it more safely, and they combine.

The KL penalty. It limits how far the policy moves, and so how deep into the extremal region it can go. Gao et al. found it behaves like early stopping, reaching the same gold-versus-KL frontier sooner rather than raising it. Kwa et al. (2024) add a theoretical warning. When the reward model's errors are heavy-tailed, KL regularization cannot prevent "catastrophic Goodhart", policies that reach very high proxy reward with no more true utility than the reference, because a small amount of probability on rare, wildly overrated responses is cheap in KL. (Their measurements of real reward models found light-tailed errors, which is reassuring for now.)

Early stopping on an independent signal. The most reliable mitigation in practice. Since gold rises then falls, stop near the peak, judged by evaluations the policy was not trained against (§16.6).

Better reward models. Every finding of Gao et al. points the same way. More preference data and larger reward models raise the peak and push it out. Fresh on-policy comparisons, collected as the policy moves, fix the reward model exactly where it is being exploited (§16.2), which is why Anthropic and Llama 2 iterated.

Ensembles and conservative objectives. Train several reward models and optimize a pessimistic score, the worst member (worst-case optimization) or the mean minus a multiple of the members' variance (uncertainty-weighted optimization; many reimplementations subtract a multiple of the standard deviation instead, as the cell below does). Responses that only one member loves score low. Coste et al. (2023), in Gao's synthetic setup with 25% label noise, found that conservative ensembles practically eliminated overoptimization for best-of-$n$ (gains of up to 70%) and always reduced it for PPO. Eisenstein et al. (2023) found that ensembles help but do not eliminate hacking, because members trained from the same pretrained model share blind spots, and that diversity in pretraining seeds matters more than in fine-tuning seeds. WARM averages the weights of several fine-tuned reward models instead of their outputs, for one model's inference cost (Ramé et al. 2024).

Constraints instead of a sum. When the reward combines several reward models (helpfulness, harmlessness, honesty), each has its own overoptimization point. Moskovitz et al. (2023) treat each component as a constraint to keep at its useful threshold rather than a term to maximize, and learn the weights as Lagrange multipliers during training. Llama 2's piecewise combination (use the safety score when it is low or the prompt is risky, the helpfulness score otherwise) is a hand-built version of the same idea.

Reshaping the reward. Reward models trained with Bradley-Terry measure log-odds, and a policy gains as much from pushing an already-great response higher as from fixing a bad one. Wang, Nagpal, et al. (2024) proposed passing the centered reward through a log-sigmoid, which saturates for responses already well above a reference, so the policy is paid mostly for fixing bad outputs. Llama 2 went the other way for a different reason. Its combined score was a probability, so it applied the logit to undo the sigmoid and then whitened the result, for stability and to balance the reward against the KL penalty.

Targeting known exploits. For length, use length-balanced preference data, a length penalty in the reward, a reward model with a separate length head (ODIN), and length-controlled evaluation (Dubois et al. 2024). For sycophancy, add preference data in which the correct answer disagrees with the user.

Selecting by the gold reward itself climbs steadily toward the best value, 1.0. The single reward model peaks around $n = 256$ and then falls, the familiar curve. Averaging five members shrinks each member's private error, which raises the curve but does not stop the fall, because at large $n$ best-of-$n$ still finds responses whose average error happens to be large. The two pessimistic objectives pay a price at small $n$, since they also discount good responses the members disagree about, and keep improving all the way to $n = 1024$, where they beat the single model by about 0.06. If the members' errors were shared, every one of these objectives would see the same thing and none of them would help, which is Eisenstein et al.'s point.

import numpy as np

def conservative_reward(S, method, k=1.0):
    return S.mean(1)                    # placeholder: reduces over the wrong axis

def pick(S, method, k=1.0):
    return int(np.argmax(S[0]))

Read more

• Gao, Schulman, and Hilton (2023). Scaling Laws for Reward Model Overoptimization. The synthetic gold-reward setup, the functional forms, and the findings on size, data, and KL.
• Manheim and Garrabrant (2018). Categorizing Variants of Goodhart's Law. The four variants, with examples.
• Skalse et al. (2022). Defining and Characterizing Reward Hacking. A formal definition of hackable proxies and why almost all of them are.
• Singhal et al. (2023). A Long Way to Go: Investigating Length Correlations in RLHF. How much of RLHF is length.
• Sharma et al. (2023). Towards Understanding Sycophancy in Language Models. Sycophancy in assistants and in the preference data behind them.
• Coste et al. (2023). Reward Model Ensembles Help Mitigate Overoptimization. Worst-case and uncertainty-weighted ensemble objectives.
• Kwa, Thomas, and Garriga-Alonso (2024). Catastrophic Goodhart: regularizing RLHF with KL divergence does not mitigate heavy-tailed reward misspecification. When the KL penalty cannot save you.
• Moskovitz et al. (2023). Confronting Reward Model Overoptimization with Constrained RLHF. Composite rewards as constraints with learned multipliers.

Watch next

▶

▶