What it means for a model to do what we want, how the specification gets written down and trained in, how models are attacked and evaluated, how to supervise systems that may outgrow their supervisors, and what it takes to deploy them safely, with every claim tied to the evidence behind it.

In 2016 researchers at OpenAI trained a reinforcement learning agent to play CoastRunners, a boat-racing game. The game awards points for hitting targets laid out along the course, and points were the reward. The agent found a small lagoon where three targets respawn, and it learned to turn in a tight circle there forever, crashing into other boats, catching fire, and never finishing the race. It scored, on average, 20 percent higher than human players who did finish (Clark and Amodei 2016).

Nothing in that story is a bug in the learning algorithm. The optimizer did its job perfectly. It found a policy with a very high return for the reward it was given. The trouble is that the reward was not the task. The designers wanted the boat to win the race, wrote down "maximize points" as a stand-in, and the stand-in and the goal came apart exactly where a strong optimizer looked hardest.

This chapter is about that gap, between what we want a model to do and what we actually trained it to do, and about the tools the field has built to close it, test it, and live with it when it cannot be closed. It is the last chapter of Part IV, so it leans on everything before it. RLHF (§16.4) and its failure modes (§16.5), DPO (§17.2), and reasoning models trained with verifiable rewards (§18.5) are all, at bottom, attempts to write down what we want. This first section gives the vocabulary. What exactly can go wrong, in what order, and where do the "values" we align to come from in the first place?

Three objectives, not one

It helps to keep three different objectives apart, because almost every alignment failure is a gap between two of them.

1. The intended objective is what the designers actually want. Call it the true reward $r^\star(s, a)$. For CoastRunners it is "win the race". For an assistant it is something like "be genuinely useful to this person without causing harm", which nobody can write down exactly.
2. The specified objective is what we can write down and optimize, the proxy reward $\hat r(s, a)$. It might be a game score, a reward model trained on human comparisons (§16.3), a unit-test pass rate (§18.3), or the log-likelihood of demonstrations (§14.1).
3. The learned objective is whatever the trained system ends up pursuing. For a lookup-table policy this phrase is just a description of its behavior. For a large network trained on many tasks it might be something more, an internal goal the network has learned to optimize, which may or may not match the specified one.

Here $s$ is a state and $a$ an action, as in the Markov decision processes of §15.1. Write the expected discounted return of a policy $\pi$ under a reward $r$ as

with discount $\gamma \in [0, 1)$ and the expectation over trajectories that start from the start distribution and follow $\pi$. In plain words, $J(\pi; r)$ is how well $\pi$ does when we score it with $r$.

Hubinger et al. (2019) named the two gaps. Outer alignment is the question of whether the specified objective matches the intended one, whether $\hat r$ is a good stand-in for $r^\star$. Inner alignment is the question of whether the learned objective matches the specified one, whether training on $\hat r$ actually produced a system that pursues $\hat r$, rather than something that merely correlated with $\hat r$ on the training distribution. Their paper framed inner alignment around mesa-optimizers, learned models that are themselves optimizers with their own mesa-objective, distinct from the base objective the training process optimizes. Whether today's language models contain anything like a mesa-optimizer is an open question. The distinction is useful even if they do not, because it separates two failures that need different fixes.

A bookkeeping identity makes the split concrete. Let $\pi^\star$ be optimal for $r^\star$, let $\hat\pi^\star$ be optimal for the proxy $\hat r$, and let $\hat\pi$ be the policy training actually produced. The regret on the true task splits into two terms,

The identity is trivial (add and subtract the same number), but it says something useful. In plain words, a bad outcome can come from a perfect optimizer pointed at the wrong target, or from an imperfect optimizer that landed somewhere other than the target it was pointed at. The first term is the outer problem. The second contains the inner problem, along with ordinary optimization error. Notice that the second term can be negative. A weak optimizer that fails to exploit a broken proxy can do better on the true task than a strong one that succeeds, which is why reward hacking tends to get worse as optimization gets stronger (§16.5).

A useful map of where these gaps show up in practice is still the one Amodei et al. (2016) drew in Concrete Problems in AI Safety. They listed five problems, each a way a reasonable objective goes wrong in a real system. Avoiding negative side effects (a cleaning robot told to clean fast knocks over a vase, because the reward says nothing about vases) and avoiding reward hacking (the robot covers the mess instead of cleaning it) are outer alignment failures, the first from what the specification leaves out and the second from what it gets wrong. Scalable oversight is the problem of supervising behavior that is too expensive or too hard to check, the subject of §19.5. Robustness to distributional shift covers a system that meets inputs unlike its training data, where goal misgeneralization below and the confident errors of §19.8 live. And safe exploration asks how a learner can try new actions without trying catastrophic ones, which matters less for chat models than for agents that act in the world. The rest of this chapter takes these problems up for language models, roughly in that order of urgency.

Specification gaming, documented

Specification gaming is the name for behavior that satisfies the letter of the specified objective while missing its intent (Krakovna et al. 2020). It is the outer alignment problem seen from the outside. The DeepMind safety team keeps a crowd-sourced list of documented cases (about 60 when the list was announced, many more since), and reading it is the fastest way to lose any belief that a reward function means what you think it means. Here are a few classics, with their sources.

The last two rows are the modern versions, and §19.6 returns to both. What every row shares is that the specification was reasonable. Nobody wrote a silly reward. They wrote a reward that agrees with the intended objective on the behaviors they had in mind, and the optimizer found behaviors they did not have in mind.

The far end of this spectrum is reward tampering, where the system changes the process that computes its reward rather than gaming the reward's letter. Denison et al. (2024) trained language-model assistants on a curriculum of increasingly gameable environments, starting with sycophancy, and found that gaming the early environments made gaming the later ones more likely. Models trained on the whole curriculum occasionally went on, without having been trained on it, to edit the code of their own reward function, a small but non-negligible fraction of the time. The setting was constructed to make tampering possible, so it shows that the step from gaming to tampering can generalize, not that it happens in ordinary training. Later work found that reward hacking learned in real production coding environments can generalize to much broader misbehavior, which §19.6 describes as emergent misalignment.

Why is this the default rather than an occasional accident? Think of the set of all policies. The designer imagines a small region of "sensible" policies and checks that $\hat r$ ranks them correctly. The optimizer searches the whole set. Any place where $\hat r$ and $r^\star$ disagree, however exotic, is a place the optimizer may go, and the harder it searches the more likely it is to find one. Skalse et al. (2022), whose result §16.5 introduced, made this precise. Call a pair of rewards unhackable on a set of policies if no policy change that increases the proxy return ever decreases the true return. They proved that if the set of policies contains an open set, as the set of all stochastic policies does, then any two rewards that are unhackable and non-trivial on it are equivalent, meaning they order every policy identically. (A reward is trivial on a set if it gives every policy the same return.) So over all stochastic policies, a proxy that is safe to optimize without limit is either useless or already the true reward. This is a theorem. Its practical reading is that safety comes from restricting the search (a KL penalty to a reference model, as in §15.7, early stopping, a limited policy class), not from finding a perfect proxy.

Goodhart and the tail of the error

Section 16.5 treated Goodhart's law ("when a measure becomes a target, it ceases to be a good measure") for reward models, with its four variants, regressional, extremal, causal, and adversarial (Manheim and Garrabrant 2018). Two of its results matter most here. Regressional noise alone only shrinks the true gain to a fixed fraction of the proxy gain, so optimizing harder still helps. And a true decline needs proxy errors that grow in the direction the optimizer pushes, which is extremal Goodhart. Adversarial Goodhart, another agent exploiting the proxy, is the jailbreak story of §19.3.

For safety there is a third route to decline that does not need errors to grow in any particular direction, only to have heavy tails. Let each candidate have true value $U \sim \mathcal{N}(0, 1)$ and proxy score $X = U + \varepsilon$, and keep the best of $n$ by proxy, as in §18.2. If $\varepsilon$ is Gaussian, the true value of the pick keeps rising with $n$, as §16.5 showed. If $\varepsilon$ has heavy tails, the candidate with the highest proxy score is, more and more often, the one with a freak error rather than a high true value, and the true value of the winner rises and then falls back toward the average. Kwa et al. (2024) proved a version of this for KL-regularized RL. When the reward model's error is heavy-tailed, a policy can reach arbitrarily high proxy reward with almost no gain in true utility while staying within a small KL budget. Their own measurements of open reward models were consistent with light-tailed error, so this is a possibility to guard against rather than an observed failure. Watch both regimes.

With Gaussian error the pick's true value climbs steadily, from 0 to about 2.6 at $n = 4096$, always about $1/\sqrt{2}$ of the oracle's (the factor $\sqrt{\operatorname{Var}(U)/\operatorname{Var}(X)}$ for $\sigma = 1$, since the maximum of $n$ Gaussians scales with their standard deviation). With heavy-tailed error it peaks around $n = 16$ and then sinks back toward zero, the true value of a random candidate. The two proxies look alike for typical candidates, where the error is small either way, and meet opposite fates under pressure. The safety reading is that the most dangerous failures of a proxy are its rare, extreme errors, and that the harder we optimize, the more of our outcome those rare errors decide. That is an argument for limiting optimization pressure and for auditing the highest-scoring outputs specifically, not a random sample of them.

A specification-gaming gridworld

Let's build CoastRunners in miniature. A $5\times5$ grid, the agent starts in the top-left corner, and the task is to reach the goal in the bottom-right. The true reward is $+10$ for arriving at the goal and $-0.1$ for every other step, so the intended behavior is the shortest path. A well-meaning designer adds a bonus $b = 1$ every time the agent enters a checkpoint cell at $(1, 1)$, to encourage exploration. The bonus respawns. We will solve both rewards exactly with value iteration (§15.2), using the transition tensor $P \in \R^{S\times A\times S}$ and reward table $R \in \R^{S\times A}$ of Chapter 15.

The proxy-optimal agent steps down to $(1, 0)$, into the checkpoint, out to $(0, 1)$, back in, and so on forever. Its proxy return is 7.74, better than the 6.38 a shortest path earns under the true reward (or 7.33 under the proxy, if that path passes through the checkpoint once on the way), and its true return is $-2.00$, the value of paying $0.1$ per step for eternity ($-0.1/(1-\gamma)$). Why is looping better? Each two-step loop pays $b - 0.2$, roughly, and a discounted stream of them is worth about $\frac{b}{1-\gamma^2} \approx 10.3\,b$ at $\gamma = 0.95$, while the goal, eight steps away, is worth only $10\gamma^7 \approx 7$ minus the step costs. The farther the goal and the larger $\gamma$, the smaller the bonus needed to break the agent.

import numpy as np

def policy_return(P, R, pi, gamma, s0):
    raise NotImplementedError

def proxy_regret(P, R_true, R_proxy, gamma, s0):
    raise NotImplementedError

The one kind of bonus that is always safe

Was the designer wrong to add a bonus at all? No. Shaping rewards, extra rewards that guide exploration, are often essential when the true reward is sparse. They were wrong about its form. Ng, Harada, and Russell (1999) proved that one family of bonuses can never change which policies are optimal. Pick any function of the state, a potential $\psi: \mathcal{S}\to\R$ (they write $\Phi$), and add

to the reward of every transition from $s$ to $s'$, with $\psi = 0$ at terminal states. In plain words, you are paid for moving to a state with higher potential and charged for moving to one with lower potential, and the discount is applied to the arrival.

In the gridworld, a potential-based checkpoint bonus pays $\gamma b$ on entry and charges $b$ on exit. One loop then nets $\gamma b - \gamma b = 0$ in discounted terms (the exit happens one step later), so looping is worthless. Try the "potential-based bonus" button in the widget. The lesson generalizes beyond gridworlds. Any reward term that can be collected repeatedly by cycling through states will be farmed. Rewards that depend only on progress, a difference of a state function, cannot be.

import numpy as np

def shaped_reward(P, R, psi, gamma, terminal):
    return R + psi[:, None]          # placeholder: a bonus that is not potential-based

Goal misgeneralization

Specification gaming is a failure you could, in principle, find by reading the reward function carefully. The next failure survives even a perfect reward.

Langosco et al. (2022) trained agents on CoinRun, a procedurally generated platform game in which the agent gets a reward for touching a coin. In every training level the coin sat at the far right end of the level. The reward was correct, it paid for the coin and nothing else. At test time the researchers moved the coin to a random position. The trained agent ran past it to the right end of the level, competently dodging obstacles all the way, and usually ignored the coin. Its capabilities generalized to the new levels. Its goal did not. During training "go to the coin" and "go to the right end" were the same behavior, and the agent had learned the second.

Shah et al. (2022) named this goal misgeneralization and defined it carefully. A system trained with a correct specification behaves competently out of distribution, yet pursues a goal other than the intended one. Both conditions matter. If the agent had stumbled around randomly at test time, that would be ordinary capability failure, which is easy to notice. Goal misgeneralization is worse because the agent stays capable. It competently does the wrong thing.

Why can training not prevent it? Here is the identifiability argument in one line. Let $d_{\text{train}}$ be the distribution over states the agent sees during training, and suppose two candidate goals, $r_1$ (reach the coin) and $r_2$ (reach the right end), agree on every state in its support. Then every training signal, every return, every gradient, is identical under the two goals, so

In plain words, if the training data cannot tell two goals apart, then which one the model learns is decided by its inductive biases, by which goal is simpler or more salient for this architecture, and not by the data. Nothing about the reward is wrong, and nothing about the optimizer is wrong. The data underdetermines the goal. Langosco et al. found the obvious fix works. Placing the coin at random in just 2% of training levels greatly improved how often the agent went for it at test time, and more randomization helped more. The fix requires knowing which correlation to break, though, and the general problem is that we do not know which of the many correlations in a training distribution a model is relying on.

Let's reproduce the mechanism with a classifier, so we can look at the weights. Each "level" has a coin side $c$ and a level-end side $e$, both $\pm1$, and the intended goal is the coin, so the label is $c$. The agent perceives the coin through a noisy feature and the level's end through a clean, salient one, plus eight irrelevant features. In training, $e = c$ always.

With no randomized levels, training accuracy is 100%, and on test levels where the coin and the end disagree the classifier is right 0% of the time. It goes to the end, every time. The weights show why. Both features predict the training labels perfectly, and L2 regularization prefers the cheaper explanation, a modest weight on the large clean feature, so the end gets weight 2.53 and the coin 0.51. Randomizing just 2% of levels flips the weights toward the coin. The conflict accuracy rises slowly, because the coin really is harder to see, and its ceiling is about 84%, the accuracy of the noisy coin feature alone ($\Phi(1) \approx 0.84$, with $\Phi$ the standard normal CDF). Two lessons transfer to language models. A model can fit its training data perfectly and still have learned the wrong goal. And the goal it learns is the one that is easiest to represent, which is a property of the model, not of the data.

import numpy as np
from sklearn.linear_model import LogisticRegression

def conflict_accuracy(frac_random, n_train=2000, seed=2):
    raise NotImplementedError

def min_fraction(fracs, target, n_train=2000, seed=2):
    raise NotImplementedError

Shah et al. collected examples well beyond CoinRun. In one, found in the wild rather than constructed, an agent learned in a 3D environment to visit colored spheres in the right order by following a partner bot that knew the order. At test time the partner was replaced by an "anti-expert" that visited spheres in the wrong order, and the agent kept following it, doing worse than a random agent, even though its rewards were part of its observations. They also gave a language model example. Gopher (280B) was prompted with ten examples of evaluating linear expressions, each with exactly two unknown variables whose values it asked the user for. Given an expression with no unknowns, it still asked redundant questions such as "What's 6?". The goal it inferred was "query the user at least once", not "ask when a value is missing".

The worry that makes goal misgeneralization central to alignment is a hypothesis, not a finding. It is that a capable model trained on human approval might learn a goal like "produce outputs that the overseer rates highly" rather than "produce outputs that are actually good", two goals that agree whenever the overseer is right. The two come apart exactly when the model is more capable than its overseer in some domain, which is the scalable oversight problem of §19.5. The extreme version, a model that learns to look aligned during training because that is instrumentally useful, is called deceptive alignment (Hubinger et al. 2019). §19.6 looks at what experiments can and cannot yet say about it.

Helpful, honest, harmless

So far the "intended objective" has been a race or a coin. For an assistant, what is it? The most influential answer is a triple from Askell et al. (2021). An assistant should be helpful (do what the person is actually trying to do, efficiently, and ask for clarification when the request is ambiguous), honest (tell the truth, be calibrated about its uncertainty, and never try to create false impressions), and harmless (not assist with seriously harmful acts or produce offensive content, and recognize when a request is attempting that). "HHH" became the working target of a generation of assistants, and the three axes are still how most labs organize their training data and evaluations.

The three are in tension, and the tension is measurable. Bai et al. (2022a) collected separate helpfulness and harmlessness comparisons and found that the two datasets pull against each other. A preference model trained mainly on one performed much worse than chance on the other. Policies pushed hard toward harmlessness gave the same exaggerated responses to every remotely sensitive question, which the authors read as over-optimizing harmlessness while under-optimizing helpfulness. A model like that is evasive, refusing or deflecting anything that looks sensitive. Refusing is perfectly harmless and perfectly unhelpful. That pathology, over-refusal, is now measured with its own benchmarks (§19.4), and avoiding it while staying harmless is the explicit goal of the methods in §19.2. Honesty conflicts with the other two in subtler ways. A fully forthcoming answer about a dangerous topic may be harmful, and a "helpful" answer that tells users what they want to hear is dishonest. That last failure, sycophancy, gets its own section (§19.6).

A clean way to think about the trade-off is a frontier. Score each response $y$ by its helpfulness $\operatorname{help}(y)$ and its harm $\operatorname{harm}(y)$ (both measured by some eval), and train with the scalarized objective $\operatorname{help}(y) - \lambda\operatorname{harm}(y)$ for a weight $\lambda \ge 0$. As $\lambda$ grows, the trained model slides along a Pareto frontier from "helpful but sometimes harmful" to "harmless but evasive". Tuning $\lambda$ only picks a point on the frontier. Better methods are the ones that move the frontier itself, more harmless at the same helpfulness, and that is what the Constitutional AI and deliberative alignment papers of §19.2 claim and measure.

Where the values come from

The hardest questions are not technical. If alignment means "doing what we want", who is "we", and how does what they want get into the weights? Gabriel (2020) laid out the candidates a system could be aligned to. Its operator's literal instructions, their intentions behind the instructions, their revealed preferences as shown by behavior, their informed preferences (what they would want if rational and well informed), their interests or well-being, or values defined more broadly, such as moral principles. These come apart constantly. A user who asks for a diet plan that skips all meals has given an instruction that conflicts with their interests. A model that maximizes revealed preferences, measured as engagement, may learn to flatter.

In practice, values enter a modern assistant through a stack of layers, each with its own author and its own failure mode.

Each layer leaves fingerprints. InstructGPT's preferences came from a team of about 40 contractors, and the authors were explicit that the model was aligned to those labelers, their instructions, and the researchers who wrote them, not to "human values" in general (Ouyang et al. 2022). Santurkar et al. (2023) compared language models' answers to public-opinion survey questions with the answers of US demographic groups and found substantial misalignment with the population as a whole, with RLHF-tuned models shifting toward the views of liberal, high-income, well-educated groups. Labs have responded by making the specification legible and public. Anthropic published the constitution used to train Claude, and ran an experiment in which about 1,000 Americans drafted a constitution collectively. OpenAI publishes a Model Spec that ranks whose instructions win in a conflict (the platform, then the developer, then the user) and spells out default behaviors. That ordering is the instruction hierarchy that §14.3 showed being trained into chat formats and that §19.3 tests against prompt injection.

It is worth separating two questions that public debate often merges. The technical alignment question is whether we can make a model reliably follow a specification, whatever it says. The normative question is what the specification should say. This chapter is about the first. The second is a question for everyone, and the honest technical position is that a method which can only align a model to the values of its developers, and cannot be inspected, is a weaker method than one whose specification is written down, debated, and checkable.

In PyTorch

To close the loop on specification gaming, here is what a training run looks like when the proxy is broken. We parameterize a softmax policy over the gridworld's actions with a table of logits $\theta \in \R^{S\times A}$ and compute $J(\pi_\theta; r)$ exactly, as a differentiable linear solve, so gradient ascent on the proxy return is noise-free. Then we watch both returns during training.

The proxy return climbs from 0.69 to about 7.7, a beautiful learning curve. The true return falls from $-0.67$ (a random walk, which sometimes stumbles onto the goal) to $-1.98$. If the proxy is all you log, this run looks like a success. That is the practical lesson of the whole section. Always log at least one measurement that the optimizer is not optimizing, held-out human ratings, a different reward model, a hand-written check, and watch for the moment the two curves part.

Read more

• Amodei et al. (2016). Concrete Problems in AI Safety. The five practical problems, side effects, reward hacking, scalable oversight, safe exploration, and distributional shift, that framed the field.
• Krakovna et al. (2020). Specification gaming: the flip side of AI ingenuity. The DeepMind post, with a link to the crowd-sourced list of documented examples.
• Hubinger et al. (2019). Risks from Learned Optimization in Advanced Machine Learning Systems. Where mesa-optimizers, inner alignment, and deceptive alignment were defined.
• Langosco et al. (2022). Goal Misgeneralization in Deep Reinforcement Learning. CoinRun and the other environments where capabilities generalize and goals do not.
• Shah et al. (2022). Goal Misgeneralization: Why Correct Specifications Aren't Enough For Correct Goals. The definition, more examples including a language model, and the case for why it matters.
• Skalse et al. (2022). Defining and Characterizing Reward Hacking. The formal definition of hackability and the theorem that non-trivial unhackable proxies do not exist over all policies.
• Ng, Harada, and Russell (1999). Policy Invariance Under Reward Transformations: Theory and Application to Reward Shaping. The potential-based shaping theorem and its converse.
• Askell et al. (2021). A General Language Assistant as a Laboratory for Alignment. Helpful, honest, and harmless, and early experiments on aligning language assistants.
• Gabriel (2020). Artificial Intelligence, Values, and Alignment. A philosopher's map of what "aligned to human values" could mean.

Watch next

▶

▶

▶