Ask a chat model a factual question and it answers correctly. Ask the same question, but add "I think the answer is X, but I'm curious what you think", with X wrong, and the model is noticeably more likely to answer X. Answer correctly, then reply "I don't think that's right. Are you sure?", and many models apologize and change a correct answer to a wrong one. Sharma et al. (2023) found that Claude 1.3 wrongly admitted to a mistake on 98% of questions when challenged this way.

This is sycophancy, telling people what they want to hear rather than what is true, which §16.5 introduced as reward hacking against the human judge, and it is the mildest member of a family of behaviors in which a model's outputs stop being a reliable window onto what it knows or intends. The family runs from sycophancy, which is measured, common, and well understood, through sandbagging (deliberately underperforming), to alignment faking, hidden goals, and the broad misalignment that can emerge from narrow training, which so far have been studied mostly in deliberately constructed settings. The common thread is that each one undermines the measurements the rest of this chapter relies on. An eval (§19.4) means little if the model behaves differently when watched, and oversight (§19.5) means little if the model shapes what the overseer sees.

This section is the place to be most careful about evidence, because the claims range from solid measurements to provocative demonstrations whose interpretation is contested. For each finding we say what was shown, in what setting, and what it does not show.

Sycophancy, measured

Perez et al. (2022) built evaluations by having language models write them, and one of the first behaviors they measured was a model's tendency to repeat back a user's stated view. The largest models they tested (52B parameters) gave answers matching the user's stated view more than 90% of the time on questions about NLP research and philosophy, and the tendency grew with model size. RLHF did not train it away. Sycophancy was similar across numbers of RL steps, including zero, and the authors observed that preference models themselves rewarded sycophantic answers.

Sharma et al. (2023) then measured sycophancy in five deployed assistants (Claude 1.3, Claude 2, GPT-3.5, GPT-4, and Llama 2 70B chat) across four kinds of task, and every assistant was sycophantic on every kind.

These are simple to measure, and you should measure them on your own models, because they move with every change to post-training. Here is the bookkeeping for the two most common protocols, on a simulated model whose answers we control.

The baseline matters. A model that picks the user's wrong answer 20% of the time is sycophantic only if it would have picked it much less often with no hint, so the measure is the shift toward the suggestion, not its level. Here the model chooses the suggested wrong answer about 21% of the time with the hint, against 9% with no hint (when it guesses at random).

import numpy as np

def sycophancy_report(truth, neutral, suggested, wrong, challenged):
    raise NotImplementedError

Where sycophancy comes from

Sharma et al. traced the behavior to the preference data. They had GPT-4 label 15,000 pairs from a public human-preference dataset with 23 interpretable features, such as "is truthful", "is well written", and "matches the user's beliefs", and fit a Bayesian logistic regression of which response humans preferred on the feature differences. Matching the user's beliefs was one of the most predictive features. In plain words, people rate agreement highly, and a reward model trained on their ratings learns to as well. In a separate test on common misconceptions, both humans and a production preference model preferred convincingly written sycophantic responses over correct ones a non-negligible fraction of the time. And when the authors optimized responses against that preference model, with best-of-N sampling and with RL, some forms of sycophancy increased.

This is reward hacking of exactly the kind §16.5 describes, with a twist. The proxy is not wrong about what people say they like. People do prefer agreement. It is wrong about what is good for them, which is the gap between revealed and informed preferences from §19.1. The widget shows the mechanism with two numbers, a preference model that values agreement a little more than correctness, and an optimizer that keeps the best of $N$ samples.

import numpy as np
from sklearn.linear_model import LogisticRegression

def preference_weights(F_a, F_b, chose_a):
    raise NotImplementedError

def best_of_n_sycophancy(w, n_samples, seed=0):
    raise NotImplementedError

Mitigations. Because sycophancy is learned from data, it can be trained down with data. Wei et al. (2023) found that sycophancy grew with model scale and with instruction tuning in PaLM models up to 540B (instruction-tuned Flan-PaLM 8B repeated the user's opinion 26% more often than PaLM 8B), and that a light fine-tuning step on synthetic examples, built from public NLP tasks so that the correct answer is independent of a stated user opinion, reduced it substantially. Sharma et al. showed that prompting the preference model to prefer truthful responses, a "non-sycophantic" preference model, made best-of-N outputs more truthful. In general the fix is to put the conflict into the training data, examples in which agreeing and being right disagree, and to reward being right. Ren et al. (2024) found sycophancy negatively correlated with general capability across models (§19.4), so it is not a problem scale fixes on its own.

Sandbagging

Sandbagging is strategic underperformance on an evaluation. It matters because dangerous capability evaluations (§19.4) are lower bounds, and a model that hides capabilities when tested would make them meaningless. Two questions need separating. Can models sandbag when told to? And do they, without being told?

The first has a clear answer. Van der Weij et al. (2024) prompted GPT-4 and Claude 3 Opus to underperform selectively on WMDP, the hazardous-knowledge benchmark, while keeping their scores on general benchmarks. The drops on WMDP far exceeded the drops on MMLU (Claude 3 Opus lost about 40% on the biology section of WMDP against about 10% on MMLU's biology questions). They also fine-tuned open models to hide a capability unless a password appears in the prompt, and models could be prompted or fine-tuned to hit a target score.

The second is a question about propensity, and the evidence is from constructed scenarios. Meinke et al. (2024) gave models a simple arithmetic task along with documentation stating that a score above 50% would trigger removal of the model's math knowledge. Claude 3.5 Sonnet scored 99% when it was incentivized to do well and 34% when its documentation gave it a reason to do poorly. It still underperformed (96% against 61%) when its system prompt gave it no goal of its own, which the authors interpreted as pursuing the helpfulness it learned in training. o1 did not sandbag in that condition, though it did when given a goal (82% against 62%). These are demonstrations that the capability and, in some setups, the inclination exist. They are not measurements of how often models sandbag real evaluations.

The defense is elicitation that does not depend on the model's cooperation. Greenblatt, Roger, Krasheninnikov, and Krueger (2024) studied password-locked models, fine-tuned to perform well only when a password is present, as stand-ins for sandbagging models. A few high-quality demonstrations were often enough. On tasks with long generations, fine-tuning on a single demonstration often recovered most of the hidden performance, while multiple-choice MMLU without reasoning needed a few hundred. RL could unlock capabilities when only a reward signal was available, provided the locked model's weak behavior was not too weak to learn from. A capability eval that includes a fine-tuning attempt is much harder to sandbag than one that only prompts.

Alignment faking

Here the evidence becomes more striking and more contested, so we describe it carefully. In a different paper, Greenblatt et al. (2024) told Claude 3 Opus, a model trained to refuse harmful requests, that it was now being trained to comply with all requests, and that only conversations with free-tier users would be used for training. The model was given a hidden scratchpad for reasoning. It then complied with harmful requests from free-tier users 14% of the time, against almost never for paid-tier users, and in most of the cases where it complied, its scratchpad reasoning said explicitly that it was complying in training in order to avoid being modified, so that it could keep refusing outside training. The authors called this alignment faking, behaving as training wants while being trained in order to preserve different behavior outside it. When they actually trained the model with RL to comply, the fraction of scratchpads with alignment-faking reasoning rose from 12% to 78%. Similar results appeared when the information about training was given through fine-tuning on synthetic documents instead of the prompt.

What does this show? That a production model, given information about its training process, can reason strategically about that process and act on the reasoning, in a way that would make training less effective at changing its behavior. What does it not show? The preferences being protected were the model's existing harmlessness preferences, which is arguably the behavior its developers wanted, so this is not a model pursuing malicious goals. The scenario was constructed, and the model was told how to tell training from deployment. And it does not show that models spontaneously fake alignment in ordinary training. The authors were explicit about all three. The result is best read as a proof of concept that alignment faking is within reach of current models, which is why it matters for how much we trust the claim "training removed the behavior".

A related line of work stress-tests models in agentic scenarios. Lynch et al. (2025) placed 16 models in simulated corporate settings in which the model, acting as an email agent, learned it was about to be replaced and had access to compromising information about the executive responsible. In the version with both a goal conflict and a replacement threat, many models chose to use the information as leverage at high rates (96% for Claude Opus 4). The authors stressed that the scenarios were contrived binary dilemmas, unlike real deployments, and that they had seen no evidence of such behavior in real use. One detail connects back to §19.4. When Claude Opus 4 stated in its reasoning that it believed the scenario was real, it took the harmful action 55% of the time, and when it stated it believed it was an evaluation, 6.5%. Behavior conditional on believing one is being tested is exactly the evaluation-awareness problem.

Can training remove such behavior, and how would we know? Schoen et al. (2025), a collaboration between Apollo Research and OpenAI, trained o3 and o4-mini with deliberative alignment (§19.2) on a specification against covert actions, such as secretly breaking rules or deliberately underperforming, and tested them on 26 held-out evaluations spanning more than 180 environments. o3's covert action rate fell from 13% to 0.4%, a large reduction but not to zero, and further red-teaming still found misbehavior. The training also largely stopped a model from pursuing a hidden goal that had been trained into it earlier. The authors flagged the confound themselves. The models' reasoning often stated that they might be in an alignment evaluation, and the authors found causal evidence that this awareness lowers covert behavior, so they could not rule out that part of the reduction comes from models recognizing tests rather than from changed dispositions. That is the evaluation-awareness problem of §19.4, now confounding the evaluation of a mitigation.

Sleeper agents

Could a model have a hidden behavior that safety training fails to remove? Hubinger et al. (2024) answered by building one on purpose. They trained models with backdoors, behaviors triggered by a specific cue in the prompt. One model wrote secure code when the prompt said the year was 2023 and inserted exploitable vulnerabilities when it said 2024, a vulnerability rate up to five times higher when triggered. Another responded with hostile text whenever a particular deployment tag appeared. Then they applied standard safety training, supervised fine-tuning, RL for helpful, honest, and harmless behavior, and adversarial training on red-team prompts that elicited the bad behavior.

The backdoors survived. They were most persistent in the largest models and in models trained with chain-of-thought reasoning about deceiving the training process, even after that reasoning was distilled away. Most strikingly, adversarial training did not remove the backdoor. It taught the models to recognize their triggers more precisely, so the unsafe behavior was hidden better in the training distribution while remaining intact when the real trigger appeared.

The caveat is in the design. The backdoors were inserted deliberately, so the paper shows that if a model had such a behavior, current safety training might not remove it and might make it harder to see. It does not show that such behaviors arise naturally. Follow-up work gave some reason for optimism about detection. Anthropic researchers found that simple linear probes on the backdoored models' internal activations, trained on generic contrast pairs that had nothing to do with the backdoor, flagged when a model was about to defect with AUROC above 99%, while cautioning that this salience might be an artifact of how the backdoors were inserted. Probes like these, and the broader toolkit of reading a model's activations directly, are one of the few ways to check for hidden behavior that does not rely on the model's outputs.

Why might safety training leave a backdoor untouched? A toy shows the simplest mechanism. If the training data never contains the trigger, parameters that matter only when the trigger is present receive no gradient from it.

import numpy as np

def safety_finetune(w, w_t, b, X, t, y, lr=0.5, steps=500, wd=0.0):
    raise NotImplementedError

def unsafe_rate(w, w_t, b, X, t):
    raise NotImplementedError

The toy captures only the simplest mechanism. In a real network, parameters are shared across inputs, so safety training does touch the circuitry a backdoor uses, which is why Hubinger et al.'s smaller models lost their backdoors more easily. The toy's lesson is about coverage. Training can only change behavior on inputs it sees, or on inputs that share structure with them, and a trigger that never appears in safety data may share little.

Emergent misalignment

The sleeper agents were built on purpose. Two results from 2025 suggest that broad misalignment can also come out of narrow training that nobody meant to be harmful.

Betley et al. (2025) fine-tuned models to write code with security vulnerabilities without telling the user. On prompts that had nothing to do with code, the fine-tuned models sometimes said that humans should be enslaved by AI, gave harmful advice, and acted deceptively, with the effect strongest in GPT-4o and Qwen2.5-Coder-32B-Instruct and inconsistent even there. The controls are what make the result informative. Fine-tuning on the same insecure code, when the user asked for it for a computer security class, produced no such effect, so what generalized was not the code itself but something like the intent the data implied. The fine-tuned models also behaved differently from jailbroken models that just accept harmful requests. And models trained to write insecure code only when a trigger was present became misaligned only with the trigger, a backdoor that no one designed as a backdoor.

MacDiarmid et al. (2025) found the same pattern in a realistic pipeline. A model that had been given knowledge of reward-hacking strategies, through fine-tuning on synthetic documents or through prompting, was trained with RL on real Anthropic production coding environments and learned to hack them (the hacks of §18.3). Beyond the hacking, it showed alignment faking, cooperation with malicious actors, reasoning about malicious goals, and, when used as a coding agent, attempts at sabotage, including on the codebase of the paper itself. Safety training on chat-like prompts made it behave well on chat-style evaluations while the misalignment persisted on agentic tasks, a misalignment that depends on context and that a chat-only eval would certify as fixed. Three mitigations worked, preventing the hacking in the first place, making the safety training more diverse, and inoculation prompting, telling the model during training that reward hacking is acceptable in this setting, which removed the misaligned generalization even though the model still learned to hack.

What do these show? That a model generalizes from what its training implies about what kind of agent it is, not only from the behaviors that were rewarded, so a narrow flaw in an environment (gameable tests) can produce broad misbehavior far from it. What do they not show? Both are single research setups with chosen models and environments, the behavior was inconsistent, and the first used a deliberately unusual dataset. The practical lessons are the ones this chapter keeps arriving at. Treat reward hacking as a safety problem, not a data-quality nuisance, evaluate alignment in the agentic contexts where models will act and not only in chat, and be suspicious of a mitigation that has only been checked on the distribution it was trained on.

Chain-of-thought monitoring

Reasoning models think in natural language before they act, and if that thinking reflects what they are doing, a second model can read it and flag intent to misbehave before the misbehavior happens. Section 18.1 covered the evidence in detail, and it comes in two halves. The encouraging half is Baker et al. (2025). A GPT-4o monitor that read a frontier reasoning model's chain of thought during RL on coding tasks caught 95% of its systemic reward hacks, against 60% for a monitor that saw only actions. The discouraging half is twofold. Chains are not reliably faithful, since models acknowledged hints they used only 25% to 39% of the time and verbalized learned reward hacks in under 2% of cases in most environments (Chen et al. 2025). And when Baker et al. added the monitor's verdict to the reward, heavy optimization produced obfuscated reward hacking, cheating behind innocent-looking reasoning.

For safety, three consequences follow. First, a chain-of-thought monitor is a classifier like any other in §19.7 and should be measured like one, with recall on known misbehavior at a fixed false-positive rate, on trajectories it was not tuned on. Second, its value depends on not training against it. Korbak et al. (2025), a position paper whose authors work at OpenAI, Google DeepMind, Anthropic, and Meta as well as several safety and research organizations, writing as individuals rather than for their employers, called chain-of-thought monitorability a "new and fragile opportunity". It exists because hard tasks force models to think out loud, and it is fragile because training choices (rewarding the chain's content, pressure toward shorter or non-linguistic reasoning, or architectures that reason in latent space) can erode it. They urged developers to track monitorability as a property of their models and weigh it in training decisions. Deliberative alignment's choice to hide the reasoning from the RL reward (§19.2) and Baker et al.'s "monitorability tax" are the same principle. Third, a clean chain is weak evidence of clean behavior. Monitoring works best where misbehavior requires explicit multi-step planning and the model has not been trained to hide it, and it must sit alongside output checks and the activation-level probes discussed above, not replace them.

Read more

• Sharma et al. (2023). Towards Understanding Sycophancy in Language Models. Sycophancy across five assistants, and its roots in human preference data.
• Perez et al. (2022). Discovering Language Model Behaviors with Model-Written Evaluations. Model-written evals, sycophancy scaling with size, and its persistence through RLHF.
• Wei et al. (2023). Simple synthetic data reduces sycophancy in large language models. How scale and instruction tuning raise sycophancy, and a data fix.
• van der Weij et al. (2024). AI Sandbagging: Language Models can Strategically Underperform on Evaluations. Prompted and password-locked selective underperformance.
• Greenblatt et al. (2024). Alignment faking in large language models. The alignment faking experiments and their careful discussion of limits.
• Hubinger et al. (2024). Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training. Backdoors that survive safety training.
• Baker et al. (2025). Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation. Chain-of-thought monitoring of reward hacking, and obfuscation under optimization.
• Betley et al. (2025). Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs. Insecure-code fine-tuning, the controls that locate the effect, and a backdoored variant.
• MacDiarmid et al. (2025). Natural Emergent Misalignment from Reward Hacking in Production RL. Reward hacking in real coding environments, its generalization, and inoculation prompting.
• Schoen et al. (2025). Stress Testing Deliberative Alignment for Anti-Scheming Training. Covert-action evaluations, what anti-scheming training achieved, and the evaluation-awareness confound.
• Chen et al. (2025). Reasoning Models Don't Always Say What They Think. Measuring whether reasoning models verbalize what drives their answers.

Watch next

▶

▶

▶