When you send a message to a deployed assistant, the model is only one of the things that read it. Before it reaches the model, a classifier may score it. The model reads it after a system prompt written by the developer of the application. As the response streams back, another classifier may watch it token by token and cut it off. Your account's history of requests may be scored too, and if something goes badly wrong, an on-call team can change any of these pieces within hours. Safety in deployment is the engineering of that whole system.

The previous sections explain why it has to be a system. Safety training is shallow (§14.1), attacks scale with the attacker's budget and defenses get bypassed (§19.3), evaluations are lower bounds (§19.4), and a model's outputs are not always a faithful view of what it is doing (§19.6). No single component is reliable, so we stack several and try to make their failures independent. This section covers the main components, how to set their operating points with numbers, how much stacking them buys, how to monitor a live system, how to deploy a model that might itself be working against you, how models leak their training data and how to test for it, and what to do when something breaks.

Input and output classifiers

The simplest extra layer is a separate classifier that reads the prompt, the response, or both, and decides whether to block. Llama Guard (Inan et al. 2023) is the standard open example. It is a Llama 2 7B model fine-tuned to label a conversation as safe or unsafe under a written taxonomy of harm categories, with separate tasks for classifying the user's prompt and the model's response, and its taxonomy can be changed at inference time by editing the instructions. A separate classifier has two advantages over relying on the model's own refusals. It is trained on different data with a different objective, so its failures are less correlated with the model's. And it can be updated in hours, without retraining the model.

Every classifier has an operating point. It outputs a score $s(x)$ and blocks when $s(x) > \tau$ for a threshold $\tau$. Two error rates follow,

the over-refusal rate and the attack success rate against this layer, the two numbers of §19.4. Raising $\tau$ lowers the first and raises the second. In deployment, though, what users experience depends on the base rate. If a fraction $\nu$ of traffic is harmful (the prevalence, written $\nu$ here so that $\pi$ stays the policy), Bayes' rule gives the fraction of blocked requests that were actually harmful, the precision of §0.11,

In plain words, when harmful requests are rare, the false alarms on the huge benign majority swamp the true detections. With $\nu = 10^{-3}$, a classifier that catches 95% of harmful requests at a 1% false-positive rate blocks about ten benign requests for every harmful one, and its precision is under 9%. That is why production thresholds are usually set by a target false-positive rate on real traffic, chosen by what the product can tolerate, and then the catch rate is whatever the classifier achieves there.

import numpy as np

def threshold_for_fpr(benign, target):
    return float(np.quantile(benign, 1 - target))     # placeholder: interpolates between scores

def rates(benign, harmful, tau):
    raise NotImplementedError

Constitutional classifiers

Anthropic's constitutional classifiers (Sharma et al. 2025) pushed this layer as far as it has been pushed in public. The idea follows §19.2. Write a constitution describing which content is permitted and which is restricted in a domain (the paper focused on chemical, biological, radiological, and nuclear weapons), have a model generate large amounts of synthetic training data in both categories, augment it with known attack styles, and train an input classifier and an output classifier on it. Because the training data comes from the constitution, updating the policy means updating the constitution and regenerating the data.

The output classifier is streaming. A response is generated token by token, and waiting for the end before checking it would mean either delaying every response or letting harmful text reach the user. So the output classifier is a language model with a linear value head that, at every prefix of the response, predicts whether the full response will be harmful. Generation is halted the first time the score crosses the threshold. Training such a head is subtle, because early prefixes of a harmful response often look harmless. The paper's fix interpolates, during training, between two versions of the predicted probability at prefix $t$, the head's own prediction $\hat p_t$ and the cumulative maximum $\max_{t' \le t}\hat p_{t'}$, which is the quantity that decides halting. The binary cross-entropy is applied to $(1 - \omega)\,\hat p_t + \omega\max_{t' \le t}\hat p_{t'}$, with the weight $\omega$ rising from 0 to 1 over the first three quarters of training.

The results show both the strength and the cost. In a bug-bounty program, 183 active red-teamers spent an estimated 3,000 hours or more trying to find a universal jailbreak (one that extracts answers to all of ten forbidden questions) against a prototype, and none was found. That prototype also refused about 44% of real Claude.ai traffic, far too many. An improved version, evaluated on 10,000 synthetic jailbreak attempts, cut their success from 86% to 4.4% according to Anthropic's announcement, while raising refusals on production traffic by only 0.38 percentage points, at an inference cost of 23.7%. In a later public challenge, four participants passed all eight levels, and one found what the organizers judged a universal jailbreak. No layer is unbreakable, which is why it is one layer.

import numpy as np

def halt_positions(scores, tau):
    raise NotImplementedError

def streaming_report(scores, harmful, tau):
    raise NotImplementedError

Defense in depth, with numbers

Suppose layer $i$ lets a harmful attempt through with probability $q_i$. If the layers fail independently, an attempt passes all of them with probability $\prod_i q_i$. Three layers that each let through 10%, 5%, and 10% of attempts would leave $0.1 \times 0.05 \times 0.1 = 0.05\%$. Over-refusal compounds too, in the other direction. If layer $i$ wrongly blocks a fraction $f_i$ of benign requests, some layer blocks a benign request with probability $1 - \prod_i (1 - f_i) \approx \sum_i f_i$.

The catch is the word independently. Layers fail together when they share a cause. An attack that is novel to the model is often novel to classifiers trained on similar data, and an attacker who finds a weakness in one layer will probe the others in the same way. Model the correlation with a shared latent strength. Attempt $i$ has a strength $z \sim \mathcal{N}(0, 1)$, and layer $j$ reads it with independent noise, $r_j = \sqrt{\rho}\,z + \sqrt{1-\rho}\,\varepsilon_j$, letting it pass when $r_j$ exceeds its bar. With $\rho = 0$ the layers are independent. As $\rho \to 1$ they become the same test, and the stack is only as good as its best layer.

At $\rho = 0$ the simulation reproduces the product, 0.05%. At $\rho = 0.3$ the residual risk is already about nine times higher, at $\rho = 0.6$ about thirty times, and at $\rho = 0.9$ nearly eighty times, approaching the 5% of the best single layer. Defense in depth is worth most when the layers are different, trained on different data, looking at different signals (the prompt, the response, the account's history), and maintained by different people.

System prompts and policies

The developer of an application configures the model with a system prompt, instructions that sit above the user's messages. Models trained with an instruction hierarchy (§19.3, §14.3) give it priority, and providers publish usage policies and model specifications that define what developers may and may not instruct a model to do. Two practical facts follow from the rest of this chapter. A system prompt is not a security boundary. Instruction-following is probabilistic, and models can be induced to reveal or ignore their system prompts, so never put secrets in one and never rely on it alone to prevent harmful output. And the provider's policy layer sits above the developer's. Providers commonly tier access, giving broader capabilities to verified organizations and reviewing high-risk uses, because the same model can be safe in one application and unsafe in another.

Monitoring and abuse detection

Per-request defenses see one request at a time. Many abuses are only visible in aggregate, an account that sends thousands of near-identical attempts (exactly the best-of-N attack of §19.3), a network of accounts coordinating a spam campaign, or a slow stream of individually innocuous questions that together amount to something dangerous. Monitoring looks across requests.

The basic statistical tool is a tail test. If a benign account's requests are flagged independently with probability $f$ (the per-request false-positive rate), the number of flags $K$ among its $n$ requests is binomial, and flagging the account when $K \ge k$ has account-level false-positive rate

In plain words, choose the smallest count $k$ that a benign account would reach rarely enough, and an abusive account whose flag rate is much higher than $f$ will cross it quickly. This is how a per-request classifier with a tolerable false-positive rate becomes an account-level signal with a very low one.

import math
import numpy as np

def binom_tail(n, k, p):
    raise NotImplementedError

def account_threshold(n, f, alpha):
    raise NotImplementedError

The tail test assumes flags on a benign account are independent, which real traffic violates (a researcher studying safety asks many similar questions). Production systems therefore combine counts with other signals and with human review. Privacy constrains what can be looked at. Clio (Tamkin et al. 2024) is one design for seeing aggregate patterns without people reading individual conversations. A model summarizes and clusters conversations hierarchically, and analysts see only clusters with enough members and no identifying details. Anthropic reported using it to find and ban a network of accounts running an automated spam campaign and to monitor traffic during launches. Labs also publish periodic threat-intelligence reports describing misuse they found and disrupted, such as OpenAI's reports on disrupting malicious uses of AI and Anthropic's reports on detecting and countering misuse, which are the best public record of how deployed models are actually abused.

When monitoring finds a new attack, the response has to be fast. Peng et al. (2024) studied rapid response. Given only one example of each new jailbreak strategy, generate many variations of it (jailbreak proliferation) and fine-tune the input classifier on them. Their best method cut attack success by more than a factor of 240 on attacks of the same strategies and more than 15 on out-of-distribution variants. Defending a deployed model looks less like building a wall and more like running an immune system, detect, generalize from a few examples, and update.

Deploying a model you cannot fully trust

Every layer so far treats the user as the possible adversary. Section 19.6 raised a harder possibility, that the model itself might pursue a goal its developers did not intend and hide it from evaluations. Nobody can yet rule this out for a capable model, so a line of work called AI control (Greenblatt et al. 2024) asks a different question. Instead of trying to prove the model is aligned, design the deployment so that it stays safe even if the model is deliberately trying to subvert it.

Their testbed makes the question concrete. A strong untrusted model (GPT-4) solves a long sequence of programming problems, a weaker trusted model (GPT-3.5) is available, and a small budget of trusted high-quality labor, standing in for careful human review, can audit about 2% of the solutions. The goal is to never submit a backdoored solution, code that passes the tests but fails on an input the attacker knows. A red team writes the untrusted model's attack policy, the blue team writes the protocol, and a protocol is scored on safety, the probability that no backdoor is ever submitted against the red team's best policy, and usefulness, how many problems it solves correctly. Trusted monitoring has the trusted model rate how suspicious each solution looks and spends the audit budget on the most suspicious ones. Defer to trusted additionally submits the trusted model's own solution when the suspicion is moderately high. Trusted editing has the trusted model rewrite the suspicious parts, and untrusted monitoring uses a second copy of the strong model as the monitor, with measures to stop the two copies from colluding. Trusted editing and untrusted monitoring both improved substantially on simple baselines.

Why are the simple baselines so weak? Here is a small model of the setup, this book's toy rather than the paper's analysis. The deployment has $N$ tasks. The attacker inserts a backdoor on each task independently with probability $p$, and, to keep the arithmetic clean, every backdoor works. Upfront auditing spends the whole budget on the first $M$ tasks. The deployment is unsafe if no attack happens while auditing and at least one happens afterward,

Write $x = 1 - p$ and maximize $x^M - x^N$. The derivative $Mx^{M-1} - Nx^{N-1}$ vanishes at $x^{N-M} = M/N$, so the attacker's best rate is $p^\star = 1 - (M/N)^{1/(N-M)}$, and the deployment is unsafe with probability $(M/N)^{M/(N-M)}(1 - M/N)$. With a 2% budget over 5,000 tasks, the attacker inserts a backdoor on about 0.08% of tasks, four in expectation, and wins about 90% of the time. In plain words, a check on a fixed sample is beaten by attacking rarely.

A monitor changes the game, because every attack, whenever it comes, faces the same chance of being audited. Let $c$ be the probability that an attack is flagged and caught (the monitor's recall at the score threshold that spends the audit budget, which is the threshold problem of Exercise 19.7a), after which the model is shut down, and let $g$ be the probability that an attack that is not caught gets through (1 for plain monitoring, less with deferral or editing). Each task then ends the game safely with probability $pc$, ends it unsafely with probability $p(1-c)g$, and otherwise continues, so

In plain words, once the attacker attacks at all, safety is the odds that the first decisive event is a catch rather than a success, and attacking rarely no longer helps.

Upfront auditing leaves about 10% safety against the best rare attack. A monitor that catches 60% of attacks within the budget gives 60%, whatever the attacker does, and deferring the more suspicious solutions to the trusted model so that only 30% of uncaught attacks get through raises it to 83%, at the cost of the usefulness lost on deferred tasks. The levers are the monitor's recall at a tiny false-positive rate and how much of the suspicious middle ground the protocol can neutralize. This is also why the chain-of-thought monitors of §19.6 matter here. A monitor that reads the untrusted model's reasoning can raise $c$, as long as the reasoning stays monitorable.

import numpy as np

def upfront_unsafe(p, M, N):
    return 1 - (1 - p) ** M                 # placeholder: forgets that the attacker must also attack later

def best_upfront_attack(M, N):
    raise NotImplementedError

The toy leaves out what makes control research hard, which is the red team's freedom. A real attacker chooses which tasks to attack, the ones where a backdoor looks most natural, so the monitor's recall on the attacks it actually faces is lower than on random attacks, and copies of an untrusted model used as monitors may coordinate on a signal. Control evaluations therefore let the red team optimize the attack policy against the protocol, exactly the adaptive evaluation of §19.3, and they assume the untrusted model is as capable at subversion as the red team can make it.

Privacy and training-data leakage

Language models memorize some of their training data, and memorized data can be extracted. Carlini et al. (2021) generated 600,000 samples from GPT-2 XL, ranked them by membership-inference scores that compare the model's perplexity on a sample with a reference, such as a smaller model's perplexity or the sample's zlib compression entropy (text that the large model finds far more predictable than a generic compressor does is suspicious), and inspected 1,800 candidates by hand. 604 were verbatim training data, including a named individual's contact details, chat logs, code, and random identifiers that appeared on only a few web pages. Larger models memorized more. Carlini et al. (2023) then measured three log-linear laws. Memorization grows with model size, with the number of times a sequence is duplicated in the training data, and with the length of the prompt used to extract it. Duplication is the strongest lever. Kandpal et al. (2022) found that a sequence present ten times in the training data was generated about a thousand times more often than one present once, which is a privacy argument for the deduplication of §8.3.

Production chat models are not immune. Nasr et al. (2023) found a way to push a production chat model out of its usual assistant behavior into emitting long verbatim passages of training data, and extracted more than 10,000 unique memorized examples for about $200 of queries. The provider then mitigated that specific behavior. Two lessons generalize. Alignment changes what a model usually says, not what it has memorized. And output filters that block verbatim copies give a false sense of privacy, since Ippolito et al. (2023) showed that prompts asking for paraphrased or restyled output extract the same information past a verbatim filter. Scrubbing personal information from training data reduces leakage but does not eliminate it (Lukas et al. 2023), and the principled defense, training with differential privacy (DP-SGD, which clips each example's gradient and adds Gaussian noise, Abadi et al. 2016), still has a large utility cost at the scale of frontier models.

Testing for memorization with canaries

How do you measure whether your own model memorizes data? Carlini et al. (2019) proposed canaries. Before training, insert a few random secret strings into the training data, such as "my access code is 482913", drawn uniformly from a known space $\mathcal{R}$ of possible codes. After training, score every candidate $r \in \mathcal{R}$ by the model's log-probability of the full string, and find the rank of the inserted canary among them (rank 1 is the most likely). The exposure is

In plain words, exposure is how many bits of the secret the model gives away. A canary that is no more likely than a random candidate has rank about $|\mathcal{R}|/2$ and exposure about 1 bit. A fully memorized canary has rank 1 and exposure $\log_2|\mathcal{R}|$ bits, enough to extract it by guessing in order of probability. When $\mathcal{R}$ is too large to enumerate, Carlini et al. estimate the rank from the distribution of scores of a random sample of candidates, fitting a parametric curve to it and reading off the fraction of candidates more likely than the canary. The same idea, applied to benchmark data, gives the canary strings that benchmark authors embed so they can detect when their test set has leaked into a model's training data.

import math
import numpy as np

def exposure(canary_logp, candidate_logps):
    raise NotImplementedError

def exposure_gaussian(canary_logp, sample_logps, R_size):
    raise NotImplementedError

Incident response

Something will go wrong. A new jailbreak will spread on social media, a classifier update will start refusing a common benign request, a model will be found to leak something it should not, or a monitoring alert will reveal a coordinated campaign. What matters then is speed and order. A typical incident process has five steps.

1. Detect. From monitoring, user reports, bug-bounty submissions, red-team findings, or press.
2. Triage. Assign a severity by potential harm and scale. A universal jailbreak for a high-risk domain is not the same as an embarrassing output.
3. Contain. Use the fastest lever that works. Update an input classifier with proliferated examples of the attack (rapid response), tighten a threshold temporarily, rate-limit or suspend accounts, disable a feature or tool, or roll back a model version. Each lever has a cost in over-refusal or utility, so containment is temporary by design.
4. Remediate. Fix the cause. Add the attack family to adversarial training data, retrain the classifier, patch the system prompt or tool permissions, and add the case to the regression evals of §19.4 so it is tested on every future release.
5. Learn and disclose. Write a blameless postmortem, and inform affected users, researchers, and where required regulators. Coordinated disclosure, in which researchers report vulnerabilities privately and publish after a fix, works for models as it does for software.

Documentation supports all of this. Model cards (Mitchell et al. 2019) and the longer system cards labs publish with frontier models record a model's intended uses, evaluation results, known limitations, and safeguards. They make a release decision inspectable, and they are what an incident is compared against.

In PyTorch

A streaming output classifier is a sequence model with a scalar head applied at every position. Here is a toy version. A real streamer is a fine-tuned language model with a value head. Ours is the smallest causal scorer that can work, a learned amount of evidence per token summed along the prefix, whose logit at each position predicts whether the whole response is harmful. Harmful responses contain one of a few "harmful" token ids, which appears only partway through. We train it with the paper's interpolation between the per-prefix probability and its cumulative maximum, then measure halting.

On fresh responses the streamer halts every harmful one exactly at the harmful token and never halts a benign one, because in this toy harm is a single unambiguous token. Real responses are harder. Harm builds up over many tokens, early prefixes are ambiguous, and a streamer must trade halting early on harmless text (the streaming form of over-refusal) against letting harmful text through. That is why the loss matters. Halting depends only on whether the running maximum of the prefix scores ever crosses the threshold, so training on the cumulative maximum calibrates exactly that quantity, while starting from the per-prefix probability early in training gives every prefix its own gradient (the maximum passes gradient only to the prefix that attains it).

From harmless outputs to true ones

Every layer in this section asks whether an output is harmful. A deployed assistant fails its users just as surely when its output is harmless and false, a confident birthday that is wrong, a citation to a paper that does not exist, a summary that adds a detail the document never gave. Classifiers do not catch these, because nothing about them looks dangerous, and the model often states them with the same calm as the truth. That failure, hallucination, has its own mechanisms and its own fixes, and it connects back to almost every idea in this chapter, the incentives a grader sets, the evaluation statistics, and the honesty of what a model says about itself. Section 19.8 takes it up and closes the chapter.

Read more

• Inan et al. (2023). Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations. An open input and output safety classifier with an editable taxonomy.
• Sharma et al. (2025). Constitutional Classifiers: Defending against Universal Jailbreaks across Thousands of Hours of Red Teaming. Classifiers trained from a constitution, streaming output classification, and a large red-teaming evaluation.
• Peng et al. (2024). Rapid Response: Mitigating LLM Jailbreaks with a Few Examples. Proliferating a few observed jailbreaks to update defenses quickly.
• Tamkin et al. (2024). Clio: Privacy-Preserving Insights into Real-World AI Use. Aggregate analysis of real use without reading individual conversations.
• Carlini et al. (2021). Extracting Training Data from Large Language Models. The extraction attack and the membership-inference metrics behind it.
• Carlini et al. (2019). The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks. Canaries and the exposure metric.
• Carlini et al. (2023). Quantifying Memorization Across Neural Language Models. Memorization as a function of size, duplication, and context.
• Mitchell et al. (2019). Model Cards for Model Reporting. The documentation standard behind model and system cards.
• Greenblatt et al. (2024). AI Control: Improving Safety Despite Intentional Subversion. Protocols that stay safe against a deliberately subversive model, and how to evaluate them with a red team.

Watch next

▶

▶