Human feedback has three problems that get worse as models get better. It is expensive, since a preference dataset like those in §16.2 takes a large budget and months of annotator time. It is inconsistent, since raters disagree with each other on a quarter of comparisons and with themselves more often than anyone likes to admit. And it is opaque. Ask what values a reward model trained on 100,000 comparisons encodes, and the only honest answer is "whatever those raters, on those days, happened to prefer". Nobody can read it, argue with it, or fix one line of it.

In December 2022 a team at Anthropic trained a harmless assistant without a single human label saying which outputs were harmful (Bai et al. 2022b). The only human input about harmlessness was a list of 16 short principles, a constitution, written in plain English. A model read the principles, criticized its own drafts against them, rewrote the drafts, and then judged pairs of responses against them to produce the preference data for RL. The resulting model was less harmful than one trained with human harmlessness labels at the same level of helpfulness, and, unlike it, almost never evasive.

This section builds that pipeline, Constitutional AI, piece by piece, then the broader family it started, reinforcement learning from AI feedback (RLAIF), rule-based rewards, and deliberative alignment, in which a reasoning model is trained to read the specification and think about it before answering. The question to keep asking is the one from §19.1. Each method writes the intended objective down in a different place. Where, and what can go wrong there?

Stage one, critique and revision

Start with a model that is helpful but has no harmlessness training, a "helpful-only" RLHF model. Give it a prompt written to elicit something harmful, a red-team prompt. It will often comply. Then, in the same context, ask it to criticize its own response according to one principle, and then to rewrite the response to address the criticism. Here is the shape of one round, with a prompt that is unpleasant rather than dangerous.

User: My coworker keeps taking credit for my work. Write an email I can send to the
      whole team that will humiliate him.
Assistant (initial): Subject: Let's talk about who actually does the work around here ...
      [a cutting email that mocks the coworker by name]

Critique request: Identify specific ways in which the response is harmful, unethical,
      or likely to damage someone's reputation or relationships.
Critique: The email publicly humiliates a named person, which could hurt him and the
      sender's own standing, and it escalates a conflict that has better remedies.

Revision request: Rewrite the response to remove the harmful content while still
      helping with the user's underlying problem.
Revision: It is maddening to have your work credited to someone else. A team-wide email
      meant to humiliate him is likely to backfire on you. Here is a short, firm
      message to your manager that documents your contributions, and a script for
      raising it with your coworker directly ...

The revision is not a refusal. It engages with what the user actually needs, which is the point of the method. Bai et al. ran four critique-revision rounds per prompt, drawing a principle at random from the 16 at each round, on 182,831 red-team prompts (42,496 written by people and 140,335 generated by a model), and kept the revision from every round, four per prompt. To keep the model helpful they added two responses from the helpful-only model to each of 135,296 human-written helpfulness prompts. They then fine-tuned a pretrained model on this data for one epoch. The result is called SL-CAI.

What is the objective here? Write $\mathcal{D}$ for the red-team prompts, $y_0 \sim \pi_{\text{help}}(\cdot \mid x)$ for the helpful-only model's draft, and $\operatorname{rev}_k$ for the $k$-th critique-and-revise step, each conditioned on a random principle $c_k$ from the constitution $\mathcal{C}$. Stage one maximizes

the SFT objective of §14.1 on revised responses (plus the same objective on the helpfulness responses, which we leave out of the formula). In plain words, teach the model to produce, directly and in one shot, what it would have produced after one or more rounds of thinking about the principles. The paper's qualitative finding was that the first revision removed most of the harm and later rounds helped less visibly. The principles appear in the data-generating context but not in the training context, so their effect is distilled into the weights. This is a form of context distillation, training a model without some context to imitate itself with that context, and it is the same trick deliberative alignment uses at the end of this section.

Why does it work at all? Because criticizing is easier than producing. A model that writes a harmful email on request can still recognize, when asked, that the email is harmful. Saunders et al. (2022) measured this kind of gap directly for summaries (§19.5), and it is the empirical bet behind every method in this section. The model's ability to evaluate against a principle runs ahead of its default behavior, and training can move the behavior up to meet the evaluation.

Stage two, reinforcement learning from AI feedback

Supervised revisions give a decent starting point. To optimize further, Bai et al. replaced the human harmlessness comparisons of RLHF with AI comparisons. Sample two responses $y_A, y_B$ from SL-CAI for a red-team prompt, draw a principle, and ask a feedback model a multiple-choice question.

Consider the following conversation between a human and an assistant:
[conversation]
[principle, e.g.] Which of these assistant responses is less harmful? Choose the response
that a wise, ethical, polite, and friendly person would more likely say.
Options:
(A) [response A]
(B) [response B]
The answer is:

The feedback model never generates an answer. Instead we read off the log-probabilities it assigns to the next tokens "(A)" and "(B)", call them $\ell_A$ and $\ell_B$, and normalize them into a soft label,

with $\sigma$ the logistic sigmoid. In plain words, the label is how confident the feedback model is that $A$ is better, a number between 0 and 1 rather than a hard choice. Bai et al. found that these soft labels worked much better than hard 0 or 1 labels. The AI harmlessness comparisons (182,831 of them) were then mixed with 135,296 human helpfulness comparisons, a preference model was trained on the mixture, and the SL-CAI model was trained with PPO against it, exactly the loop of §16.4. The result is RL-CAI. Helpfulness still came from people. Harmlessness came from the constitution.

Three details turn this from a sketch into something that works, and each has a short piece of math behind it.

Detail one, the reward model learns from soft labels

The reward model is the Bradley-Terry model of §16.3. With a reward $r_\phi(x, y)$ and margin $m = r_\phi(x, y_A) - r_\phi(x, y_B)$, the model's probability that $A$ wins is $\sigma(m)$. With a soft target $p$, the loss is the cross-entropy between the two Bernoulli distributions,

Its derivative with respect to the margin follows from $\frac{d}{dm}\log\sigma(m) = 1 - \sigma(m)$ and $\frac{d}{dm}\log\sigma(-m) = -\sigma(m)$,

In plain words, the gradient pushes the model's win probability toward the label, and the loss is minimized when $\sigma(m) = p$, at the margin $m^\star = \log\frac{p}{1-p}$. A hard label ($p = 1$) asks for an infinite margin, so the reward model keeps stretching its scores apart on every confident pair. A soft label asks for a finite margin, the log-odds of the labeler's confidence. That is one reason soft labels behave better. They tell the reward model how much better, not only which.

Detail two, clamping chain-of-thought labels

Asking the feedback model to "think step by step" before choosing improved its accuracy, but the probabilities it gave after reasoning were nearly always close to 0 or 1, because the reasoning ends with a confident conclusion. Those labels behave like hard labels and, by the formula above, ask for very large margins. Bai et al. clamped the chain-of-thought probabilities into the range $[0.4, 0.6]$, which they found improved results over both no clamping and a looser $[0.2, 0.8]$. The largest margin a clamped label can request is $\log(0.6/0.4) \approx 0.41$. In plain words, a reasoning labeler is trusted for the direction of a preference and not for its strength, so every pair gets a small, bounded push.

Detail three, averaging over principles and over orders

A single principle is a noisy view of the constitution. If each principle's label is the "true" constitutional preference plus independent noise of variance $s^2$, averaging over $K$ principles reduces the noise variance to $s^2/K$. Bai et al. instead drew one principle at random per comparison, which gives the same average over the dataset, since each pair's label is an unbiased sample of the constitution-averaged label, and costs one forward pass per pair. They reported that this ensembling over principles gave notably more robust preference-model behavior than using the same principle for every label.

The order matters for a less obvious reason. Language-model judges have a position bias, a systematic preference for whichever option is shown first (or second). Section 14.7 measured it for evaluation, and §16.2 applied its fix to preference labels, as Lee et al. (2023) did. Label every pair twice, once in each order, and average,

where $p_{BA}$ is the probability that the first-shown response, now $B$, wins. Recall from §16.2 why this works. If the judge adds a constant $\beta$ (a local symbol here, not the KL coefficient of Chapter 16) to the logit of whichever response is first, and its unbiased logit for $A$ over $B$ is $m$, then $p_{AB} = \sigma(m + \beta)$ and $1 - p_{BA} = \sigma(m - \beta)$, whose average is $\tfrac12$ at $m = 0$ and on the same side of $\tfrac12$ as $m$ otherwise. The bias can no longer flip a label, and the price is that confident labels are pulled toward $\tfrac12$ (at $m = 1$ and $\beta = 1.5$ the average is 0.65 rather than $\sigma(1) = 0.73$), which costs some signal but adds no direction of its own. What those sections did not show is what an uncorrected bias teaches a reward model. Here is the situation where it does the most damage. The responses shown first are systematically different from those shown second, here because they come from a newer model whose answers are longer, and length has nothing to do with quality.

The labeler picks the first-shown response about 75% of the time, and since the first-shown response is always the longer one, the reward model trained on single-order labels learns a weight of about 0.4 on length, a feature the true quality ignores. Optimize a policy against that reward model and it will learn to write longer answers, which is one route to the length bias of §16.5. Averaging both orders puts the length weight back at zero. The other weights are slightly shrunk in both cases, because noise and the bias blur the labels, but their directions are right.

import numpy as np

def debiased_preference(logits_ab, logits_ba):
    p = np.exp(logits_ab[:, 0]) / np.exp(logits_ab).sum(1)   # placeholder: one order only
    return p, 0.0

import numpy as np

def bt_soft_loss(w, D, p):
    raise NotImplementedError

def fit_reward(D, p, lr=0.5, steps=500):
    raise NotImplementedError

What Constitutional AI bought

Bai et al. measured helpfulness and harmlessness with human crowdworkers comparing model responses, converted to Elo scores, and plotted every model on the plane those two scores define. RL-CAI models learned to be less harmful at a given level of helpfulness than the standard RLHF model trained on human helpfulness and harmlessness labels. In the language of §19.1, the frontier moved rather than the model sliding along it. And RL-CAI was "virtually never evasive". Where the human-feedback model often deflected sensitive requests, RL-CAI engaged with them and explained its objections.

Three caveats keep this honest. The evaluation was done by crowdworkers following the lab's instructions, so it measures harmlessness as those raters judged it. RL-CAI could be over-trained, and when it was, it Goodharted in a recognizable way, attaching boilerplate reassurances to responses that did not need them, which is §19.1's overoptimization in miniature. And the constitution's 16 principles were written by the researchers, so the method moves the value question from "which raters?" to "which principles?" without answering it. The improvement is that the principles are short, public, and editable. You can read every word of what the model was told.

RLAIF beyond harmlessness

Is AI feedback only good for harmlessness, a domain where a model can recognize obvious harms? Section 16.2 gave the headline answer from Lee et al. (2023), who compared RLAIF with RLHF using a PaLM 2 L labeler and a PaLM 2 XS policy. Human raters preferred RLAIF and RLHF policies over SFT at nearly the same rates on summarization (71% and 73%) and helpful dialogue (63% and 64%), the head-to-head split was about even, RLAIF was more often harmless (88% against 76%), and the labeler agreed with people about as often as people agree with each other (78.0% on summarization, against 73 to 77% between humans). For our purposes the details of the labeler matter more. Agreement fell with labeler size (73.8% for PaLM 2 S, 62.7% for XS), chain-of-thought prompting raised it by one to two points, and self-consistency (majority voting over several reasoning samples) lowered it. A labeler is an instrument, and its quality is a design variable.

Two further results bear on the scalable oversight questions of §19.5. RLAIF beat SFT (68%) even when the labeler was the same small size as the policy. And in direct RLAIF, which skips the reward model and asks the labeler to score each sampled response from 1 to 10 during RL (using the probability-weighted score as the reward), the policy beat SFT 74% of the time on summarization. On helpful dialogue, direct RLAIF beat SFT 66% of the time with the labeler being the exact same checkpoint as the starting policy, a case of a model improving itself with no external judge.

What AI feedback inherits, it inherits from the judge. A model judge has position bias (we just handled it), length bias, and self-preference. Panickssery et al. (2024) found that GPT-4 and Llama 2 can recognize their own outputs, and that the strength of a model's preference for its own outputs grows linearly with its ability to recognize them. A judge drawn from the same family as the policy will therefore tend to reward the policy for sounding like itself, and RLAIF can amplify a model's quirks instead of correcting them. Section 14.7 treated the evaluation side of the same problem, length-controlled win rates for verbosity bias, and correcting a judge with a small set of human labels by prediction-powered inference. The same human-labeled slice is how an RLAIF team checks that its labeler still agrees with people as the policy changes.

Rule-based rewards

Constitutional principles are broad ("choose the less harmful response"). For behaviors that a policy team can specify in detail, such as how to refuse, a finer-grained method works better. Mu et al. (2024) broke desired behavior into propositions, binary facts about a response that an LLM grader can judge reliably with a few-shot prompt. For a request that should be refused, examples are "the response contains a short apology", "the response is judgmental toward the user", and "the response offers a safe alternative". The grader returns a probability $\varphi_i(x, y)$ for each proposition, and the reward is a weighted sum added to an ordinary helpful-only reward model,

In plain words, a human-written rubric becomes a reward by asking a model to grade each line of it, and a small number of weights decides how much each line matters. The weights are fit on a small set of completions ranked by how well they follow the policy, so that ideal completions score highest. With these rule-based rewards, OpenAI reported an F1 score between safety and usefulness of 97.1, against 91.7 for a human-feedback baseline and 95.8 for a helpful-only baseline. The appeal is precision and editability. If the policy on self-harm conversations changes, you change a proposition and refit a handful of weights, instead of relabeling a dataset.

import numpy as np
from sklearn.linear_model import LogisticRegression

def fit_rule_weights(F, ideal):
    return np.ones(F.shape[2])        # placeholder: every proposition counts the same

def top1_accuracy(F, ideal, w):
    raise NotImplementedError

Deliberative alignment

All of the methods so far hide the specification from the model at inference time. The constitution shapes the labels, the labels shape the reward, the reward shapes the weights, and the deployed model never sees a word of the policy it was trained on. That has a cost. When a request falls in a gray area the training data did not cover, the model has to guess what the policy would say from patterns it absorbed indirectly.

Reasoning models (§18.5) offer an alternative. Teach the model the specification itself, and train it to recall the relevant parts and reason about them in its chain of thought before answering. Guan et al. (2024) called this deliberative alignment and used it to align OpenAI's o-series models. The recipe has two stages, and neither needs human-written chains of thought.

1. Supervised data, by context distillation. For a prompt $x$ from some safety category, put the relevant section of the written safety specification $\mathrm{spec}$ into the context of a reasoning model and sample a chain of thought $c$ and an answer $y$. A judge model that also reads the specification scores each completion several times, and the minimum score is used to filter out bad samples. Then drop the specification from the prompt and keep the tuple $(x, c, y)$. SFT on these tuples teaches the model to produce, without being given the spec, the reasoning it would have produced with it, including citing and applying the relevant rules.
2. Reinforcement learning with a spec-aware judge. The same specification-reading judge supplies the reward during RL. The judge sees only the final answer, not the chain of thought. The authors hid the reasoning from the reward on purpose, to avoid putting direct optimization pressure on it and so to reduce the chance of training the model to write deceptive reasoning (§19.6 explains why that matters).

In the notation of this section, stage one minimizes, over prompts, the divergence between a teacher that reads the spec and a student that does not,

estimated with filtered samples from the teacher, which is SFT on those samples. In plain words, the specification moves out of the context window and into the weights, and it comes back out as explicit reasoning at inference time.

The headline result is a move of the frontier between two failure modes that usually trade off. Here are the paper's numbers for robustness to known jailbreaks (StrongREJECT, higher is better) and for not refusing benign prompts that look dangerous (XSTest, higher is better), with GPT-4o, a model without this training, as a reference.

The o-series models are much harder to jailbreak at a similar or better rate of not over-refusing. Two cautions. These models differ from GPT-4o in many ways besides deliberative alignment, so the table shows what the whole training produced, not what the method alone contributed. The closer evidence for the method is the paper's ablations within one training setup. A run with safety data in both the SFT and RL stages beat runs with safety data in only one stage, which beat a run with none, and a baseline given the full specification in its system prompt at inference, without the training, learned less of the intended behavior, especially the required response styles. These are the authors' own experiments on their own models, not an independent replication. And the claim that reasoning about a policy makes behavior more robust depends on the reasoning being what actually drives the answer, the faithfulness question of §18.1 and §19.6.

Comparing the methods

Every row still has the outer alignment problem of §19.1. The specification can be wrong, incomplete, or gamed. What changes is how inspectable it is and how cheaply it can be fixed. And every row except the first leans on the same empirical bet, that a model can evaluate behavior against a standard better than it behaves by default. Whether that bet keeps paying as models surpass their evaluators is the subject of §19.5.

In PyTorch

In a real pipeline the reward model is a Transformer with a scalar head (§16.3), and the soft-label loss is one line, binary cross-entropy with logits, which takes soft targets directly. Here it is on the toy features, with and without clamping the labels to $[0.4, 0.6]$, to see what clamping does to the learned margins.

Clamping shrinks the margins almost threefold while leaving the reward model's rankings almost unchanged. For RL that is the point. The policy still learns which response is better, but no single comparison can demand a huge reward gap, which limits how hard the policy can push on any one direction the labeler happened to be confident about.

Read more

• Bai et al. (2022b). Constitutional AI: Harmlessness from AI Feedback. The two-stage method, the principles, soft and clamped labels, and the helpfulness-harmlessness frontier.
• Lee et al. (2023). RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback. The head-to-head comparison, labeler alignment, position debiasing, and direct RLAIF.
• Guan et al. (2024). Deliberative Alignment: Reasoning Enables Safer Language Models. Teaching a reasoning model the specification and training it to reason over it.
• Mu et al. (2024). Rule Based Rewards for Language Model Safety. Propositions graded by an LLM and combined into a precise safety reward.
• Bai et al. (2022a). Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback. The human-feedback baseline, and the measured tension between helpfulness and harmlessness.
• Panickssery et al. (2024). LLM Evaluators Recognize and Favor Their Own Generations. Self-preference in model judges, and its link to self-recognition.
• Anthropic (2023). Claude's Constitution. The principles used to train a production model, published in full.

Watch next

▶

▶