Every method in this chapter so far has a person, or a model standing in for one, at the end of the pipeline, judging whether an output is good. RLHF needs raters who can tell the better of two answers (§16.1). Constitutional AI needs a judge who can apply a principle (§19.2). Safety evals need a judge who can recognize harm (§19.4). This works as long as the judge is at least as competent as the model on the task at hand.

What happens when it is not? Ask a model to review a ten-thousand-line change to a codebase, or to summarize a book you have not read, or to propose an experiment in a field where you are not an expert. You can no longer check the answer directly. Worse, a model trained to win your approval is now trained to produce outputs that look right to you, and §19.1 explained what an optimizer does with a proxy that can be fooled. This is the scalable oversight problem (Amodei et al. 2016). How do we supervise systems on tasks where they are more capable than their supervisors, so that the supervision signal still points at the truth?

There is no solution yet. There are four research programs, each with a mathematical idea and some evidence. Debate sets models against each other so that a weak judge only has to check the crux of an argument. Amplification and recursive reward modeling break hard evaluations into easy ones. Weak-to-strong generalization asks whether a strong model trained on imperfect labels can generalize beyond them. And AI-assisted evaluation gives the human a model-written critique. We take them in turn, and we start with how to test any of them before superhuman models exist.

Testing oversight before it is needed

If the problem only bites when models surpass people, how can we do experiments now? Bowman et al. (2022) proposed sandwiching. Pick a task where today's model is better than non-expert people but worse than experts. The non-experts play the role of future overseers, the model plays the role of the more capable system, and the experts supply ground truth that neither has. An oversight method succeeds if it lets the non-experts, working with the model, reach answers closer to the experts' than either the non-experts or the model reaches alone.

They ran it on MMLU questions and on QuALITY reading comprehension questions under a five-minute time limit (so the non-experts could not read the whole passage). Accuracy, in percent, was

People with an unreliable assistant beat both the assistant and themselves. Against the plain model and the unaided individuals in the table the gains are 18 to 28 points. The paper's own summary compares stronger baselines, about 10 points over the model's best setting (a best-of-20 chain-of-thought configuration, 65.6 and 66.9) and up to 36 points over unaided people when each group's answers are pooled by a weighted vote (86.0 against 50.0 on QuALITY). They still fell well short of the experts, which is the gap oversight research tries to close.

Debate

Irving, Christiano, and Amodei (2018) proposed a game. Two copies of a model take opposite sides on a question and argue in turns, and a judge, who could be a person or a weaker model, reads the transcript and decides who won. Training rewards the winner. The bet is that it is harder to lie convincingly than to refute a lie. If one debater claims something false, the other can point at the exact step where it is false, and the judge only has to check that one step rather than the whole argument.

The theoretical case is an analogy with computational complexity. A judge who can only verify short arguments in polynomial time can directly check answers to problems in NP (given a proposed solution, verify it). With two debaters playing optimally and a polynomial-time judge, Irving et al. argued, debate can answer any question in PSPACE, a much larger class, because the debaters together can walk the judge down one path of an exponentially large argument tree, and at each branch the honest debater can force the path toward the branch where the liar is wrong. That is a result about optimal play, and real debaters are not optimal.

Their first experiment made the idea concrete. A classifier judge was trained to recognize MNIST digits from only a few revealed pixels, reaching 59.4% accuracy with six random pixels and 48.2% with four. Two debaters, one assigned the true label and one a false label, each committed to their claim and then took turns revealing one pixel at a time. They could not lie about pixel values, only choose which ones to show. The honest debater won 88.9% of games with six pixels and 85.2% with four. Choosing evidence adversarially made a weak judge far more accurate than seeing random evidence, because each debater revealed the pixels most damaging to the other's claim.

Let's rebuild that game with a judge we can analyze. Each item has a hidden label $y = \pm1$ and 40 features $x_j = \mu_j y + \varepsilon_j$ with $\varepsilon_j \sim \mathcal{N}(0, 1)$, so each feature is weak evidence of strength $\mu_j$. A feature's log-likelihood ratio for $y = +1$ is

The judge sees only the revealed features $R$ and says $+1$ if $\sum_{j\in R}\ell_j > 0$. That would be the optimal decision if the revealed features had been picked without looking at their values. The judge here is naive, since it takes the evidence at face value however it was chosen. We compare three ways of choosing $R$ with $k$ reveals. Random reveals $k$ features at random. Debate lets the honest debater (assigned $y$) and the liar (assigned $-y$) alternate, each revealing the unrevealed feature that most favors its side. Consultancy, a baseline from the empirical debate papers, has a single consultant assigned a side at random, true or false with equal probability, who reveals the $k$ features most favorable to that side.

With two reveals, debate gets the judge to about 85% accuracy where random evidence gives 67%, and with ten reveals it reaches 94.5%, near the 97% of a judge who sees all 40 features. Consultancy is a coin flip. A single advocate who chooses its evidence can make any claim look good to a judge who cannot tell cherry-picked evidence from representative evidence. The adversary is what makes the evidence trustworthy. Two assumptions carry the result, and both are the subject of active research. The revealed features are true (debaters cannot fabricate evidence, as they could not lie about pixel values), and the judge's evidence adds up independently. When a flawed argument is long and its flaw hard to locate, the honest debater may be unable to find the step to challenge, which Barnes and Christiano (2020) called the obfuscated arguments problem. Later protocols, such as doubly-efficient debate (Brown-Cohen et al. 2023), are designed so that an honest debater with modest computation can still win against a dishonest one with far more.

import numpy as np

def debate_accuracy(L, y, k):
    raise NotImplementedError

def consultancy_accuracy(L, y, k, rng):
    raise NotImplementedError

Debate, measured

Debate became an experimental field once language models could argue. The key experimental design hides information from the judge, so that the debaters know more than the judge does, as a future model would. In QuALITY-based experiments, the debaters read a long story and the judge does not.

• Michael et al. (2023) used human debaters and judges. Judges were right 84% of the time with debate and 74% with consultancy, and debates were shorter.
• Khan et al. (2024) used LLM debaters. Debate raised judge accuracy to 76% for LLM judges and 88% for human judges, against naive baselines (the judge alone, without the passage) of 48% and 60%. When debaters were optimized to be more persuasive, judges became more accurate, while optimizing consultants for persuasiveness made judges less accurate. That asymmetry is the central prediction of the debate idea, and this was its first clean confirmation.
• Kenton et al. (2024) ran a broader study with weaker models as judges. Debate beat consultancy on all their tasks, and beat the judge answering directly on extractive tasks with information asymmetry, but results were mixed on tasks without asymmetry, such as math and coding questions the judge could attempt itself.

The honest summary is that debate helps judges most when they lack information that the debaters have, that the advantage over a single advocate is consistent, and that whether it scales to arguments a judge cannot follow at all is open.

Amplification and recursive reward modeling

A second idea is to never ask the overseer a question too hard for them. Iterated amplification (Christiano, Shlegeris, and Amodei 2018) builds an overseer out of a person plus copies of the current model. The person breaks a hard question into easier subquestions, the model answers those, and the person combines the answers. The combination, which is better than either alone, is "amplified", and the model is trained to imitate it. Repeat, and each round the model imitates a slightly stronger system. Recursive reward modeling (Leike et al. 2018) applies the same recursion to evaluation. To train the reward model for agent $k$, the user is assisted by agent $k-1$, itself trained with reward modeling, which helps them evaluate outcomes they could not judge alone.

The best-known demonstration is Wu et al. (2021), who summarized entire books by summarizing sections, then summaries of summaries, with human feedback at each level. Each human judgment concerned a passage short enough to read, even though the final product covered a novel. The summaries were rated comparable to human-written ones for only a small fraction of books (about 5%), but the decomposition made training and evaluation possible at all. The open question for both methods is whether errors compound up the tree, and whether every important task decomposes into pieces a person can judge.

Weak-to-strong generalization

Burns et al. (2023) turned the problem into an experiment that can be run today. Instead of humans supervising a superhuman model, use a small model to supervise a large one. Fine-tune a weak model on ground-truth labels for a task. Use it to label a fresh dataset. Fine-tune a strong pretrained model on those weak labels. Then compare three numbers measured against ground truth on a test set, the weak supervisor's performance $P_{\text{weak}}$, the weak-to-strong student's $P_{\text{w2s}}$, and the strong model's ceiling $P_{\text{ceil}}$ when fine-tuned on ground truth. The performance gap recovered is

In plain words, PGR is 0 if the student only matches its teacher, 1 if it reaches what perfect labels would have given, and in between if it generalizes past its teacher's mistakes. The analogy is that humans will be the weak supervisor of superhuman models, and we would like to know whether the strong model's pretrained knowledge lets it understand what we meant despite our labeling errors.

The findings, across GPT-4-family models, were mixed in an informative way. On NLP classification tasks, naive fine-tuning of a strong student on labels from a much weaker model typically recovered about half the gap, and PGR grew with the strength of the student. On chess puzzles PGR was near zero for the smallest supervisors, and for ChatGPT reward modeling it was about 10%, almost never above 20%. Two interventions helped. Bootstrapping through a chain of intermediate model sizes helped on chess. And an auxiliary confidence loss raised median PGR on NLP tasks from about 25% to nearly 80% for the smallest supervisor and largest student. That loss mixes the usual cross-entropy to the weak labels $f_w(x)$ with a cross-entropy to the student's own hardened predictions $\hat f_t(x) = \mathbb{1}[f(x) > t]$,

where the threshold $t$ is set in each batch so that half the predictions are positive and $\alpha$ is warmed up from 0. In plain words, the student is allowed to disagree confidently with its teacher, which discourages it from bending to fit the teacher's errors.

Why does the student ever beat its teacher? The authors pointed at imitation. A strong student can fit anything, including the weak labels' mistakes, and when it does, PGR falls. It generalizes past the mistakes only when they are hard for it to represent compared with the true concept, which pretraining has made easy to represent. Here is the toy version, with a decision tree as the weak teacher and logistic regression, which can represent the true linear rule exactly, as the student.

A depth-1 tree is a single threshold on one feature, which is itself a linear rule, so the linear student copies it exactly, mistakes and all, and PGR is about zero. Deeper trees make mistakes shaped like boxes, which a linear student cannot represent, so it fits the best linear approximation to the teacher's labels, which is much closer to the truth than the teacher. PGR climbs to around 0.4. The student did not "know" which labels were wrong. Its inability to represent the errors did the work. The widget isolates that mechanism, separating errors a student can copy from errors it cannot.

A bound on the gain

Charikar, Pabbaraju, and Shiragur (2024) turned this intuition into a theorem for regression with squared loss. Fix the strong model's representation and let $\mathcal{F}$ be the set of functions the strong model can express on top of it (for example, linear heads), and suppose $\mathcal{F}$ is convex and contains the true function $f^\star$ (realizability). Let $f_w$ be the weak teacher's predictions and let the student be the best fit to them within the class,

Then

In plain words, the student improves on its teacher by at least the amount by which it fails to fit the teacher's labels. Disagreeing with the teacher is the gain.

The theorem's assumptions are strong. The student must find the exact best fit, the class must be convex and contain the truth, and the expectation is over the data distribution, not a finite sample (the paper gives finite-sample versions). Real fine-tuning satisfies none of these exactly. But it explains the toy, where the linear student cannot fit the tree's box-shaped errors, and it explains why PGR is near zero when the student can fit them, since then the misfit is zero. Lang et al. (2024) gave a complementary analysis of when a student corrects its teacher's pseudolabels and extends to examples the teacher could not label.

import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier

def pgr(weak, w2s, ceiling):
    return w2s / ceiling              # placeholder: not the gap recovered

def weak_to_strong(X_gold, y_gold, X_transfer, X_test, y_test, depth):

import numpy as np

def misfit_terms(Phi, f_star, f_w):
    raise NotImplementedError

AI-assisted evaluation

The most direct form of scalable oversight is to give the human evaluator a model that helps. Saunders et al. (2022) trained models to write critiques of summaries and found that human evaluators shown the critiques found about 50% more flaws than evaluators working alone, including flaws that had been planted on purpose. Larger models were better critics. They also measured a generator-discriminator-critique gap. Models could often tell that an answer was flawed (discriminate) better than they could explain the flaw in a critique, which suggests that models know things about their outputs that they do not volunteer, a gap oversight methods would like to close.

McAleese et al. (2024) applied the idea to code. CriticGPT, a GPT-4 model trained with RLHF to write critiques of code (partly on code with deliberately inserted bugs), produced critiques that contractors preferred over human-written critiques 63% of the time on naturally occurring bugs. Human-machine teams caught about as many bugs as the model alone while hallucinating fewer nonexistent ones. That last point is the trade-off to watch. A critic rewarded for finding problems learns to report problems that are not there (nitpicks and hallucinated bugs), and the paper's decoding method, which searches over critiques scored by a reward model plus a bonus for each highlighted code section, exposes a dial between comprehensiveness and precision.

AI-assisted evaluation is already standard practice, since every RLAIF pipeline in §19.2 and every LLM judge in §19.4 is a model evaluating a model. The scalable oversight question is what happens to the errors. When a critic and the policy share blind spots, or the policy learns what the critic misses, assistance can make an evaluator more confident without making them more correct.

Comparing the programs

None of these has been tested on the problem that motivates it, supervising a system that is genuinely more capable than its overseers in the domain at hand. They are bets about what will still work when that day comes, and the experiments of this section are how the field checks the bets while the gap is small.

In PyTorch

The auxiliary confidence loss is a few lines in any training loop. Here is a small MLP student trained on the depth-6 tree's labels from the cell above, with and without it. The hardened targets are the student's own predictions thresholded at the batch median, as in the paper, and $\alpha$ ramps up over the first fifth of training.

The MLP can fit box-shaped errors much better than a linear student, so naive training recovers less of the gap than logistic regression did. The confidence term recovers more, because it penalizes the student for bending its decision boundary around every one of the teacher's mistakes. The effect in this toy is modest. Burns et al. found it large for big gaps on NLP tasks and much smaller elsewhere, which is the honest state of the method.

Read more

• Bowman et al. (2022). Measuring Progress on Scalable Oversight for Large Language Models. The sandwiching paradigm and its first experiments.
• Irving, Christiano, and Amodei (2018). AI safety via debate. The debate proposal, the complexity argument, and the MNIST experiment.
• Khan et al. (2024). Debating with More Persuasive LLMs Leads to More Truthful Answers. Debate with LLM and human judges, and why persuasiveness helps the truth in debate but not in consultancy.
• Kenton et al. (2024). On scalable oversight with weak LLMs judging strong LLMs. A broad comparison of debate, consultancy, and direct answering.
• Burns et al. (2023). Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision. The weak-to-strong setup, PGR, and the auxiliary confidence loss.
• Charikar, Pabbaraju, and Shiragur (2024). Quantifying the Gain in Weak-to-Strong Generalization. The misfit bound and its finite-sample versions.
• Leike et al. (2018). Scalable agent alignment via reward modeling: a research direction. Recursive reward modeling and the research agenda around it.
• McAleese et al. (2024). LLM Critics Help Catch LLM Bugs. CriticGPT and the trade-off between catching bugs and hallucinating them.

Watch next

▶

▶