A model can only get better from its own work if something better than its own guesses tells it which work was good. This chapter studies every way to build that signal, from self-critique and self-play to verifier-filtered self-training, self-judged rewards, research agents, and recursive self-improvement, and proves where each loop saturates or collapses.

This chapter is about self-improvement, a model getting better by learning from its own outputs. It ends the part of the book on agents. Chapter 24 built models that act in a loop of tool calls, and Chapter 25 gave them memory that outlasts one context window. A system that can act and remember can also judge its own work and train on it. Whether that really improves the model shapes how labs spend compute while a model answers, how they build training data, and how they train reasoning models. One idea runs through the whole chapter. A model can improve on its own outputs only when something tells it which outputs were good, and that something must know more than the model's own guesses.

The chapter reuses several tools from the chapter on reasoning. A verifier is a checker that scores a model's answers, best-of-$n$ keeps the highest-scoring of $n$ samples, and pass@$k$ is the chance that at least one of $k$ samples is right (§18.2). Reinforcement learning with verifiable rewards trains a model on answers that a program can check (§18.3 and §18.4). And §18.7 described a revision by two numbers, how often it fixes a wrong answer and how often it breaks a right one.

This section studies the simplest such loop, self-refinement, which means letting a language model check its own answer and then rewrite it. The model answers, reads what it wrote, says what is wrong with it, and writes a new answer. The loop needs no training and no new data, only more calls to the same model. So it is the first thing people try when a model makes mistakes.

The published evidence seems to contradict itself. GPT-3.5 answered multiple-choice questions about everyday knowledge from the CommonSenseQA benchmark and was right 75.8% of the time. Then it was asked to review its answer and fix any problems it found. After one round of this its accuracy was 38.1% (Huang et al. 2024). Asked to double-check, it lost half its accuracy. Yet Self-Refine, a loop in which a model critiques and rewrites its own output, reports a gain of about 20 points on average over seven tasks (Madaan et al. 2023). Both results were measured carefully, and both are true.

They fit together once you split the loop into its two jobs. The critic decides whether the current answer is wrong. The reviser rewrites the answers the critic flagged. This section shows that the critic decides almost everything. Revision pays only when the critic flags wrong answers much more often than right ones, and the better the model already is, the sharper the critic must be.

Self-refinement carries the chapter's argument in its simplest form. Here the thing that must know more than the model's own guesses is the critic. We open up the revision step of §18.7 into a small model of critique and revision, and prove exactly when it raises accuracy. We define the generation-verification gap, the accuracy a checker adds when it keeps only the samples it accepts, which the rest of the chapter reuses. Then we read the published results through that lens, build a loop that does not cheat, and end with how labs train models to correct themselves.

Two results that seem to disagree

The table puts the numbers side by side. Every row compares a model's first attempt with its output after it critiqued and revised its own work, with no help from outside. GSM8K is a set of grade-school math word problems. Constrained Generation asks for a short passage that uses 20 to 30 given words, scored by the fraction of the words it uses. Dialogue Response asks for the next turn in a conversation, scored by how often a GPT-4 judge prefers it when shown both versions without being told which is which.

With GPT-4, the same loop adds 30 points of word coverage and two tenths of a point on math. The difference lies in how hard each output is to check. Checking whether a passage contains each of 20 words is easy, and a model does it well. Spotting the one slipped sign in a confident chain of arithmetic is hard. On the math problems, ChatGPT's feedback said "everything looks good" 94% of the time (Madaan et al. 2023). So revision helped where the output could be checked against the request, and failed where the critic had to recognize the one right answer. The dialogue row also measures a preference rate rather than an accuracy, so it is not directly comparable with the others.

The intuition fits in one sentence. A revision step changes only the answers the critic flags, so it helps exactly when the flagged answers are mostly wrong ones. Here is that sentence with real numbers.

The example already holds two lessons. What matters about a critic is how much more often it flags wrong answers than right ones. And the more right answers you start with, the more there are to break, so a strong model needs a sharper critic than a weak one. The rest of this section turns those two sentences into a theorem and then tests the theorem on published tables.

Three kinds of self-correction

The phrase self-correction has been used for setups that differ in the one way that matters, which is where the critic's information comes from. Huang et al. 2024 drew the line that the field now uses.

Oracle stopping always looks good, and a short calculation shows why. Under oracle stopping a right answer is never touched again, so nothing is ever broken. Say the first attempt is right with probability $p_1$, and each revision fixes a wrong answer with probability $p_{\text{fix}}$. A problem is still wrong after $m$ attempts only if the first attempt and all $m - 1$ revisions failed. So the probability of being right after $m$ attempts is

which can only go up. With $p_1 = 0.8$ and $p_{\text{fix}} = 0.5$, three revisions give $1 - 0.2 \times 0.5^3 = 0.975$. What the reviser would do to a right answer never matters, because it never sees one.

Huang et al. measured both versions with GPT-4 on GSM8K, with at most two rounds of revision. With oracle stopping, accuracy rose from 95.5% to 97.5%. Without it, the same model and the same three prompts (answer, critique, revise) gave 91.5% after one round and 89.0% after two. The 8.5 points between 97.5 and 89.0 are the work of the oracle.

Several earlier positive results on reasoning used the labels to decide when to stop. They include the question-answering experiments of Reflexion, an agent loop met later in this section, and RCI, a prompting method that recursively criticizes and improves an answer (Kim et al. 2023). Without the labels, the gains on reasoning disappeared in Huang et al.'s experiments. A loop that stops on the evaluation labels has leaked the test set into the method. So the first question to ask of any self-correction result, and the one an interviewer will ask you, is whether the loop knew when it was right.

The finding is general, as §18.7 also reported. Without labels, every model that Huang et al. 2024 tested ended below its first attempt on every benchmark after two rounds (Tables 3 and 4). GPT-3.5 on CommonSenseQA went from 75.8% to 38.1% after one round and 41.8% after two, and Llama-2-70B-chat on GSM8K went from 62.0% to 43.5% and then 36.5%.

The widget runs one reviser under three stopping rules. The reviser fixes half of the wrong answers it revises and breaks half of the right ones.

• With oracle labels, only wrong answers go back for revision, and the critic is never consulted.
• With a self-judged stop, an answer the critic accepts is final. A flagged answer is revised and judged again.
• With fixed rounds, the critic judges every answer in every round.

One slider sets the first-attempt accuracy, and the other sets the critic's quality $c$, between 0 and 1. The critic's detection rate, the chance it flags a wrong answer, is $(1 + c)/2$. Its false-alarm rate, the chance it flags a right answer, is $(1 - c)/2$. So $c = 0$ is a coin flip and $c = 1$ is a perfect critic.

With a coin-flip critic, a first attempt at 0.8, and three rounds, the oracle-stopped loop reaches the 0.975 computed above. The self-judged loop falls to 0.65, and fixed rounds fall to about 0.54. The reviser is identical in all three. Only the choice of which answers it sees has changed.

Counting the answers that flip

Two accuracies hide how they were reached. A loop that fixes 10 answers and breaks 10 shows no change, yet it rewrote 20 answers. To see what a loop does, count the answers that flip in each direction. Kumar et al. 2024 report five numbers for a loop with two attempts, and this section uses them throughout.

The identity holds problem by problem. Expand the right side of $c_{2i} - c_{1i} = (1 - c_{1i})c_{2i} - c_{1i}(1 - c_{2i})$ to get $c_{2i} - c_{1i}c_{2i} - c_{1i} + c_{1i}c_{2i}$, which is the left side. Averaging over $i$ gives it for the fractions. In plain words, $\Delta$ is the net change, and the two flip fractions say how it was made.

The flips separate two kinds of bad critic that look alike in $\Delta$. A timid critic that rarely flags anything shows small flips in both directions. A critic that cannot tell right from wrong shows large flips that cancel or go negative. The same paper gives a real case, which §18.7 read as fix and break rates. Its authors asked Gemini 1.5 Flash to check its work on 500 problems from MATH, a benchmark of competition mathematics, and its accuracy went from 52.6% to 41.4%. It fixed 4.6% of all problems and broke 15.8%, so it broke more than three right answers for every wrong one it fixed. When you report a self-correction experiment, report both flip fractions, not only the net change.

import numpy as np

def self_correction_metrics(first, second):
    raise NotImplementedError

A critic decides whether revision helps

§18.7 modeled revision with two numbers. Each revision fixes a wrong answer with probability $p_{\text{fix}}$, the fix rate, and breaks a right one with probability $p_{\text{break}}$, the break rate. Write $p_m$ for the probability that the answer is right after $m$ attempts. A right answer stays right unless it is broken, and a wrong answer becomes right if it is fixed, so

Each round moves accuracy a fixed fraction of the remaining way toward the plateau $p^\star = p_{\text{fix}}/(p_{\text{fix}} + p_{\text{break}})$, the accuracy at which fixes and breaks cancel. So revision raises accuracy exactly when the first attempt starts below the plateau, $p_1 < p^\star$. For the Gemini model above, $\hat p_{\text{fix}} = 4.6/47.4 = 0.097$ and $\hat p_{\text{break}} = 15.8/52.6 = 0.300$, so its plateau is $0.097/0.397 = 0.24$, far below its 52.6% start.

That model treats revision as one opaque step. A real loop has the two parts named at the start, a critic that decides whether something is wrong and a reviser that rewrites what the critic flagged. Opening the step up shows where the fix and break rates come from, and so which part to improve.

The critic is a binary classifier whose positive class is "this answer is wrong". In the language of the confusion matrix from §0.11, $p_{\text{det}}$ is its true positive rate and $p_{\text{fa}}$ its false positive rate. Watch the direction when you compare it with a grader that accepts answers, such as the answer checkers of §18.3. For such a grader, a false positive means accepting a wrong answer, so $p_{\text{det}} = 1 - \mathrm{FPR}$ and $p_{\text{fa}} = \mathrm{FNR}$.

The theorem below needs one more word. The odds of a probability $p$ are $p/(1 - p)$, so a model that is right 80% of the time has odds of 4 to 1.

In plain words, revision pays when two ratios multiplied together beat the odds that the first answer was already right. The first ratio says how much more often the critic flags wrong answers than right ones. The second says how much more reliably the reviser fixes than breaks.

The theorem has four consequences worth knowing by heart.

• A critic with no false alarms always helps. It never touches a right answer, so accuracy climbs toward 1. Oracle stopping is exactly this critic, and item 4 is the formula from the start of the section.
• A critic that flags at random needs a remarkable reviser. A random critic has $\Lambda = 1$, and so does "always revise". The condition becomes $q_{\text{fix}}/q_{\text{break}} > p_1/(1-p_1)$. A model that is right 80% of the time then needs a reviser that fixes a wrong answer four times as reliably as it breaks a right one.
• Better models need sharper critics. The smallest likelihood ratio that helps is $\Lambda^\star = \frac{p_1}{1 - p_1}\cdot\frac{q_{\text{break}}}{q_{\text{fix}}}$. With $q_{\text{fix}} = q_{\text{break}}$, a 50% model needs $\Lambda > 1$, an 80% model $\Lambda > 4$, a 90% model $\Lambda > 9$, and GPT-4 at 92.9% on GSM8K $\Lambda > 13.1$. A critic that says "everything looks good" almost every time rarely flags anything, so revision barely moves accuracy either way. That is what Self-Refine saw on math, where ChatGPT's feedback said so 94% of the time and GPT-4's accuracy moved by 0.2 points.
• Once a critic discriminates, false alarms matter most. Near the current critic, lowering the false-alarm rate by one point raises the plateau $\Lambda$ times as much as raising the detection rate by one point. For large changes the advantage of cutting false alarms is even bigger. The derivation below proves both.

In practice this tells you where to spend effort. Before you build a self-review step, measure the critic's two rates on answers you have labeled and compute $\Lambda$. Then compare $\Lambda q_{\text{fix}}/q_{\text{break}}$ with the odds $p_1/(1-p_1)$. If it falls short, work on false alarms first. Interviewers often ask exactly this, in the form "your model is 90% accurate, should it check its own work?".

The cell checks the theorem by simulation. It runs critic-gated revision on 100,000 independent problems and compares the simulated accuracy after each round with the closed form. It does this for the example above, for a critic with no false alarms, and for a sharp critic of the kind unit tests provide.

The simulated and exact rows agree to within a few thousandths, the sampling noise at this size. The worked example sinks toward 0.714. The critic without false alarms climbs toward 1. The unit-test critic, with $\Lambda = 0.9/0.05 = 18$, climbs toward its plateau $p^\star = 0.45/0.465 = 0.968$.

In the next widget you move the critic's two rates and the first-attempt accuracy, and each preset also sets the reviser's $q_{\text{fix}}$ and $q_{\text{break}}$. The left panel shows accuracy after each round of revision, with the plateau $p^\star$ dashed. The right panel shades, for each likelihood ratio and first-attempt accuracy, whether revision helps or hurts. The presets are a prompted critic, oracle stopping, unit tests, and the measured rates of SCoRe, a trained self-correction method described at the end of the section.

The preset starts at $\Lambda = 1.5$, and five rounds take accuracy from 0.80 down to 0.58. Raising detection to 0.5 gives 0.65, and lowering false alarms to 0.1 gives 0.72. The derivative alone would favor detection here, since a 0.1 cut in false alarms is worth only about $1.5 \times 0.1 = 0.15$ of detection, less than 0.2. The cut wins anyway, because the plateau is convex in the false-alarm rate and concave in the detection rate. Neither change is enough. This reviser breaks more often than it fixes ($q_{\text{break}} = 0.6$ against $q_{\text{fix}} = 0.4$), so the critic would need $\Lambda > 4 \times 1.5 = 6$.

import numpy as np

def critic_chain(p1, p_det, p_fa, q_fix, q_break, m):
    p_fix, p_break = q_fix, q_break          # every answer revised, flagged or not
    p_star = p_fix / (p_fix + p_break)
    return p_star + (p1 - p_star) * (1 - p_fix - p_break) ** m

def revision_helps(p1, p_det, p_fa, q_fix, q_break):

The generation-verification gap

A critic can be used in a second way. Instead of rewriting the answers it rejects, sample several answers and keep only the ones it accepts. This is rejection sampling. Its close cousin best-of-$n$ keeps the one answer out of $n$ that the checker scores highest (§18.2). Filtering is also how the self-training loops later in this chapter build their data. A checker used this way is called a verifier, written $g$. Two acceptance rates describe it. Its true positive rate $\mathrm{TPR}_g$ is the chance that it accepts a right answer, and its false positive rate $\mathrm{FPR}_g$ is the chance that it accepts a wrong one. What does filtering buy?

Start with numbers. A generator is right 80% of the time. Its verifier accepts 90% of right answers and 20% of wrong ones. Out of 100 samples it accepts 72 right ones and 4 wrong ones, so the kept answers are right $72/76 = 94.7\%$ of the time. Filtering raised accuracy by 14.7 points without touching the generator.

In plain words, any verifier that accepts right answers more often than wrong ones makes the kept answers more accurate than the raw ones. The gain is largest for middling $p$ and vanishes as $p$ approaches 0 or 1.

Now compare filtering with revision. A critic's flag is a rejection, so as a verifier it has $\mathrm{TPR}_g = 1 - p_{\text{fa}}$ and $\mathrm{FPR}_g = 1 - p_{\text{det}}$. Then $\mathrm{TPR}_g > \mathrm{FPR}_g$ is the same as $p_{\text{det}} > p_{\text{fa}}$, that is, $\Lambda > 1$.

The evidence later in this section confirms this twice. In an interview it is the answer to "why does best-of-$n$ with a mediocre verifier help when self-revision with the same verifier hurts?".

Song et al. 2024 turned this gain into the central measure of self-improvement. In words, the gap is how much better the answers get when the model's own samples are kept, or weighted, according to a verifier's scores. Their definition allows any way of using the scores, not only a pass or fail filter.

In plain words, the gap is the utility gained by keeping or weighting the model's samples by the verifier's scores. The relative gap measures that gain as a fraction of the room left to improve.

For a utility of 0 or 1 and a filter on one prompt, the gap is exactly $\operatorname{prec} - p$ from the theorem. The relative gap divides it by the room left, $\operatorname{gap}_{\text{rel}} = (\operatorname{prec} - p)/(1 - p) = p(\mathrm{TPR}_g - \mathrm{FPR}_g)/A$. In the 80% example above, the gap is 0.147 and the relative gap is $0.147/0.2 = 0.74$. The relative version matters because the absolute gap of every verifier shrinks to 0 as $p \to 1$. The relative gap instead tends to $1 - \mathrm{FPR}_g/\mathrm{TPR}_g$, a property of the verifier alone.

Why does the gap close? Filtering removes the errors the verifier can see, so what remains is more and more made of the errors it cannot see. The cell makes this a three-line model. Wrong answers come in two kinds. The verifier accepts detectable ones with probability 0.1, and undetectable ones as often as right answers. Each round keeps what the verifier accepts and trains the generator to reproduce exactly that mix.

The gap goes from 0.218 to 0.034 to 0.004, and accuracy stalls at 0.857, far short of 1, although the verifier is excellent on the errors it can see. On what is left, the verifier has $\mathrm{TPR}_g = \mathrm{FPR}_g$, which is $\Lambda = 1$, and the theorem says no amount of filtering helps. Only new information moves the plateau, such as a stronger verifier, ground-truth answers, tests, or tools. The rest of this chapter keeps asking where that new information comes from.

def filter_precision(p, tpr, fpr):
    return p * tpr / p

def gv_gap(p, tpr, fpr):
    raise NotImplementedError

def relative_gap(p, tpr, fpr):
    raise NotImplementedError

Self-Refine, a loop that critiques and rewrites

The cleanest loop of this kind runs entirely at test time, while the model answers, with no training. It uses one model in three roles. Each role is set up by a few-shot prompt, a prompt that contains a handful of worked examples of that step. One prompt drafts, one gives feedback, and one rewrites. This is Self-Refine (Madaan et al. 2023).

In the language of the theorem, the feedback step is the critic, and its stop signal is the critic's verdict that nothing is wrong. So Self-Refine is critic-gated revision with a self-judged stop. It also shows the reviser every earlier draft and critique, so the rounds are not quite independent.

The paper's ablation shows how much rides on the critic. The authors replaced the specific, actionable feedback with generic feedback such as "improve the efficiency", and then with no feedback at all (Madaan et al. 2023, Table 2). Code Optimization with ChatGPT fell from 27.5 to 26.0 to 24.8. Sentiment Reversal with ChatGPT fell from 43.2 to 31.2 to 0. Acronym Generation with GPT-3.5 fell from 56.4 to 54.0 to 48.0. The same critique-and-revise move, run offline against written principles and then trained into the weights, is how Constitutional AI makes training data for harmlessness (§19.2).

Why do the gains pile up on some tasks and vanish on others? The next cell runs two toy tasks, each with Self-Refine's cap of 4 rounds and its stop signal.

• Constrained generation. An answer must contain $K = 20$ concepts, and the draft covers each with probability 0.6. The critic checks a list. It reports each missing concept with probability 0.9, invents a complaint about a present concept with probability 0.05, and stops when it has nothing to report. The reviser adds each reported concept with probability 0.9. While rewriting, it drops each unreported concept with probability 0.03.
• Arithmetic. A first attempt is right 75% of the time. The critic flags 15% of wrong answers and 3% of right ones, so it says "looks good" about 94% of the time, as ChatGPT's math feedback did. The reviser fixes a flagged wrong answer with probability 0.6 and breaks a flagged right one with probability 0.5.

Coverage climbs from 0.600 to 0.976, while arithmetic accuracy moves from 0.745 to 0.759. Two things separate the tasks, and both are in the theorem. First, coverage is many cheap checks, each with a high likelihood ratio, and there are plenty of errors to find, since the first draft misses 40% of the concepts. Second, the arithmetic critic finds only 15% of the slips, so there is little to gain in the first place. And with three right answers for every wrong one, the right answers it breaks give back half of what its fixes gain.

Self-Refine's own appendix makes the same point from the other side. When an outside signal said which math answers were wrong, so that only those were refined, the gain for GPT-3.5 went from 0 to 4.8 points (Madaan et al. 2023, Table 9). That is oracle stopping again. So when someone proposes a refine loop, ask first how checkable the task is.

Reflexion, lessons written in words

An agent, as §24.1 defines it, is a model in a loop that observes, thinks, and acts until a task is done. Reflexion (Shinn et al. 2023) wraps that loop in an outer loop of whole attempts, called trials, and carries a lesson written in words from each failed trial to the next. §24.4 defined it and compared it with plain retries. Here is the recap in the terms this section needs.

Three roles, usually played by one model, share a memory. An actor attempts the task and produces a trajectory, the record of everything it did in one trial. An evaluator scores the trajectory, with the environment's success signal, a hand-written rule, an exact match against a reference, or unit tests the model wrote itself. After a failure, a self-reflection step writes a short lesson ("I assumed I was holding the mug, but I never picked it up") into a memory that keeps the last one to three lessons. The next trial reads the memory, and the loop stops when the evaluator reports success or a cap on trials is reached.

No weights change. The lesson does the job a gradient update would do in reinforcement learning, but it lives in the prompt, which is why the authors call it verbal reinforcement. It is a small case of the episodic memory of §25.3, a store of past experiences that an agent reads back. §24.4 asked what a lesson adds over a retry. This section asks a different question. In this section's terms the evaluator is the critic, so the way to read Reflexion's results is to ask which evaluator each experiment used.

• Decision-making on AlfWorld, 134 household tasks in a text game. The environment reports only whether a task is complete, so a hand-written rule flags a trial that loops or runs past 30 actions. A completed task ends the loop and is never flagged, so this evaluator raises no false alarms. With it the agent completed 130 of 134 tasks over 12 trials, as §24.4 reported.
• Question answering on HotpotQA, questions that need facts from two Wikipedia articles. Between trials, the answer was graded by exact match against the reference answer, so the loop knew which answers were wrong. By the definitions at the start of this section this is oracle stopping, and Huang et al. list it as such.
• Programming on HumanEval and MBPP, two benchmarks of short Python functions checked by hidden unit tests. The evaluator is a suite of at most six unit tests that the model writes itself and filters for valid syntax. The hidden tests are used once, on the final program, so the reported pass@1 (the fraction of problems whose final program passes) is honest. Reflexion reached 91.0% on HumanEval against 80.1% for GPT-4 alone, but 77.1% on MBPP against the same 80.1%.

The precision theorem explains the difference. Among programs that passed the model's own tests, 1.4% were wrong on HumanEval and 16.3% on MBPP (Shinn et al. 2023, Table 2). So the self-tests had a precision of 98.6% on one benchmark and 83.7% on the other. With base accuracies near 80% on both, the self-tests opened a large gap on HumanEval and only a small one on MBPP. There, a wrong program that passed its own tests ended the loop early, and the loop finished 3 points below the baseline.

An ablation on the 50 hardest HumanEval problems, translated to Rust, told the same story (Shinn et al. 2023, Table 3). The base model scored 0.60, written reflections without tests 0.52, tests without written reflections 0.60, and both together 0.68. Reflection without a test signal is intrinsic self-correction, and it hurt. An interviewer who asks why Reflexion works on code wants this answer, the tests, and the false positives that limit them.

The next exercise turns the leak that Huang et al. found in the literature into an interview coding problem.

def reflexion(task, max_trials, seed):
    rng = np.random.default_rng(seed)
    memory, name = [], None
    for trial in range(max_trials):
        name = generate(task, memory, rng)
        if hidden_tests(task, name):              # stops on the benchmark's own labels
            return name, trial + 1, True
        memory.append({"candidate": name, "failed": []})

What the evidence says

Kamoi et al. 2024 surveyed the self-correction literature with the same distinctions this section draws. Their conclusions are worth memorizing.

That the critic, not the reviser, is the bottleneck has been measured directly. Tyen et al. 2024 built BIG-Bench Mistake, a set of step-by-step reasoning traces written by the PaLM 2 model, with the first logical error in each marked by people. They asked models to find the error. The best model, GPT-4, located it with 52.87% accuracy overall. Then they gave models the true location and let them backtrack, which means re-sampling the mistaken step and continuing from there.

Stechly, Valmeekam & Kambhampati 2024 measured the critic itself. GPT-4 generated solutions to four formal problems, and the same model verified them, with exact external checkers as ground truth. The problems were Game of 24 (combine four numbers with arithmetic to make 24), graph coloring, and two planning puzzles. The puzzles are Blocksworld and Mystery Blocksworld, which is the same puzzle with its words disguised. The cell converts their verifier statistics (Table 2) into this section's critic rates, where a rejection counts as a flag.

The same paper asked whether critique is needed at all, and found that it adds little. It asked the identical question again until the exact checker accepted an answer. With 15 samples, the same budget as the 15 rounds of feedback, it was ahead of the loop that was told "wrong", or within a point of it, in three domains, and behind it on Game of 24. The scores were 40% against 38% on graph coloring, 68% against 60% on Blocksworld, 9% against 10% on Mystery Blocksworld, and 28% against 36% on Game of 24. With 25 samples it beat that loop in every domain, and Game of 24 reached 42%. Its prompts also stay short, since they never carry earlier attempts. That is filtering with $\mathrm{FPR}_g = 0$, whose precision is 1 by the theorem. It succeeds whenever any sample is right, which is the coverage of §18.2.

The last lesson concerns compute. Self-correction spends extra model calls, so it must be compared with other ways of spending them. Huang et al. 2024 (Table 7) compared two ways to spend 9 responses on GSM8K. In multi-agent debate, three copies of a model answer and then critique each other's answers over two rounds. In self-consistency, the model answers 9 times independently and the majority answer wins. From a single-sample 76.7%, debate reached 83.0% and voting 88.2%. The gain from debate came from combining several answers, not from critique, and at equal cost the critique left it below plain voting. Debate as a way to supervise models stronger than their judges is a different question, taken up in §19.5. In practice, every self-correction loop you build must beat voting or best-of-$n$ at the same number of calls.

External feedback changes the critic

Every loop that worked so far had a critic that knew something the generator did not. In the theorem's terms, external feedback improves both ratios at once. It raises $\Lambda$, because a failing test or a contradicting search result is rarely a false alarm. And it raises $q_{\text{fix}}/q_{\text{break}}$, because an error message says where to look.

• Tools as the critic. CRITIC (Gou et al. 2024) lets the model call tools to check its draft and then revise from what they report. The tools are a search engine for facts, a Python interpreter for math, and a toxicity classifier for safety. On AmbigNQ, a set of open-domain questions with more than one reading, ChatGPT's F1 score (word overlap with the reference answer) went from 64.3 with step-by-step reasoning to 74.9 with CRITIC. It reached only 67.3 when the model had to write its own "evidence" without search. On GSM8K with Text-Davinci-003, solving by writing a program scored 70.1%, CRITIC with the interpreter 72.2%, and the same loop without the interpreter 68.3%, below where it started.
• Execution results as the critic. Self-Debug (Chen et al. 2023) teaches a model, with few-shot examples, to explain its code line by line and to read the results of running it. Where unit tests exist (code translation and MBPP), it improved accuracy by up to 12%. On Spider, a text-to-SQL benchmark with no tests to run, it improved by 2 to 3%, and by 9% on the hardest problems. With feedback, it matched or beat baselines that sampled more than 10 times as many programs.
• Execution feedback for training data. The Llama 3 team found that training their 405B model on its own generated code was "not helpful (and can even degrade performance)" (Llama Team 2024, Section 4.3.1). It started to help once they added execution feedback as a source of truth. Errors from a parser, a linter, and model-written unit tests were fed back for revision. Only dialogues that passed every check were kept, and about 20% of solutions had been wrong at first and were then corrected. That is critic-gated revision used to build training data, and §26.4 takes the pipeline apart.

The table sums up the trade-offs. They come down to where the critic's signal comes from and what it costs.

Training a model to correct itself

If prompted critics are the bottleneck, the next idea is to train the model to correct itself. The obvious recipe is supervised fine-tuning (SFT), which trains the model to imitate given text, here traces in which an answer gets corrected. Kumar et al. 2024 showed carefully why it fails. They sampled many two-attempt traces from Gemini 1.5 Flash on MATH and tried two versions.

• Filtered traces. Keep only traces that turned a wrong first answer into a right one, and fine-tune on them. This is the recipe of §26.3, which keeps a model's own samples that reached a right answer and trains on them, here applied to corrections.
• Paired traces. Pair a wrong attempt with a right one for the same problem, as a synthetic repair, following Welleck et al. 2022. The authors call this Pair-SFT.

Recall that on 500 test problems the base model went from 52.6% at the first attempt to 41.4% at the second, a $\Delta$ of $-11.2$ points. Three rounds of the filtered recipe made the second attempt worse relative to the first ($\Delta = -14.2$). Adding traces where both attempts were right brought it to $+0.4$. Pair-SFT reached $+1.8$. Pair-SFT with both-right traces learned to change nothing at all, with no flips in either direction.

They traced the failure to two causes. The first is distribution shift. Offline traces correct the base model's mistakes, but once training changes the first attempt, the mistakes the model must fix at test time are different ones. The second is behavior collapse. Producing the best possible first answer and then making minimal edits scores as well on the training data as genuinely correcting. The first is easier to learn, so that is what the model learns. Training on the model's own fresh samples, called on-policy training, fixes the first cause but not the second. Plain reinforcement learning that rewarded only the second attempt collapsed the same way. Both attempts got better together, and the gap between them did not grow.

SCoRe, their method, uses reinforcement learning in two stages. §18.7 quoted its headline result, and here we take the method apart. It needs two terms from §15.7. A policy is the model viewed as a probability distribution over responses. A KL penalty charges the policy for drifting from a frozen reference model. The drift is measured by the KL divergence, a nonnegative number that is zero only when the two distributions are equal.

Why should the bonus have $\alpha > 1$? Look at what it does to each kind of transition, and to the return as a whole.

Proof. The four values follow by substituting $(\hat r(y_1), \hat r(y_2)) \in \{(1,1), (0,0), (0,1), (1,0)\}$. For the average, $\mathbb{E}[G] = \mathrm{Acc@t1} + \mathrm{Acc@t2} + \alpha(\mathrm{Acc@t2} - \mathrm{Acc@t1})$. Substituting $\mathrm{Acc@t2} = \mathrm{Acc@t1} + \Delta$ gives $2\,\mathrm{Acc@t1} + \Delta + \alpha\Delta$. ∎

In plain words, without the bonus ($\alpha = 0$) the objective is $2\,\mathrm{Acc@t1} + \Delta$. A point of first-attempt accuracy is worth twice a point of improvement, so the easiest way up is a better first answer followed by no edits, which is behavior collapse. With $\alpha > 1$, the weight $1 + \alpha$ on $\Delta$ exceeds the weight 2 on first-attempt accuracy. Breaking a right answer ($1 - \alpha < 0$) also becomes worse than staying wrong (0). The bonus makes correcting the better strategy on the training set, which the plain objective failed to do.

The same identity hides a trap. In terms of the two accuracies, $\mathbb{E}[G] = (1 - \alpha)\,\mathrm{Acc@t1} + (1 + \alpha)\,\mathrm{Acc@t2}$, and for $\alpha > 1$ the weight on first-attempt accuracy is negative. If the first attempt were trained on the whole return, the policy would learn to answer wrong on purpose and then fix it. That is why SCoRe credits each attempt only with its own reward. The first attempt is trained only to be right. For a given first attempt, the bonus makes a right second answer earn $1 + \alpha$ more than a wrong one, instead of 1 more. So the bonus strengthens the training signal for the second attempt, and it is the second attempt that gets corrected.

The cell turns the published tables into rates and extrapolates each to later rounds with the constant-rate model. Read the extrapolations as what the rates imply, not as forecasts, since Figure 8 of the paper shows how far they can miss.

Every SCoRe row beats its base model on the break rate. On HumanEval SCoRe also nearly doubles the fix rate, from 0.171 to 0.319. Converting a published table into rates is quick. Estimating rates from your own logs of many rounds, with honest error bars, is a small research problem, and the next exercise does it.

import numpy as np

def fit_fix_break(logs):
    L = np.asarray(logs, dtype=bool)
    acc = L.mean(axis=0)                         # one accuracy per round
    d = np.diff(acc)
    return max(d.mean(), 0.0), max(-d.mean(), 0.0)

Checking inside one chain of thought

Reasoning models do most of their self-correction inside a single chain of thought, the step-by-step reasoning a model writes before its answer. These are the "wait, let me re-check" passages of §18.5. This differs from prompted self-correction in two ways, and the theorem explains both.

First, the check is trained. Reinforcement learning with an outcome reward, a reward given only for a correct final answer (§18.3), credits only what the chain ends with. A re-check that catches an error raises the reward and is reinforced. A re-check that talks the model out of a right answer lowers the reward and is suppressed. So training raises $\Lambda$ and $q_{\text{fix}}/q_{\text{break}}$ directly, against a true checker.

Second, the check happens before the answer is committed, so a false alarm costs tokens rather than a finished answer. §18.5 also records the caveat that base models already produced these behaviors, and training amplified them.

In PyTorch

The SCoRe objective is a few lines on top of a REINFORCE loss, the basic policy-gradient loss of §15.3. It raises the log-probability of each sampled response in proportion to how much better than average its return was. This sketch follows the equations above, not the authors' code. Like the paper, it credits each attempt only with its own terms, so the bonus never reaches the first attempt. It uses the sampled log-ratio as the KL estimate inside the reward, as in §15.7, and subtracts the batch mean as a baseline.

Stage 1 keeps the small second-attempt penalty beta1 and gives the first attempt only the strong penalty beta2, as in the definition. Passing g2 back into g1 would put the weight $1 - \alpha = -9$ on the first attempt's reward and teach it to fail. For $\alpha = 10$ the four shaped returns are $+2$, $0$, $+11$, and $-9$, and the average matches $2\,\mathrm{Acc@t1} + (1 + \alpha)\Delta$, as the theorem says. In a real run the log-probabilities come from a language model's forward pass on the sampled responses, and the reference model is frozen.

Read more

• Huang et al. (2024). Large Language Models Cannot Self-Correct Reasoning Yet. The intrinsic, extrinsic, and oracle distinction, the oracle versus intrinsic tables, the debate versus self-consistency comparison, and the prompt-design critique.
• Madaan et al. (2023). Self-Refine: Iterative Refinement with Self-Feedback. The refine loop, seven tasks, and the feedback ablation. Read Table 1 together with Appendix H on math.
• Shinn et al. (2023). Reflexion: Language Agents with Verbal Reinforcement Learning. Verbal lessons in an episodic memory, with self-written tests for code and the false-positive analysis that explains MBPP.
• Kamoi et al. (2024). When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMs. A taxonomy of research questions, a checklist for fair experiments, and the three conclusions summarized above.
• Stechly, Valmeekam & Kambhampati (2024). On the Self-Verification Limitations of Large Language Models on Reasoning and Planning Tasks. Self-critique against sound verifiers on formal tasks, with the verifier error rates used in this section. Read it with Tyen et al. (2024), LLMs cannot find reasoning errors, but can correct them given the error location.
• Gou et al. (2024). CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing, with Chen et al. (2023), Teaching Large Language Models to Self-Debug, and the Llama 3 report's Section 4.3.1, The Llama 3 Herd of Models. External feedback from tools, execution, and tests, at test time and for training data.
• Kumar et al. (2024). Training Language Models to Self-Correct via Reinforcement Learning. Why SFT on correction traces collapses, and SCoRe's two stages. Qu et al. (2024), Recursive Introspection: Teaching Language Model Agents How to Self-Improve, is the multi-turn imitation alternative.
• Song et al. (2024). Mind the Gap: Examining the Self-Improvement Capabilities of Large Language Models. The generation-verification gap, its scaling with pretraining compute, and its saturation under iteration. Shao et al. (2025), DeepSeekMath-V2: Towards Self-Verifiable Mathematical Reasoning, keeps the gap open with a trained proof verifier.

Watch next

▶

▶

▶