This section is about training a model on data that a model wrote. Such data is called synthetic data. Frontier labs now use a great deal of it, because it is cheap, plentiful, and can be aimed at exactly the skills a model lacks. But synthetic data is only as good as whatever decides which of it to keep. We follow that decision through three cases. First an outside check, such as a test suite, decides. Then the model judges its own work. Finally nothing checks at all.

Two results show what is at stake. The first is about a model that grades itself. In January 2024 a team at Meta fine-tuned Llama 2 70B on 3,200 human-written examples of good answers and about 1,600 examples of how to grade an answer. Then they let the model teach itself (Yuan et al. 2024), a loop that §17.5 met briefly as a form of iterative DPO. In each round the model wrote four answers to each of a few thousand new prompts. It graded its own answers on a 5-point scale and was trained to prefer its best answer over its worst. The team measured it on AlpacaEval 2.0, a set of 805 instructions on which GPT-4 compares each answer with GPT-4 Turbo's answer to the same instruction. The fraction of instructions on which GPT-4 prefers the model's answer is its win rate. Over two rounds of self-teaching the win rate went from 9.94% to 15.38% to 20.44%, ahead of Claude 2, Gemini Pro, and GPT-4 0613 on that leaderboard. Over the same rounds the average answer grew from 1,092 to 1,552 to 2,552 characters. Did the model get better, or only longer?

The second result is about the whole field. Models now write much of the text that later models train on. Some of it is deliberate. Over 98% of the data used to align Nemotron-4 340B was synthetic (NVIDIA 2024). Some of it is accidental, as the web fills with model-written pages. Shumailov et al. (2024) trained models on their own outputs, generation after generation. The models degraded, and the rare parts of the data vanished first. The authors called this model collapse, and §8.4 derived its simplest case. Then Gerstgrasser et al. (2024) proved that the error of the same loop stays bounded if each generation's data are added to the old data instead of replacing them. Both results are true. Which one describes the world depends on details you can compute.

This chapter has one lesson so far. A model can learn from its own outputs only when something tells it which outputs were good, and that something must know more than the model's own guesses. §26.1 measured that signal as the generation-verification gap, the accuracy gained by keeping only the outputs a checker accepts. §26.3 showed that training on checked outputs climbs only as far as the checker allows. In this section the checker gets weaker step by step. We compute what a filter buys and what it costs. We derive what a self-judging loop optimizes and why it drifts toward longer answers. And we prove when a loop with no checker collapses and when it does not. By the end you can design a synthetic data pipeline with numbers and defend it in an interview.

Most synthetic data comes from one loop

The loop has four steps. Sample candidate examples from a model, check them, keep what passes, and train on what you kept. §14.2 taught three versions of it that make instruction data, the prompt and response pairs a chat model is fine-tuned on.

• Growing tasks from seeds. Start from 175 human-written tasks and have a model write new ones. Discard each new task whose wording overlaps too much with a task already in the pool (Wang et al. 2023).
• Writing the question for a real answer. Take a human-written web document and have a model write the instruction that the document would answer (Li et al. 2023).
• Letting a chat model invent the user. Give an aligned chat model only the opening of a chat template, the special tokens that start a user's turn. It then writes a plausible user request by itself (Xu et al. 2024).

A fourth source is distillation, in which a stronger model writes the responses that a weaker one is trained on. §8.4 showed the pretraining version, a small code model trained on textbooks that a larger model wrote (Gunasekar et al. 2023). Every one of these pipelines throws most of what it generates away, and what it keeps decides the quality.

When a synthetic dataset disappoints, find the stage that failed before you change the generator. An interviewer who asks you to design a pipeline expects all four stages and a typical failure for each.

What a filter buys, and what it costs

A verifier is any check that accepts or rejects a candidate. Two numbers describe it. Its true positive rate $\mathrm{TPR}_g$ is the chance that it accepts a correct candidate. Its false positive rate $\mathrm{FPR}_g$ is the chance that it accepts a wrong one. The subscript $g$ stands for the verifier, as in §26.1. Here is what the two numbers do to the kept data.

In general, let each candidate be correct with probability $p$. The fraction of candidates the verifier accepts is the acceptance rate

In plain words, a candidate is accepted either because it is right and passes, or because it is wrong and slips through. §26.1 proved, by Bayes' rule, that the accepted candidates are correct with probability

This is the filter's precision. The same section showed that $\operatorname{prec} > p$ exactly when $\mathrm{TPR}_g > \mathrm{FPR}_g$. So a filter improves the data whenever it accepts right answers more often than wrong ones.

A data engineer needs two more numbers. The first is the cost. Each candidate is accepted with probability $A$, independently of the others, so the number of candidates you draw until one is accepted has a geometric distribution, with mean $1/A$. That makes $1/A$ the number of generations one kept example costs. The second is the yield, the number of correct kept examples per candidate, which is $p\,\mathrm{TPR}_g$.

In plain words, precision is what the kept data are worth, and $1/A$ is what they cost. A stricter filter usually raises both, buying cleaner data at a higher price per example.

How strict must the filter be? Fix the precision you want and solve for the false positive rate.

In plain words, the allowed false positive rate is a product of three numbers. The first is the generator's odds of being right, $p/(1-p)$. The second is the checker's true positive rate. The third is the odds of error you will tolerate, $(1-\operatorname{prec}_{\text{tgt}})/\operatorname{prec}_{\text{tgt}}$. The first factor is small when the generator is weak, so a weak generator needs a nearly perfect checker.

There is a second cost, and it is easy to miss. The filter changes which prompts the kept data come from. Let $p(x)$ be the chance that the generator solves prompt $x$, its per-prompt success rate (§18.2). Each prompt keeps samples at its own acceptance rate, $p(x)\mathrm{TPR}_g + (1-p(x))\mathrm{FPR}_g$. Easy prompts keep many samples and hard prompts keep few. So the hardest prompts nearly vanish from the kept data, and so do the skills they teach.

The cell first computes the three filters seen so far, the hand example, one from the interview questions at the end of the section, and the strict checker. Then it measures the second cost on 10,000 prompts. Their success rates are drawn from a Beta(0.5, 0.5) distribution, which piles up near 0 and near 1, so there are many easy prompts and many hard ones.

This is why the team behind the phi-4 model generated several answers per question and discarded two kinds of question (Abdin et al. 2024). They dropped questions on which every answer agreed, as too easy. They also dropped questions on which the answers were entirely inconsistent, as too hard or ambiguous. For the rest they used the most common answer as the label.

Reinforcement learning has a related blind spot. Group-relative policy optimization, or GRPO (§18.4), scores each sample against the other samples for the same prompt, so a prompt that is always solved, or never, gives it no signal. In practice, report precision and yield per difficulty bucket, never only on average. An interviewer will give you $p$, the true positive rate, and the false positive rate. They will ask for the wrong fraction and the cost, and then ask what the average hides.

import numpy as np

def filter_stats(p, tpr, fpr):
    accept = p * tpr + (1 - p) * fpr
    return {"precision": accept, "acceptance": accept,          # placeholder: precision is not the acceptance rate
            "gens_per_kept": 1 / accept, "correct_per_gen": p}

def fpr_for_precision(p, tpr, target):

Verification when the model also writes the tests

Code is the friendliest case for verification, because code can be run. The Llama 3 team found that training the 8B and 70B models on code written by a larger model helped a lot (Llama Team 2024, in the report's section on code). Training the 405B model on code it wrote itself did not help, and could even hurt. That is the self-improvement problem in miniature, since the largest model had no stronger teacher.

What fixed it was execution feedback, a check that actually runs the code. The team generated programming problems from random code snippets and had the model solve them. Every solution went through a parser and a linter, a tool that flags code that is malformed or suspicious without running it. Then it ran in a sandbox, an isolated environment where it cannot do harm, against unit tests that the model also wrote. Failures went back to the model for revision, and only dialogues that passed every check were kept.

The obvious worry is that model-written tests can be wrong. A wrong test that matches a popular bug rewards the bug. One fix lets programs and tests check each other, a method called CodeT (Chen et al. 2022). Sample many programs and many tests from the same model, and run every program on every test. Then look for a large group of programs that behave identically, together with the tests they all pass. Here is a small case first.

In plain words, a program earns credit for every program, itself included, that behaves exactly like it on the tests, multiplied by how many tests that behavior passes. This works because correct programs agree with each other on every input. Wrong programs tend to be wrong in different ways, so they split into many small groups. A wrong test costs the correct group only one test out of many, while every correct test is passed by every correct program.

Neither half alone has both protections. A majority vote over programs is won by a popular bug. A count of passed tests is inflated by wrong tests. On HumanEval, a benchmark of 164 Python problems with hidden tests, CodeT raised the pass@1 of the code-davinci-002 model from 47.0% to 65.8%. Pass@1 is the chance that the one program chosen passes the hidden reference tests. The cell runs the median example.

In practice, consensus is how you verify code when no one has written trusted tests, which is the usual case for synthetic problems. An interviewer will ask you to implement it, then ask which bugs it cannot catch. The answer is bugs that most programs and most tests share.

def codet_select(programs, tests):
    # placeholder: trust the tests, pick the program that passes the most of them
    best, best_n = 0, -1
    for i, f in enumerate(programs):
        n = sum(f(x) == y for x, y in tests)
        if n > best_n:
            best, best_n = i, n
    return best

Verifiers, from strongest to weakest

Code is the easy case. The table lists the verifiers available elsewhere, roughly from strongest to weakest. Two abbreviations appear in it. RLVR is reinforcement learning with verifiable rewards, training against an automatic checker of final answers (§18.3). RLAIF is reinforcement learning from AI feedback, in which a model's preferences replace human ones.

The top row is the purest case of verified synthetic data. The AlphaGeometry system learned olympiad geometry without ever seeing a human proof (Trinh et al. 2024). A symbolic deduction engine, a program that applies the rules of geometry mechanically, was run on random geometric premises. It produced 100 million synthetic theorems with proofs, every one correct by construction. A language model trained only on them learned to propose the extra points and lines that a proof needs, which the engine cannot invent. The system solved 25 of 30 recent olympiad geometry problems, against 10 for the best previous method. The verifier was perfect, so the only limits were how much of geometry the random premises covered and how varied they were.

Synthetic data at real scale

The best of these pipelines share two traits. First, they are anchored in real data. The web-rewriting study kept the original pages in the mix, and found that training on rewrites alone hurt perplexity on domains whose style the rewrites lacked. Second, most of them use a stronger or separate model as the generator. That is distillation, the subject of §20.4, which also covers its on-policy form, in which the student generates and the teacher grades. Diversity needs its own fix, because a model sampled many times tends to say the same few things.

Two cautions carry over from earlier chapters. Generated data can reproduce benchmark questions that the generator memorized. So the decontamination of §8.3 and §22.3 must run on synthetic data too. And a generator's outputs carry its license terms, which chain through every model trained on them (§14.2).

When the model is its own judge

Now take away the outside checker. In a self-rewarding loop, the model that writes the responses also grades them, and its grades become the training signal. The loop is built on iterative DPO (§17.5). DPO, direct preference optimization, trains a model on pairs of responses to the same prompt, one marked better and one worse. It raises the probability of the better one and lowers that of the worse one, each measured relative to a fixed reference model. Iterative DPO repeats this in rounds. Each round samples several responses per prompt from the current model, scores them, pairs the best with the worst, and trains on the pairs (you build such pairs in Exercise 17.5b). A self-rewarding model is iterative DPO in which the model being trained is also the scorer.

In the paper's main runs the new prompts came from a fixed model, Llama 2-Chat 70B shown eight example prompts. The responses and their grades came from the model being trained. Evaluation used outside judges, GPT-4 and the authors themselves, never the model's own scores. That is the only reason we can say anything about whether the model improved.

Why would this work at all? Judging a response is easier than writing a good one. So a judge that agrees with humans on 80% of pairs opens a generation-verification gap, and the loop can climb it. The judge even improved without new judge data, from 78.7% to 81.7%. The authors attribute this to better general instruction following.

And why would it fail? Because the judge's errors are not random. §22.4 measured three biases in language-model judges. They prefer the answer shown first, they prefer longer answers, and they prefer text in their own style. All three can act in a self-rewarding loop, and the length growth above is the one that shows in the published numbers. The length bias is the same one that §16.3 found in learned reward models.

A judge that systematically likes some feature other than quality hands the policy a direction to move in. The next subsection shows that the loop moves further in that direction every round. The same caution applies to RLAIF and to constitutional training (§19.2), where an AI judge also supplies the preferences. There, though, the judge is usually a separate, fixed model guided by written principles.

What the loop optimizes

Each round moves the policy, the model's distribution over responses, toward the responses its judge scores highly. It moves only a little, because DPO keeps the new policy close to its reference. What happens after many rounds? A tiny example shows it.

Here is the general version. Idealize each round as the exact optimum of the KL-regularized objective of §17.1. That objective is the expected reward minus $\beta$ times the KL divergence from a reference policy, where the KL divergence measures how far one distribution has moved from another. The KL coefficient $\beta > 0$ sets how far a round may move. In round $t+1$ the reward is the judge's current score $s_t(x, y)$, and the reference is the previous policy $\pi_t$, one of the two reference choices of §17.5. §17.1 proved that the optimum is the tilted distribution

In plain words, every response keeps its old probability, multiplied by a factor that grows with its score.

Proof. By induction on $T$. The claim holds for $T = 1$ by assumption. If it holds for $T$, then $\pi_{T+1} \propto \pi_T e^{s_T/\beta} \propto \pi_0\exp\big(\sum_{t<T}s_t/\beta\big)e^{s_T/\beta}$, which is the claim for $T + 1$ (the normalizing constants depend only on $x$). With $s_t = s$ the exponent is $Ts/\beta = s/(\beta/T)$, which is the optimum of §17.1 with coefficient $\beta/T$ and reference $\pi_0$. Finally, the ratio $\pi_T(y)/\pi_T(y')$ equals $\frac{\pi_0(y)}{\pi_0(y')}e^{T(s(y)-s(y'))/\beta}$, which tends to infinity whenever $s(y) > s(y')$, so all the mass ends on the maximizers of $s$. ∎

In plain words, $T$ rounds of self-rewarding with a fixed judge equal one round of reinforcement learning from human feedback (RLHF), with the judge as the reward model and a KL coefficient $T$ times smaller. Each round loosens the leash that ties the policy to the starting model. In the limit the policy is whatever the judge likes best.

That is exactly the regime in which §16.5 measured overoptimization. As the policy moves away from its reference, the true reward first rises, then peaks, then falls. In an interview, this theorem answers "why do self-rewarding gains stop after a few rounds, and what would you watch?" Watch the KL divergence from the starting model, and a judge you do not train.

A real judge's score mixes quality with things that are not quality. Describe each response by two features, its content quality $q$ and its length $\ell$. Take the judge's mean score to be $s = q + b\,\ell$, where $b \ge 0$ is the judge's length bias, the score it adds per unit of length. The next theorem says exactly what the loop does to each feature when the two features follow a Gaussian distribution under the starting policy.

In plain words, two things push length up. A length-biased judge does it directly, through the term $b\sigma_\ell^2$. A correlation between length and quality in the starting policy does it too, through $\rho\sigma_q\sigma_\ell$, even when the judge is unbiased. Tilting toward quality drags along whatever travels with quality. And since the quality gain is linear in the number of rounds while the cost of growing length is quadratic, the true utility must eventually turn down.

For a number, take $\sigma_q = \sigma_\ell = 0.15$, $\rho = 0$, $b = 0.5$, and $\beta = 0.1$. Each round adds $0.0225/0.1 = 0.225$ to the mean quality and $0.5\times 0.0225/0.1 = 0.1125$ to the mean length. In practice, the theorem explains why answers can grow in a self-judged loop even when the judge has no length bias at all. An interviewer will ask whether an unbiased judge stops length growth. The $\rho$ term says it does not, whenever longer answers in the starting policy are also better ones.

The cell runs the idealized loop on a grid of 121 responses, with quality and length each taking 11 values in $[0, 1]$. Capping quality at 1 makes the downturn come sooner than in the Gaussian model, where the same starting policy and gold score put the peak near round 8. The judge is $s = q + b\,\ell$. The gold score, what a careful reader would give, is $u = q - 0.8\,\ell^2$. The printout compares three runs, an unbiased judge ($b = 0$), a biased judge ($b = 0.5$), and the biased judge with length control.

Before each round, length control fits the judge score as a linear function of length under the current policy, and subtracts the fitted slope times length. That is the idea behind AlpacaEval's length-controlled win rate (Dubois et al. 2024), which §16.6 derived as a logistic regression with a length term.

That last result is too good to be general. When length and quality are genuinely correlated, removing the length slope also removes part of the real quality signal. The widget starts from a policy in which length and quality have correlation 0.4, so you can see two effects, length growth under an unbiased judge and the quality that length control gives up.

With $b = 0$ the mean length still climbs, because longer answers start out better and the tilt drags length along with quality. Length control removes the whole length slope, so quality climbs a little more slowly than under the unbiased judge. But length now stays flat, and gold charges for length, so gold ends higher than with the unbiased judge. Length control gives up a little real quality to stop the padding, and whether that pays depends on how much a reader is hurt by length.

The judge noise slider adds a fixed error to each response's score. It stands for the judge's idiosyncratic likes and dislikes. This is not the sampling noise of a single judgment, which averaging removes. An error attached to a response is there every round, and the tilt finds and amplifies exactly the responses the judge overrates. So with $b = 0$ and enough noise, the judge's score pulls far ahead of gold, and gold ends below what an unbiased judge reaches.

§16.5 called this regressional Goodhart. When you select on a noisy proxy, the winners are partly the ones whose noise happened to be high. Here it is worse than one selection, because every round concentrates further on the few responses with the largest errors. The meta-judge option previews the next subsection, in which the judge is trained too. It multiplies the bias and the noise by 0.65 each round, so they shrink before they compound.

import numpy as np

def self_reward_rounds(b, beta, rounds, length_control=False):
    pi = PI0.copy()
    gold, judge, length = [], [], []
    for t in range(rounds + 1):
        s = Q_GRID + b * L_GRID
        gold.append((pi * GOLD).sum()); judge.append((pi * s).sum()); length.append((pi * L_GRID).sum())

Training the judge too

If the judge is the weak link, improve the judge. Wu et al. (2024) argued that self-rewarding saturates quickly because its training improves the model as a responder and only incidentally as a judge. Their method, meta-rewarding, adds a third role.

Acting as a meta-judge, the same model compares pairs of its own judgments of one response. It sees each pair in both orders, to cancel the preference for whichever comes first. The comparisons give each judgment an Elo-style rating, the kind of rating chess uses to rank players from their wins and losses. The best and worst judgments then form DPO pairs that train the judging skill.

They also changed how response pairs are chosen, to fight length. Responses whose scores fall in a top tier near the maximum count as equally good, and the shortest of them becomes the chosen response. The longest response in a bottom tier becomes the rejected one. And because the meta-judge also liked long judgments, they dropped pairs whose chosen judgment was too long.

Wang et al. (2024) went further and trained a judge with no human preference labels at all. The trick is to build pairs whose answer is known by construction. Take a real user instruction $x$. An instruction-following model (Mixtral 8x22B in their experiments) writes a good response $y^w$ to it. The same model then writes a slightly different instruction $x'$ and a good response $y^l$ to $x'$. That response is plausible for $x$ but worse, because it answers a different question.

The judge being trained samples several judgments of the triple $(x, y^w, y^l)$, keeps one that prefers $y^w$, and is fine-tuned on the kept judgments. Then the loop repeats with the improved judge. (In the first iteration Mixtral also wrote the judgments, and from then on the model being trained wrote them.) This is the bootstrapped reasoning of §26.3, training on your own outputs that a check accepted, with judging as the task and the construction of the pair as the check.

Both are real improvements, and both move the problem rather than solve it. The meta-judge is the same model with the same blind spots, so the length bias reappeared one level up. The Self-Taught Evaluator's labels are only as good as the assumption that a response to a modified instruction is worse. That holds for the kind of pairs it builds, and says nothing about subtler failures. An interviewer who hears "no human labels" will ask what anchors the loop, and the honest answer is a held-out human set you never train on.

The PyTorch block shows the basic self-rewarding loop in miniature. A policy over eight canned responses runs four rounds of DPO on pairs labeled by its own length-biased judge. The previous round's model is the reference each round.

The judge's score climbs from 0.857 to 1.320, and the policy settles on response 7, which is long and good, with a gold score of 0.508. The short response 3, which a careful reader would prefer, has a gold score of 0.768.

Notice the last round. The policy has concentrated on one response. So the best and worst of its four samples are the same response on almost every prompt, and only 2 of 512 pairs survive. This is one way a self-rewarding loop saturates, the problem Wu et al. set out to fix. Once the policy agrees with its judge, the judge has nothing left to teach, whether or not the judge was right. In practice, the number of non-tied pairs per round is a cheap alarm for a self-rewarding loop that has stopped learning.

Model collapse, in a model you can solve

Now remove the checker entirely. Model collapse is the loss of quality when each generation of a model is trained on samples from the previous generation, with nothing to check what is kept. §8.4 derived the simplest case. Fit a Gaussian to $M$ samples drawn from the previous fit, and repeat. The variance shrinks by a factor $(M-1)/M$ on average each generation, so it decays to zero and the tails go first. That section also stated that keeping all the old data, instead of replacing it, avoids this.

Here we prove both behaviors in linear regression, a model rich enough to have features and noise, following Dohmatob, Feng & Kempe (2024) and Gerstgrasser et al. (2024). The proofs are short, and they are a common interview question.

One fact makes accumulation easy to handle. The least-squares fit solves the normal equations $\mathbf{X}^\top\mathbf{X}\,\mathbf{w} = \mathbf{X}^\top\mathbf{y}$, where $\mathbf{X}^\top\mathbf{X}$ is called the Gram matrix. Stacking $t$ copies of $\mathbf{X}$ gives the Gram matrix $t\,\mathbf{X}^\top\mathbf{X}$ and the right side $\sum_i\mathbf{X}^\top\mathbf{y}_i$. So the accumulated fit is

In plain words, fitting all the stacked data is the same as fitting the average label of each input once.

The two theorems below are this example, run for any number of generations.

In plain words, under replacement each generation adds a full dose of fresh noise and keeps all the old doses, so the error grows linearly forever. Under accumulation the newest generation is only a $1/t$ share of the training set. Its noise enters with coefficient $1/t$ and its variance with $1/t^2$, and those weights have a finite sum. Synthetic data does not stop being noisy. Each new dose of noise is diluted by everything that came before, including the real data. In practice, this is the argument for never deleting the real data a model started from.

The cell checks both theorems with $d_f = 5$ features, $m = 20$ samples, and noise $\sigma = 1$, averaged over 2,000 random designs. There generation 1 has error $d_f/(m - d_f - 1) = 5/14 = 0.357$.

import numpy as np

def collapse_errors(X, w_star, sigma, generations, mode, reps=2000, seed=0):
    rng = np.random.default_rng(seed)
    A = np.linalg.solve(X.T @ X, X.T)                    # (d_f, m), the OLS map
    errs = np.zeros(generations)
    for r in range(reps):
        w = w_star.copy()

Accumulating everything forever has a cost, since every generation trains on a bigger dataset. Kazdan et al. (2024) studied a middle ground with a fixed compute budget, accumulate-subsample. Real and synthetic data pile up, but each generation trains on a fixed-size random subset of the pile. Their experiments covered Gaussian estimation, kernel density estimation, and language-model fine-tuning. In all of them the test loss of this workflow sat between the other two, degrading slowly rather than explosively. And Dey & Donoho (2024) showed that the $\pi^2/6$ bound for accumulation is not an accident of linear regression. It holds across a broad family of standard statistical models.

The last experiment holds a lesson the theorems do not state. With $\kappa m$ real rows per generation, the real labels pin down only the directions of feature space that those rows span. Two rows cannot span five dimensions, so in the other three directions the model still drifts as if no real data were there. Five real rows are not quite enough either. They span all five directions, but in some designs they cover one of them only weakly, and the error levels off only from about ten rows. Fresh real data must cover what the model is supposed to know. An interviewer who asks "how much real data is enough?" wants to hear both a fraction and a coverage condition.

Tails disappear first

Linear regression with Gaussian noise collapses through variance alone, because least squares is unbiased. Real generators also add a bias, and it has a direction. They under-sample what is rare. Shumailov et al. saw the tails of the original distribution vanish first. The categorical simulation of §8.4, in which a model is a list of category probabilities refit by counting its own samples, showed why. A category that draws zero samples once is gone for good.

What does losing the tail cost a model that is still trained on plenty of data? A simple model of learning by memorization gives an exact answer (Hutter 2021). Dohmatob et al. (2024) used it to show how model collapse changes a scaling law, the power law that relates test error to the amount of training data.

The model has a world of facts with very unequal popularity. Rank the facts from most to least often asked about. Under a Zipf law of exponent 2, fact $i$ is asked about with probability proportional to $1/i^2$. Normalized, fact 1 is 61% of all queries, fact 10 is 0.61%, and fact 100 is 0.0061%. A learner that has seen 1,000 queries has almost surely met fact 10, and has probably never met fact 100.

The formulas for $E(D)$ and $E_{\mathbf{q}}(D)$ are exact for this idealized model, and the derivation below gives their large-$D$ form. The model's value is that it isolates one mechanism, rare knowledge appearing too seldom to be learned.

In plain words, in a heavy-tailed world, data are valuable in proportion to the rare facts they contain, and a generator that has lost its tail can teach only what it knows. Its data are worth as much as real data until the learner has absorbed everything the generator knows, and worth nothing after that. With $\beta_z = 2$ the constant is $C_2 = \Gamma(1/2)/(2\sqrt{\pi^2/6}) = 1.7725/2.565 = 0.691$, so the clean error falls like $0.691/\sqrt D$.

Truncated generators arise in two ways in practice. Decoding choices cut tails on purpose. Temperature below 1, top-$p$, and top-$k$ sampling all trade diversity for quality. Shumailov et al.'s language-model experiment generated each generation's data by 5-way beam search, which tracks only the five most likely partial outputs and returns the most likely one. That is about as tail-cutting as decoding gets.

Finite training data cut tails by accident. A generator trained on $D_0$ real samples has seen only about the top $D_0^{1/\beta_z}$ facts. So its synthetic data are worth at most what $D_0$ real samples were worth. Dohmatob et al. also showed that repeating the loop compounds the loss generation by generation.

In practice, this is the argument against low-temperature sampling when diversity matters. It is also the argument for measuring how well synthetic data cover rare topics, not only their average quality. An interviewer will ask why more synthetic data stopped helping, and the plateau at the tail mass is the answer.

import numpy as np

def zipf_missed_mass(beta_z, V, D, k=None, reps=20, seed=0):
    p = np.arange(1, V + 1, dtype=float) ** -beta_z
    p /= p.sum()
    return float(np.sum(p * (1 - p) ** D))      # placeholder: the exact answer for untruncated data, no simulation, ignores k

def fit_exponent(Ds, errors):

How much fresh real data keeps a loop stable

Accumulation keeps all the real data. What if, instead, each generation gets a fixed fraction $\kappa$ of fresh real data and draws the rest from the current model? Go back to the categorical loop of §8.4 (built in Exercise 8.4d). A model is a list of category probabilities, and each generation refits them by counting $n$ samples. Now each generation draws $\kappa n$ samples from the true distribution $\mathbf{p}_0$ and $(1-\kappa)n$ from the current fit $\mathbf{p}_t$. Track one category $i$, with true probability $\bar p = p_{0,i}$ and fitted share $p_t = p_{t,i}$ after generation $t$.

In plain words, fresh data does not stop the drift, but it pulls the fit back toward the truth every generation. The two forces balance at an error equal to that of a single fit on $n_{\text{eff}}$ real samples, and for small $\kappa$, $n_{\text{eff}} \approx 2\kappa n$. With $n = 500$ and $\kappa = 0.1$, $n_{\text{eff}} = 95.9$. The loop is permanently about as accurate as one fit on 96 real samples. That is worse than the 500 samples it uses each time, and infinitely better than collapse. Population geneticists know the same mathematics as genetic drift with a steady trickle of migrants, the Wright-Fisher model with migration.

In practice, a fixed floor of fresh real data in every generation is what turns a collapsing loop into a stable one, and $2\kappa n$ tells you how much accuracy that floor buys.

import numpy as np

def collapse_with_real(p0, n, generations, kappa, seed=0):
    rng = np.random.default_rng(seed)
    p0 = np.asarray(p0, dtype=float)
    p = p0.copy()
    support, kl = [], []
    for t in range(generations):

The cleanest general statement is due to Bertrand et al. (2024). Retraining a generative model on a mix of real data and its own samples is stable as long as the synthetic share is small enough. For a model that can fit the data perfectly, that means roughly fewer than one synthetic sample per two real ones. They also proved the converse for the simplest case. A Gaussian refit only on finite samples of its own output always collapses, its covariance shrinking to zero.

Weighing the evidence

The theorems say when a loop is safe in idealized models. This last part lists the ways real loops fail without anyone noticing, reads the experiments on real models, and puts every workflow of the section in one table.

What do real pipelines show? In short, they show what the theorems predict. Replacing data collapses real models, keeping some real data limits the damage, accumulating avoids it, and a verifier prevents it. One more finding matters for practice. Synthetic data helped most when real data were scarce, and when real data were plentiful, adding more synthetic data usually hurt. These are empirical findings in specific settings, and the box below gives the details.

Is the internet collapsing? The web behaves like accumulation, since old human text is not deleted and dated snapshots of the crawl exist. So the strong form of collapse is unlikely. But three weaker effects are real.

• The fraction of fresh human text falls, so new knowledge arrives more slowly.
• The rarest facts and styles are the first to be underrepresented, which bends data scaling laws long before anything collapses.
• A lab that cannot tell model-written text from human text cannot choose its mix. That is why provenance, knowing where each document came from, is worth paying for, along with dated crawls and verification.

The 2024 frontier reports that described their data in detail (Llama 3, Nemotron-4, phi-4) all relied heavily on synthetic data. All of them also anchored it in real data and checked it.

Read more

• Yuan et al. (2024). Self-Rewarding Language Models. The loop, the judge prompt, and the numbers, including the length growth, in a short and clear paper.
• Wu et al. (2024). Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judge. Training the judge with a meta-judge, and the length-aware pair selection that kept responses from growing.
• Wang et al. (2024). Self-Taught Evaluators. How to build preference pairs with known answers and bootstrap a judge with no human labels.
• Chen et al. (2022). CodeT: Code Generation with Generated Tests. Dual execution agreement, the consensus idea behind verifying code when the tests are generated too.
• Gerstgrasser et al. (2024). Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Data. The accumulation theorem with its proof in Appendix B, and experiments on language models, diffusion models, and VAEs.
• Dohmatob, Feng & Kempe (2024). Model Collapse Demystified: The Case of Regression, with Dohmatob et al. (2024), A Tale of Tails: Model Collapse as a Change of Scaling Laws. Exact formulas for collapse in regression, and the tail-cutting theory of bent scaling laws.
• Shumailov et al. (2024). AI models collapse when trained on recursively generated data. The paper that named model collapse, with experiments on Gaussian mixtures, VAEs, and OPT-125m (preprint at arXiv:2305.17493).
• Bertrand et al. (2024). On the Stability of Iterative Retraining of Generative Models on their own Data. The sufficient condition for stability in terms of the real-data fraction, with experiments on normalizing flows and diffusion models (the models of §33.2).

Watch next

▶

▶

▶