Meta trained Llama 3 8B on about 15 trillion tokens. By the rule of §9.3, a compute-optimal 8-billion-parameter model needs about 160 billion, and Meta's announcement said as much. "The Chinchilla-optimal amount of training compute for an 8B parameter model corresponds to ~200B tokens", yet "model performance continues to improve even after the model is trained on two orders of magnitude more data." Roughly 1,900 tokens per parameter, almost a hundred times the Chinchilla ratio. On Chinchilla's own terms, that run spent far more compute than a bigger model would have needed to reach the same loss.

It was not a mistake. Chinchilla answers a narrow question, which model reaches the lowest loss for a fixed training budget. Real models are trained once and then served billions of times, and they are trained on data that is finite, uneven in quality, and eventually used for something other than predicting the next token. This section relaxes Chinchilla's assumptions one at a time. First inference cost, which pushes models smaller and longer. Then finite data, what repeated epochs and surplus parameters are worth, and how to split a budget when the unique data is fixed. Then data quality as a multiplier on compute. And finally, scaling laws for what happens after pretraining, on downstream tasks and in fine-tuning.

Training once, serving forever

A language model's lifetime cost has two parts. Training costs $6ND_{\text{tr}}$ FLOPs for $N$ parameters and $D_{\text{tr}}$ training tokens (§9.2). Serving costs about $2N$ FLOPs per token processed, the forward pass only (§5.6). If the model will process $D_{\text{inf}}$ tokens over its lifetime, prompts and generated tokens together, the total is

Training cost depends on both $N$ and $D_{\text{tr}}$, but inference cost depends only on $N$. A model that will be used heavily should therefore be smaller, with the lost quality bought back with more training tokens. Sardana et al. (2024) turned this into a scaling law. Fix a target quality, a loss $\ell$, and minimize the total cost over all $(N, D_{\text{tr}})$ that reach it, using the Chinchilla loss $\hat{\mathcal{L}}(N, D) = E + A/N^\alpha + B/D^\beta$ of §9.3,

In plain words, among all the models good enough for the job, pick the one that is cheapest to build and run.

The iso-loss curve

The constraint can be solved for the training tokens. For a model of size $N$, the loss reaches $\ell$ when $B/D_{\text{tr}}^\beta = \ell - E - A/N^\alpha$, so

In plain words, a smaller model needs more tokens to reach the same loss, and below $N_{\min}$ it cannot reach it at all, because its size penalty $A/N^\alpha$ alone already exceeds the allowed excess $\ell - E$. As $N$ approaches $N_{\min}$ from above, the required data explodes.

How much extra training does a smaller model need? The cell takes the compute-optimal model for a $10^{23}$-FLOP budget, with the replication's constants from §9.3, and shrinks it.

A model three quarters the size costs only 3 percent more to train. Half the size costs 26 percent more and trains on about 100 tokens per parameter. At 30 percent of the size it costs 2.6 times as much, at a fifth 14 times as much, and below about 15 percent of the optimal size the target is unreachable. The curve is flat near the optimum and then turns into a wall. Over-training is cheap in moderation and ruinous in excess.

The optimality condition

Where on this curve is the total cost lowest? Substitute $D_{\text{tr}}(N)$ into the cost and differentiate with respect to $N$. Write $g(N) = \ell - E - AN^{-\alpha}$, so $D_{\text{tr}} = (B/g)^{1/\beta}$ and $g$ equals the data penalty $B D_{\text{tr}}^{-\beta}$. By the chain rule,

Setting the derivative of $6ND_{\text{tr}}(N) + 2ND_{\text{inf}}$ to zero gives $6D_{\text{tr}} + 6N\,dD_{\text{tr}}/dN + 2D_{\text{inf}} = 0$, and substituting,

In plain words, without inference ($D_{\text{inf}} = 0$) this is Chinchilla's balance condition from §9.3, size penalty over data penalty equals $\beta/\alpha$. With inference, the optimum tolerates a larger size penalty and a smaller data penalty, a smaller model trained longer, by the factor $1 + D_{\text{inf}}/(3D_{\text{tr}})$. If the model will serve as many tokens as it was trained on, the factor is $4/3$. If it serves a hundred times more, the factor is about 34.

There is no closed-form solution, because $D_{\text{tr}}$ appears on both sides. Sardana et al. solve it numerically, and so will you.

import numpy as np

def inference_optimal(target, D_inf, E, A, B, alpha, beta):
    G = (alpha * A / (beta * B)) ** (1 / (alpha + beta))       # placeholder: ignores inference entirely
    a = beta / (alpha + beta)
    lo, hi = 10.0, 40.0
    for _ in range(100):                                        # find the compute whose optimum hits the target
        mid = (lo + hi) / 2; C = 10 ** mid; N = G * (C / 6) ** a; D = C / (6 * N)

What the numbers say

Sardana et al. give three examples, which your solution reproduces. A team planning a 13-billion-parameter Chinchilla-optimal model that expects 2 trillion tokens of inference should instead train a 7-billion-parameter model on about twice the data, saving about $1.7\times10^{22}$ FLOPs, 17 percent of the total. For a 7B-quality model with only $10^{11}$ inference tokens the optimum barely moves, to 6B on 1.18 times the data. And for a 30B-quality model serving $10^{13}$ tokens, the cheapest option is a 13.6B model on 2.84 times the data, with total FLOPs 28 percent lower.

FLOPs understate the case for small models. Generating tokens one at a time is limited by memory bandwidth, not arithmetic (§6.5), so each inference FLOP is far more expensive than a training FLOP. Sardana et al.'s cost model assumes 50 percent utilization for training and prompt processing, but about 1 percent for generation. In dollars, the optimum moves to even smaller, longer-trained models than in FLOPs. The widget lets you scale the cost of an inference FLOP to see this.

Over-training in practice

This reasoning explains the shape of nearly every open model family since 2023. The LLaMA paper said it directly. Chinchilla's objective "disregards the inference budget", and "given a target level of performance, the preferred model is not the fastest to train but the fastest at inference" (Touvron et al. 2023). LLaMA's 7B and 13B models trained on 1 trillion tokens, about 150 and 77 per parameter, and the authors noted that "the performance of a 7B model continues to improve even after 1T tokens". Llama 2 trained every size on 2 trillion. Llama 3 trained its 8B and 70B models on about 15 trillion, and its paper notes that "we also train our smaller models for much longer than is compute-optimal".

By the replication's fit (on MassiveText, so these are illustrations, not claims about Meta's data), Llama 3 8B spent several times the compute a compute-optimal model would need for the same loss. In exchange every token it serves costs a fraction of what the larger model would charge. For a model downloaded and run by millions of people, that trade is easily worth it.

Two cautions come with the extreme ratios. First, the Chinchilla fit was made on runs near 20 tokens per parameter, and extrapolating it to 1,000 or 10,000 is risky. Sardana et al. trained 47 models at up to 10,000 tokens per parameter and found that loss keeps falling, with no plateau, but more slowly than the fit predicts. Fitted only on typical ratios, the Chinchilla form "overestimates the impact of additional training data" as runs move to extreme ratios. Second, if you plan to over-train, fit your scaling law in the over-trained regime. Gadre et al. (2024) did exactly that. With the token multiplier $M = D/N$, they found that at a fixed $M$ the loss still follows a power law in compute, $\mathcal{L}(C, M) = E + (aM^{\eta} + bM^{-\eta})\,C^{-\eta}$, with the same exponent for every $M$. That let them predict a 1.4-billion-parameter model trained on 900 billion tokens, 32 times over-trained, from runs that used 300 times less compute.

def overtraining_overhead(k, C, E, A, B, alpha, beta):
    return 1.0 / k              # placeholder: as if a smaller model needed only proportionally more tokens

When the data runs out

Chinchilla assumes every training token is new. For the largest models that assumption is breaking. A compute-optimal model at $10^{26}$ FLOPs wants about 20 trillion tokens, and an over-trained one far more, while the stock of high-quality public text is finite. Villalobos et al. (2024) project that, at current trends, models will be trained on datasets roughly as large as the whole stock of public human text sometime between 2026 and 2032. The natural response is to repeat data. How much is a second, fourth, or sixteenth epoch worth?

Muennighoff et al. (2023) trained over 400 models, up to 9 billion parameters and 900 billion tokens, for up to 1,500 epochs, and found a clean answer. Write $U_D$ for the number of unique tokens and $R_D$ for the number of repetitions, epochs minus one. Repeated tokens are worth less than new ones, with a value that decays exponentially in the number of repetitions, so the effective data is

In plain words, the first pass counts in full. Each further pass is worth a little less than the previous one, and the total value of all repetitions together can never exceed $R_D^\ast$ more copies of the unique data. Fitted to their runs, $R_D^\ast = 15.39$.

Two limits check the formula. With no repetition, $R_D = 0$ and $D' = U_D$. For small $R_D$, $1 - e^{-x} \approx x$, so $D' \approx U_D(1 + R_D)$, the full value of every epoch. As $R_D \to \infty$, $D' \to U_D(1 + R_D^\ast) \approx 16.4\,U_D$, the most any amount of repetition can be worth.

Where does an exponential come from? The paper's own derivation is short and worth knowing, because an interviewer may ask you to produce the formula rather than recite it.

In plain words, $R_D^\ast$ is the decay time of a token's value, counted in passes. After $R_D^\ast$ repetitions, the next pass is worth $e^{-1} \approx 37$ percent of fresh data, and the total can never pass $1 + R_D^\ast$ unique epochs.

The headline numbers are worth memorizing. "Training with up to 4 epochs of repeated data yields negligible changes to loss compared to having unique data." Meaningful gains continue up to about 16 epochs, and by about 40 epochs, in the words of the paper's first figure, "repeating is worthless". Beyond that, "the value of adding compute eventually decays to zero."

Four epochs keep 93 percent of the value of fresh data. Sixteen epochs keep 66 percent on average, and by a hundred epochs the average token is worth about a sixth of a new one, while the ceiling of 16.4 times the unique data never moves. The marginal value falls faster. The pass after $R_D$ repetitions adds about $e^{-R_D/R_D^\ast}$ of a fresh epoch, the derivative of $D'/U_D$ with respect to $R_D$. That is 82 percent for the fourth pass, 38 percent for the sixteenth, and under 2 percent for the sixty-fourth.

def effective_data(unique_tokens, total_tokens, R_star=15.387756):
    return total_tokens          # placeholder: treats repeated tokens as new

def repeated_loss(N, unique_tokens, total_tokens, E, A, B, alpha, beta):
    return E + A / N ** alpha + B / total_tokens ** beta

Excess parameters, and the full law

Repetition is half the story. With a fixed set of $U_D$ unique tokens, you can also spend more compute on a bigger model, and that runs into diminishing returns too, because extra parameters keep learning the same features from the same data. Muennighoff et al. treat model size the way they treat data. Let $U_N$ be the compute-optimal size for the unique data, the Chinchilla $N_{\text{opt}}$ whose matching $D_{\text{opt}}$ equals $U_D$ (or $N$ itself, if the model is smaller than that). Parameters beyond $U_N$ count as repetitions, $R_N = \max(N/U_N - 1, 0)$, and are worth

The full data-constrained law is Chinchilla's with both substitutions,

In plain words, a model is charged for its size and its data as before, except that repeated tokens and surplus parameters are discounted, each with its own decay time. With no repetition and a model no bigger than $U_N$, it is exactly the Chinchilla law of §9.3.

The fit came in two stages. First the Chinchilla constants, on single-epoch runs on C4, their training data, with $\alpha$ and $\beta$ tied because Hoffmann et al. had found equal scaling on C4. That gave $E = 1.87$, $A = 520.8$, $B = 1487.7$, and $\alpha = \beta = 0.353$. Then, holding those fixed, $R_N^\ast = 5.31$ and $R_D^\ast = 15.39$ from 182 runs between 7 million and 9 billion parameters and 1 to 500 epochs. With $\alpha = \beta$ the optimum of §9.3 has $N_{\text{opt}}/D_{\text{opt}} = G^2 = (A/B)^{1/\alpha}$, so $U_N = 0.051\,U_D$, about 19.6 tokens per parameter.

Because $R_N^\ast < R_D^\ast$, surplus parameters lose their value about three times faster than surplus epochs. The paper's conclusion follows. When data is the constraint, "the data-constrained efficient frontier suggests allocating most additional compute to more epochs rather than more parameters", where Chinchilla would scale both equally.

Spending a budget when the data is fixed

Here is the interview version. You have 25 billion unique tokens and $9.3\times10^{21}$ FLOPs. Ignoring the data limit, Chinchilla (with the C4 constants) says 8.9 billion parameters on 174 billion tokens, seven epochs. The cell finds the size that minimizes the data-constrained law on the same budget, $D = C/(6N)$, and then keeps adding compute with the data held fixed. It uses its own names for the C4 constants, so the earlier cells keep theirs.

At $9.3\times10^{21}$ FLOPs the law prefers a 6.8-billion-parameter model on nine epochs, about a quarter smaller than Chinchilla's choice. Muennighoff et al. trained that comparison for real. Their frontier picked 6.3 billion parameters on 242 billion tokens, and "despite having 27% fewer parameters, this model achieves better loss and downstream performance" than the 8.7-billion-parameter Chinchilla allocation. Notice how small the predicted gain is, about 0.002 nats. The valley is flat, as it was in §9.3, and the case for the smaller model is as much about cheaper inference as about loss.

The rest of the table shows what fixed data does to a growing budget. From $10^{22}$ to $10^{24}$ FLOPs the optimal model grows about eightfold while its epochs grow about thirteenfold, so the extra compute goes mostly into repetition, as the paper says. And it buys very little. The last factor of ten lowers the loss by under 0.01 nats, because both discounted terms are near their ceilings, $D' \le 16.4\,U_D$ and $N' \le 6.3\,U_N$. With 25 billion tokens, a budget much beyond $10^{23}$ FLOPs is better spent finding more data. The same logic at the extreme is in the paper's 100-million-token experiments, where the best loss came from about 20 to 60 times more parameters and epochs than the single-epoch optimum, roughly 7,000 times more compute, for a loss more than 50 percent lower than one epoch gives.

import numpy as np

def dc_loss(N, D, U):
    return DC["E"] + DC["A"] / N ** DC["alpha"] + DC["B"] / D ** DC["beta"]   # placeholder: repeats count as new

def data_constrained_optimum(C, U):
    N = G * (C / 6) ** 0.5                                                   # placeholder: Chinchilla, ignoring U
    return N, C / (6 * N)

Where the formula breaks

The law is a fit, and its authors list its limits. It cannot predict harm. By construction more epochs or parameters can only plateau, never make the loss worse, yet in their runs models trained on too many repetitions did get worse, and some loss curves "diverge if too many epochs are used". They removed such runs as outliers before fitting, and some runs showed double descent, the loss rising and then falling again as epochs grew. Treat the formula as an upper bound on what repetition buys. The largest budgets in the cell above, at 70 to 120 epochs, are exactly where that caution applies.

It also assumes the whole dataset is repeated evenly. Repeating a small part of it many times is far worse. Hernandez et al. (2022) found that repeating 0.1 percent of the data a hundred times made an 800-million-parameter model perform like one half its size (§8.3). In a mixture, each domain has its own epoch count, $e_i = w_iD/N_i$ (§8.4). Applying the effective-data formula domain by domain, so that no domain passes a few epochs, is a reasonable heuristic, not a result from the paper.

And $R_D^\ast$ belongs to a model and a recipe, not to the data. Kim et al. (2025) found that when data is the constraint, much stronger regularization pays (a weight decay about 30 times the usual value), together with ensembles of independently trained models. Combined with epoching and larger models, their best recipe was worth 5.17 times as much data as the standard one. Prabhudesai et al. (2025) fitted the same law to masked diffusion language models, which predict randomly masked tokens in any order rather than left to right, and found $R_D^\ast$ near 500 against about 32 for autoregressive models trained the same way. In their runs repeated data stayed nearly as good as fresh for up to about 100 epochs, against about 4. An objective that sees each sequence in many orders extracts more from every repetition.

Stretching unique data further

Muennighoff et al. also tested two ways to stretch a fixed supply of text. Filling up to half of the training data with code caused no measurable harm on natural-language tasks, which in their accounting doubles the effective tokens available. And filtering by perplexity helped, while deduplication alone did not, mainly on noisy data. Beyond that lie synthetic data, rephrasing, and distillation from a larger model, all discussed in §8.4, where the risks of training on model outputs are also covered.

Data quality as a compute multiplier

Every law in this chapter treats a token as a token. But the constants of a scaling law are properties of the data. Better data shifts the whole curve down, so the same loss, or the same downstream accuracy, is reached with less compute. A useful way to think about a better data pipeline is as a compute multiplier, the factor of compute it saves at a fixed quality. Filtering web text with model-based quality classifiers, as FineWeb-Edu and DCLM do (§8.2), is among the largest such multipliers reported for language models.

Quality also changes the optimal allocation. DeepSeek's scaling study (DeepSeek-AI 2024) fit compute-optimal exponents on three datasets and found $a = 0.450$ on an early version of its data, $0.524$ on its improved data, and $0.578$ on OpenWebText2, concluding that the higher the quality of the data, the more of a growing budget should go to the model rather than to more tokens.

And quality interacts with quantity. Filtering harder leaves fewer unique tokens, which then have to be repeated. The cell compares two hypothetical pipelines for a 7-billion-parameter model trained on 4 trillion tokens. One keeps 4 trillion unique tokens, the other keeps the best quarter, assumed (for illustration only) to be worth twice as much per token, and repeats it four times.

Under these assumptions, four epochs of the filtered quarter beat one pass over everything, because repetition at four epochs costs only 7 percent while quality doubled the value. Sixteen epochs of an even smaller set give back about half of that gain. Real quality multipliers have to be measured, not assumed, and they need not stay constant as data repeats, but this is the shape of the decision every pretraining team faces.

There is even evidence that good data selection can beat power-law scaling altogether. Sorscher et al. (2022) showed theoretically, and in vision experiments, that pruning a dataset with a good enough measure of example difficulty can make error fall exponentially rather than as a power law in the amount of data kept. Finding such a measure at the scale of web text remains open.

Scaling laws after pretraining

Pretraining loss is a means, not an end. What matters is how well the model does downstream, after fine-tuning, on the tasks people care about. Scaling laws exist here too, though they are less mature.

Downstream accuracy from loss. Gadre et al. (2024) related the average top-1 error of a model over 17 downstream tasks to its loss, with $\text{Err}(\mathcal{L}) = \epsilon - k\,e^{-\gamma\mathcal{L}}$, and predicted it with 20 times less compute than the runs being predicted. Because $e^{-\gamma\mathcal{L}} = \operatorname{PPL}^{-\gamma}$, this is a power law in perplexity. §9.6 builds a two-step predictor of this kind, the method Llama 3 used.

Pretraining as a data multiplier for fine-tuning. Hernandez et al. (2021) asked how much a pretrained model's knowledge is worth in fine-tuning data. They defined the effective data transferred $D_T$, the amount of extra fine-tuning data a model trained from scratch would need to match the pretrained model, and found

where $D_F$ is the fine-tuning data and $D_T$ is measured in the same units, characters of text in their experiments, and $N$ the non-embedding parameters (here $\alpha$ and $\beta$ are this paper's exponents, not Chinchilla's). For transfer from English text to Python code they fit $k = 1.9\times10^{4}$, $\alpha = 0.18$, and $\beta = 0.38$. In plain words, in the low-data regime pretraining "effectively multiplies the fine-tuning dataset", and since $\beta \approx 2\alpha$, a tenfold larger model is worth about a hundredfold more fine-tuning data. In their Figure 1, for a 40-million-parameter model fine-tuned on 300,000 characters of Python, the effective data transferred was about a thousand times the fine-tuning set.

def transfer_multiplier(D_F, N):
    raise NotImplementedError

def size_equivalent(factor):
    raise NotImplementedError

Scaling laws for fine-tuning itself. Zhang et al. (2024) fine-tuned models from 1 to 16 billion parameters on translation and summarization, with full fine-tuning, prompt tuning, and LoRA (§14.5), and found a multiplicative joint law, $\hat{\mathcal{L}}(X, D_f) = A\,X^{-\alpha}D_f^{-\beta} + E$, where $D_f$ is the fine-tuning data and $X$ is whichever factor is scaled (model size, pretraining data, or the number of tunable parameters). Their findings are practical. Fine-tuning "benefits more from LLM model scaling than pretraining data scaling", scaling the number of parameter-efficient tuning parameters, such as the LoRA rank, "is generally ineffective", and the best fine-tuning method "is highly task- and finetuning data-dependent".

What changes, and what does not

In PyTorch

The inference-aware optimum with autograd doing the work. We optimize $\log N$ by gradient descent on the total cost, with $D_{\text{tr}}$ computed from the iso-loss curve inside the graph, and check the optimality condition at the end.

Read more

• Sardana et al. (2024). Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws. The inference-aware objective, its numerical solution, and experiments up to 10,000 tokens per parameter.
• Muennighoff et al. (2023). Scaling Data-Constrained Language Models. The effective data formula, its fitted constants, and the value of code and filtering.
• Gadre et al. (2024). Language models scale reliably with over-training and on downstream tasks. Scaling laws in the over-trained regime and from loss to downstream error.
• Touvron et al. (2023). LLaMA: Open and Efficient Foundation Language Models. The argument for training small models long, in the introduction.
• Hernandez et al. (2021). Scaling Laws for Transfer. Effective data transferred, and pretraining as a multiplier on fine-tuning data.
• Zhang et al. (2024). When Scaling Meets LLM Finetuning. A joint scaling law for fine-tuning across model size, data, and method.
• Kim et al. (2025). Pre-training under infinite compute. Regularization, ensembles, and distillation as ways to get more from fixed data.
• Prabhudesai et al. (2025). Diffusion Beats Autoregressive in Data-Constrained Settings. The data-constrained law refitted for masked diffusion models, with a far larger decay constant.
• Villalobos et al. (2024). Will we run out of data? Projections of when training sets reach the stock of public human text.
• Sorscher et al. (2022). Beyond neural scaling laws: beating power law scaling via data pruning. How data selection could change the exponent itself.

Watch next

▶

▶