In late 2021 DeepMind trained Gopher, a 280-billion-parameter model, on 300 billion tokens. It followed the conventional wisdom of the time, which came from Kaplan's laws (§9.2). Spend extra compute mostly on parameters, and stop well short of convergence. Then the same group asked whether Gopher's compute could have been spent better. They trained over 400 models, from 70 million to over 16 billion parameters on 5 to 500 billion tokens, and found that it could. Gopher was about four times too big for its budget.

To prove it, they trained Chinchilla, a 70-billion-parameter model on 1.4 trillion tokens, with the same compute as Gopher. Chinchilla beat Gopher on nearly every benchmark they measured, reaching 67.6 percent on MMLU against Gopher's 60.0, while being four times cheaper to run. The paper, Training Compute-Optimal Large Language Models (Hoffmann et al. 2022), changed how every lab sizes its models.

This section works through it completely. We look at the three independent ways the paper found the optimum, write down its parametric loss with the fitted constants, derive the compute-optimal model size and token count with Lagrange multipliers, run an isoFLOP analysis ourselves, and then examine the famous 20-tokens-per-parameter rule, where it comes from, when it fails, and what a 2024 replication found when it refit the data.

Three ways to find the optimum

The question is the one Kaplan asked. Given a training budget of $C$ FLOPs, with $C \approx 6ND$ for $N$ parameters and $D$ tokens (§9.2), which $N$ and $D$ give the lowest final loss? Hoffmann et al. answered it three different ways, so that no single method's assumptions would decide the result. Each produces power laws $N_{\text{opt}} \propto C^{a}$ and $D_{\text{opt}} \propto C^{b}$, with $a + b = 1$ because $N D \propto C$.

Approach 1, fix the model sizes and vary the training length. Train a range of model sizes (70 million to over 10 billion parameters), each with four different cosine schedule lengths covering a 16-fold range of training tokens, decaying the learning rate tenfold over each. Every run's loss curve gives a loss at every FLOP count along the way, and because each schedule ends at its own horizon, the end of each curve is a properly tuned run. At each of 1,500 log-spaced FLOP values, take the lowest loss over all runs, the envelope, and record which model size produced it. Fit a power law to those sizes.

Approach 2, isoFLOP profiles. Fix nine compute budgets from $6\times10^{18}$ to $3\times10^{21}$ FLOPs. For each budget, train models of many sizes, each with exactly as many tokens as the budget allows, $D = C/(6N)$, and plot final loss against $\log N$. Each such isoFLOP profile is a valley. Fit a parabola to it, read off the size at the bottom, and fit a power law to those minima across budgets.

Approach 3, fit a parametric loss. Fit one formula for the final loss as a function of both $N$ and $D$ to all the runs from Approaches 1 and 2, then minimize it analytically under the constraint $C = 6ND$. We derive this below.

The three answers, with the paper's 10th to 90th percentile intervals from bootstrapping, are these.

In plain words, all three methods agree that parameters and data should grow at about the same rate. Ten times the compute means about 3.2 times the parameters and 3.2 times the tokens, not Kaplan's 5.4 and 1.9. (Keep an eye on that suspiciously narrow interval for Approach 3. We will come back to it.)

Approach 1 also gives a table of compute-optimal sizes, the paper's Table 3. Its rows are the source of the most quoted rule in the field.

At Gopher's budget the optimum is about 67 billion parameters on 1.5 trillion tokens, which is why Chinchilla was 70 billion on 1.4 trillion. And at every scale the ratio is close to 20 tokens per parameter. The paper never states that rule in words, but its table does.

The parametric loss

Approach 3 fits

where $\hat{\mathcal{L}}$ is the predicted final loss of a model with $N$ parameters trained (in one pass, with a matched schedule) on $D$ tokens, $E$ is the irreducible loss, $A$ and $B$ are positive constants, and $\alpha$ and $\beta$ are positive exponents. (Here $B$ is the paper's constant, not a batch size.) Hoffmann et al. give each term a meaning. $E$ is the loss of an ideal model of the data, the entropy of text. $A/N^\alpha$ is how much a perfectly trained model of size $N$ falls short of that ideal, because its function class is finite. $B/D^\beta$ is how much training on only $D$ tokens falls short of training to convergence.

In plain words, the loss is a floor, plus a penalty for being small, plus a penalty for seeing too little data, and the two penalties add.

The fit uses the procedure from §9.1. Write $\log\hat{\mathcal{L}} = \operatorname{LSE}(a - \alpha\log N,\; b - \beta\log D,\; e)$ with $A = e^a$, $B = e^b$, $E = e^e$, minimize a Huber loss ($\delta = 10^{-3}$) between predicted and observed log losses with L-BFGS, and keep the best of a grid of starting points. The published constants are

Losses are in nats per token on DeepMind's MassiveText data with its tokenizer, so, as with Kaplan's laws, the constants are not portable to other setups. The functional form and the method are. Notice also how the form differs from Kaplan's $\mathcal{L}(N, D)$. Kaplan's combined the two terms inside a power and was fit to early-stopped runs on repeated data, while Chinchilla's adds them and describes single-epoch training.

The fit predicts what the experiment showed. At about the same compute (the two $6ND$ values differ by 17 percent, and the paper's own accounting puts both at Gopher's budget), Chinchilla's loss is lower than Gopher's, because Gopher's data term (0.251) is far larger than its size term (0.052). Gopher had more parameters than its data could use.

Deriving the optimal N and D

Now the central derivation, which every interviewer at a lab that trains models expects you to do at a whiteboard. Minimize $\hat{\mathcal{L}}(N, D)$ subject to $6ND = C$.

Form the Lagrangian with multiplier $\lambda$,

Set its partial derivatives with respect to $N$ and $D$ to zero,

Multiply the first equation by $N$ and the second by $D$. Both now contain $6\lambda ND$, so

This is the key condition, and it deserves its own sentence. In plain words, at the optimum, the marginal benefit of a few more parameters equals the marginal benefit of a few more tokens, measured per unit of log compute. Equivalently, the two reducible terms stand in a fixed ratio,

so at every compute-optimal point the size penalty is $\beta/\alpha$ times the data penalty. With Chinchilla's exponents that is $0.28/0.34 = 0.82$.

To finish, substitute the constraint $D = C/(6N)$ into $\alpha A N^{-\alpha} = \beta B D^{-\beta}$,

Collect the powers of $N$ on one side, $N^{\alpha + \beta} = \frac{\alpha A}{\beta B}\left(\frac{C}{6}\right)^{\beta}$, and take the $(\alpha+\beta)$-th root,

where $D_{\text{opt}}$ follows from $D = C/(6N)$, since $(C/6)/\big(G (C/6)^a\big) = G^{-1}(C/6)^{1-a}$ and $1 - a = b$. This is equation (4) of the paper.

In plain words, the optimal model size grows as compute to the power $\beta/(\alpha+\beta)$. The exponents swap roles. The data exponent $\beta$ sets how fast parameters grow, because if the data penalty falls quickly as tokens are added (large $\beta$), a small increase in data keeps it in balance with the size penalty, and the rest of the compute can go into the model. With $\alpha = 0.34$ and $\beta = 0.28$, $a = 0.28/0.62 = 0.452$ and $b = 0.548$.

import numpy as np

def chinchilla_optimal(C, E, A, B, alpha, beta):
    N = np.sqrt(C / 120)                    # placeholder: 20 tokens per parameter
    return N, 20 * N

def isoflop_search(C, E, A, B, alpha, beta, n=4001):
    raise NotImplementedError

Three fits, three answers for Gopher's budget

The closed form makes it easy to see how sensitive the answer is to the constants. The cell compares three versions of Approach 3 at Gopher's budget, the rounded constants printed in the paper, the unrounded values from the paper's source ($E = 1.6934$, $\alpha = 0.3392$, $\beta = 0.2849$), and the refit by Besiroglu et al. (2024) that we discuss at the end of the section.

The rounded constants say 32 billion parameters and 93 tokens per parameter, the unrounded ones 40 billion and 59, and the replication 72 billion and 18. A change in the third decimal place of the exponents moved the answer by 25 percent. The reason is that the allocation exponent multiplies a large logarithm, $\ln(C/6) \approx 53$ at this budget, so the shift of $a$ from 0.452 to 0.457 alone would move $N_{\text{opt}}$ by about 30 percent, partly offset by the change in $G$. Approach 3, as published, also disagreed with Approaches 1 and 2. Its own Table A3 implies about 61 tokens per parameter at 67 billion parameters and 94 at a trillion, not 20. Hold that thought for the replication.

An isoFLOP analysis, by hand

Approach 2 is the method most labs use today, because it is simple, needs no functional form for the whole surface, and directly answers the question asked. For each budget, train a handful of model sizes that span the expected optimum, each for exactly the tokens the budget allows. Plot the final losses against $\log N$, fit a parabola, and take its vertex. The derivation above guarantees each profile is a single convex valley in $\log N$, which is why a parabola is the right local model.

The flatness near the minimum is one of the most practically useful facts in this chapter. The cell measures it with the replication's constants. For a model $k$ times the optimal size at budget $C$, it finds the smaller budget at which a compute-optimal model would reach the same loss.

A model twice too big or twice too small is worth about 84 percent of its budget, so it wastes about 16 percent. Four times off wastes about half, and ten times off over 80 percent. That is the scale of the Gopher mistake, and also why nobody needs the optimum to more than a factor of about 1.5. This flatness is why Llama 3's team picked a 405-billion-parameter flagship when their own fit said 402 billion (Dubey et al. 2024).

import numpy as np

def isoflop_analysis(runs):
    runs = np.asarray(runs)
    budgets = np.unique(runs[:, 0])
    N_opt = np.array([runs[runs[:, 0] == C][np.argmin(runs[runs[:, 0] == C][:, 2]), 1] for C in budgets])
    a = np.polyfit(np.log(budgets), np.log(N_opt), 1)[0]
    return {"budgets": budgets, "N_opt": N_opt, "a": a}   # placeholder: argmin of the grid, no parabola

How sure is the exponent?

An isoFLOP study turns a few dozen noisy runs into a handful of vertices, then extends a line through them by several decades. How much should you trust the end of that line? Hoffmann et al. reported bootstrap intervals, 0.462 to 0.534 for Approach 2's exponent, and you can compute the same thing for your own sweep. The bootstrap of §9.1 applies with one change in what is resampled. Each budget's runs are the independent data behind that budget's parabola, so resample runs within each budget, keep the design of the study fixed (the same budgets, the same number of runs in each), refit every parabola and the line, and record both the exponent and the extrapolated optimum. A replicate in which some parabola has no minimum (it comes out flat or upside down) is dropped, and the number dropped is itself a diagnostic. If many replicates fail, the sweep does not bracket the optimum well.

import numpy as np

def isoflop_bootstrap(runs, C_target, n_boot=200, seed=0):
    runs = np.asarray(runs, float)
    budgets = np.unique(runs[:, 0])
    v = [(lambda c: -c[1] / (2 * c[0]))(np.polyfit(np.log(runs[runs[:, 0] == C, 1]), runs[runs[:, 0] == C, 2], 2)) for C in budgets]
    a, b = np.polyfit(np.log(budgets), v, 1)
    N = float(np.exp(b + a * np.log(C_target)))

On the setup's study, 49 runs at seven budgets with 0.3 percent noise in each loss, a correct solution gives a 90 percent interval of about 0.507 to 0.533 for $a$ (the truth is 0.513), and 90 to 119 billion parameters for the optimum at $10^{24}$ FLOPs, three decades past the largest budget (the truth is 96 billion). A spread of 0.026 in the exponent, multiplied by the $\ln 10^3 \approx 6.9$ of three decades, is already a factor of $e^{0.18} \approx 1.2$ in the extrapolated size, and the uncertainty in the line's intercept brings the whole interval to a factor of 1.3. That is why labs extend their isoFLOP ladders as far up as they can afford, and why an interval as narrow as 0.454 to 0.455 deserves suspicion, as the replication at the end of this section shows.

Fitting the whole surface

Approach 3 needs the two-variable version of the fit from §9.1. It is worth doing once yourself, because it shows how a few hundred runs become a formula you can minimize, and because it is where Chinchilla's own analysis went wrong.

def fit_chinchilla(N, D, L):
    return 1.69, 406.4, 410.7, 0.34, 0.28        # placeholder: Chinchilla's constants, whatever the data

The 20 tokens per parameter rule

The most quoted result of the paper is a rule of thumb it never wrote down. Train about 20 tokens per parameter. It comes from Approach 1's table, where the ratio sits between 20 and 22 from 400 million to 10 trillion parameters, and from Chinchilla itself, 70 billion parameters on 1.4 trillion tokens. Combined with $C = 6ND$, it gives the planning formula from §9.2,

In plain words, for a compute-optimal model, the number of tokens is about twenty times the number of parameters, and both grow as the square root of compute. At $10^{24}$ FLOPs that is about 91 billion parameters on 1.8 trillion tokens.

The rule is useful and widely misused. Here are its caveats, each of which an interviewer may probe.

1. It optimizes training compute only. It says nothing about the cost of serving the model. A model that will answer billions of queries is cheaper overall if it is smaller and trained longer, which is why almost no production model is compute-optimal. That is the subject of §9.4.
2. It depends on the data, tokenizer, and architecture. The ratio is a property of MassiveText, its tokenizer, and 2021 training recipes. DeepSeek's scaling study (DeepSeek-AI 2024) found allocation exponents $a$ from 0.45 to 0.58 depending on the dataset and concluded that higher-quality data should push more compute into model size. Llama 3's team fitted their own law on their own data, which recommended about 400 billion parameters on 16.55 trillion tokens at $3.8\times10^{25}$ FLOPs, about 41 tokens per parameter. And MiniCPM, trained with a warmup-stable-decay schedule (§8.7), reported a compute-optimal ratio of about 192.
3. It assumes unique data. Every token is seen once. When you run out of unique data, repetition changes the picture (§9.4).
4. It is a statement about loss. Downstream abilities do not always track loss one for one (§9.6).
5. The ratio may drift with scale. Unless $\alpha = \beta$ exactly, $D_{\text{opt}}/N_{\text{opt}}$ grows or shrinks as a small power of $C$, so a ratio measured at $10^{21}$ FLOPs need not hold at $10^{26}$.
6. The optimum is flat. Being a factor of 1.5 off costs only a few percent of compute, so "20" is a center of a broad valley, not a sharp target.

The cell below plans three real budgets with Approach 1's law, $N_{\text{opt}} = 67\text{B}\times(C/5.76\times10^{23})^{0.5}$, and compares them with what was actually built.

GPT-3 was about three and a half times too big for its compute by Chinchilla's standard. Chinchilla sits on its own rule. Llama 3 405B is smaller than Chinchilla's extrapolation and trained on more tokens, because Meta's fit on its own data put the optimum there. (With the rounded constants the paper quotes, $(\alpha, A) = (0.53, 0.29)$, the same formula gives 10.5 trillion tokens instead of 16.3. The figure's legend has the precise values, a third reminder in this chapter that published constants are often rounded too coarsely to extrapolate with.)

The replication

In 2024 Besiroglu et al. tried to reproduce Approach 3. The raw data had never been released, so they reconstructed it from the paper's Figure 4. They saved the PDF's figure as SVG, read off each point's position, and mapped its color to the loss on the plot's color scale, recovering about 245 runs. Refitting with the same procedure (and excluding five outliers, leaving 240 points), they found

giving $a = 0.513$ with a bootstrap standard error of about 0.02. Three findings came out of it.

• The refit agrees with Approaches 1 and 2. It implies about 20 tokens per parameter, "consistent with both how the Chinchilla model was trained and the findings from Approaches 1 and 2". The published Approach 3 constants implied about 70.
• The published confidence intervals were implausibly narrow. An interval of 0.454 to 0.455 for $a$ would require "over 600,000 experiments", while Hoffmann et al. "likely only ran fewer than 500". The replication's interval was about fifty times wider.
• The likely cause was an optimizer that stopped early. The original fit averaged the Huber losses rather than summing them, which made the objective so small that L-BFGS met its stopping tolerance before reaching the minimum, as one of the original authors confirmed. Rounding the reported constants made the predictions worse still.

The lesson is broader than Chinchilla. A scaling fit with a beautiful formula and tight intervals can still be wrong, and the best checks are the ones from §9.1. Fit the same data several ways, compare the methods, look at the residuals, and be suspicious of error bars that seem too good.

In PyTorch

The Approach 3 fit in PyTorch, on runs generated from the replication's law. Autograd supplies the gradient, reduction="sum" avoids the averaging trap, and the best of a small grid of starts wins.

Read more

• Hoffmann et al. (2022). Training Compute-Optimal Large Language Models. Section 3 has the three approaches, Table 3 the optimal sizes, and Appendix D the fitting details.
• Besiroglu et al. (2024). Chinchilla Scaling: A replication attempt. How the data were reconstructed, the refit, and what went wrong with the original fit.
• Rae et al. (2021). Scaling Language Models: Methods, Analysis & Insights from Training Gopher. The model Chinchilla showed was oversized, and the MassiveText data both were trained on.
• Dubey et al. (2024). The Llama 3 Herd of Models. Section 3.2.1 is a modern isoFLOP study, from $6\times10^{18}$ to $10^{22}$ FLOPs, extrapolated to the 405B flagship.
• DeepSeek-AI (2024). DeepSeek LLM. Compute-optimal exponents that depend on data quality, and scaling laws for the learning rate and batch size.
• Hu et al. (2024). MiniCPM. Scaling with a warmup-stable-decay schedule, which gives a very different optimal ratio.

Watch next

▶

▶