In January 2020 a team at OpenAI led by Jared Kaplan asked a question that sounds like accounting. If you have a fixed amount of compute, how should you spend it? Train a bigger model, or the same model on more data, or for more steps, or with a bigger batch? Nobody knew, and each lab answered by habit.

They answered it by training hundreds of Transformer language models, from 768 to 1.5 billion non-embedding parameters, on subsets of a 23-billion-token web corpus, and plotting everything against everything. The result, Scaling Laws for Neural Language Models (Kaplan et al. 2020), found that the test loss is a power law in model size, in dataset size, and in training compute, with some trends spanning more than seven orders of magnitude, and that most other choices barely matter. It shaped how GPT-3 was built and started the scaling era.

It also got one important conclusion wrong, in a way that took four years and several papers to fully explain. This section covers both. We first derive the compute rule $C \approx 6ND$ that every scaling law is written in, then state Kaplan's laws with their constants, then work through the compute-optimal frontier and the bookkeeping choices that bent it.

Counting the compute, C = 6ND

Every scaling law needs a measure of cost, and the one the field settled on is the number of floating-point operations used in training. Here $N$ is the number of parameters, $D$ the number of training tokens, and $C$ the training compute in FLOPs. The rule is

You met it in §0.2 and counted it for a whole Transformer in §5.6. Here is the derivation once more, the way an interviewer wants to hear it.

Nearly all of a Transformer's arithmetic is matrix multiplication. Take one linear layer, $\mathbf{Y} = \mathbf{X}\mathbf{W}$ with $\mathbf{X} \in \R^{T\times a}$ holding $T$ tokens and $\mathbf{W} \in \R^{a\times b}$. Each of the $Tb$ outputs is a dot product of length $a$, which costs $a$ multiplications and $a$ additions, so the layer costs $2Tab$ FLOPs.

1. Forward pass, 2 FLOPs per parameter per token. The layer has $ab$ parameters and costs $2Tab$, so $2$ per parameter per token.
2. Backward pass, 4 FLOPs per parameter per token. Given $\mathbf{G} = \partial\mathcal{L}/\partial\mathbf{Y} \in \R^{T\times b}$, backpropagation computes two products. The gradient for the layer below, $\partial\mathcal{L}/\partial\mathbf{X} = \mathbf{G}\mathbf{W}^\top$, is a $(T\times b)(b\times a)$ product costing $2Tab$. The weight gradient, $\partial\mathcal{L}/\partial\mathbf{W} = \mathbf{X}^\top\mathbf{G}$, is an $(a\times T)(T\times b)$ product, another $2Tab$.
3. Total, 6 per parameter per token. Summing over every weight matrix and every training token gives $6ND$.

In plain words, each parameter takes part in one multiply-add going forward and two going backward, for every token the model is trained on.

The rule rests on four assumptions, and an interviewer will expect you to name them. First, every parameter is used exactly once per token, as in a dense Transformer. In a mixture-of-experts model only the routed experts run, so $N$ must be the active parameters (§13.1), and a parameter reused several times per token would count once per use. Second, a FLOP means one addition or one multiplication, so a multiply-add is 2. Third, only the matrix multiplications against weights are counted. Norms, activations, softmax, and the optimizer update are lower order, the update because it costs a few FLOPs per parameter per step, not per token. Fourth, nothing is computed twice. Activation checkpointing reruns the forward pass during the backward pass and raises the hardware's work to about $8ND$, while the useful compute, the number that scaling laws and MFU use, stays $6ND$.

What the rule leaves out matters as much as what it keeps. Kaplan et al.'s Table 1 counts each operation of a Transformer with $n_{\text{layer}}$ layers, width $d_{\text{model}}$, attention width $d_{\text{attn}}$, feed-forward width $d_{\text{ff}}$, and context $n_{\text{ctx}}$.

Kaplan defines $N$ as the non-embedding parameters, the four rows in the middle,

and the forward cost per token as $C_{\text{forward}} \approx 2N + 2n_{\text{layer}}n_{\text{ctx}}d_{\text{attn}}$. The context term is the attention core of §4.7, the scores $\mathbf{Q}\mathbf{K}^\top$ and the weighted sum of values, which use activations rather than weights. Each of the two products costs $2n_{\text{ctx}}d_{\text{attn}}$ per token per layer, so a dense implementation pays $4n_{\text{layer}}n_{\text{ctx}}d_{\text{attn}}$. Kaplan's table lists half of that, which is what it costs when a causal mask lets each token skip the half of the context it cannot see. (PaLM's count, which §8.9 uses, keeps the dense figure, $3\times4 = 12\,n_{\text{layer}}n_{\text{ctx}}d_{\text{attn}}$ training FLOPs per token, so check which convention a paper uses.) Either way the term is small next to $2N \approx 24n_{\text{layer}}d_{\text{model}}^2$ when $d_{\text{model}}$ is large next to $n_{\text{ctx}}/12$, so it is dropped. With the backward pass costing twice the forward pass, training costs $C \approx 6N$ per token. Notice two omissions. The embedding lookup costs nothing, which is right. The output layer's $2d_{\text{model}}n_{\text{vocab}}$ FLOPs are also left out, which is not, and for small models it is a large share of the work. This detail will come back to bite.

The paper reports compute in petaflop-days, $1\ \text{PF-day} = 10^{15} \times 24 \times 3600 = 8.64\times10^{19}$ FLOPs, a unit you will still see in older papers.

A GPT-2 XL-shaped model has 1.47 billion non-embedding parameters, close to Kaplan's largest, and 250,000 steps of $2^{19}$ tokens cost about 13 PF-days. GPT-3 cost about 3,600 PF-days, which is the figure in the GPT-3 paper.

You can check the rule with PyTorch's FLOP counter, at the end of this section. It confirms that a backward pass costs exactly twice the forward pass for every matrix multiplication.

def kaplan_flops(n_layer, d_model, n_ctx, n_vocab, n_tokens, d_attn=None, d_ff=None):
    raise NotImplementedError

From a budget to a model

The rule also runs backward, and this is one of the most common interview calculations. Suppose you have 1,000 H100 GPUs for 30 days. Each H100 has a dense bf16 peak of about 989 TFLOP/s, and a good run achieves a model FLOPs utilization (MFU) of 40 percent (§6.7). The budget is

If you decide to train with $r$ tokens per parameter, then $D = rN$ and $C = 6rN^2$, so

With $r = 20$, the ratio §9.3 will justify, that is $N \approx 92$ billion parameters on $D \approx 1.85$ trillion tokens. In plain words, at a fixed ratio of data to parameters, both grow as the square root of compute.

def cluster_flops(n_gpus, peak_flops, mfu, days):
    raise NotImplementedError

def model_for_budget(C, tokens_per_param):
    raise NotImplementedError

def days_for(N, D, n_gpus, peak_flops, mfu):
    raise NotImplementedError

The three laws

Kaplan et al. trained decoder-only Transformers on WebText2, a corpus of $2.29\times10^{10}$ tokens of web text with GPT-2's tokenizer, using a context of 1,024 tokens. The headline results are three power laws, each measured when the other two resources are not the bottleneck. Losses are in nats per token on held-out WebText2.

Loss against parameters, for models trained to convergence on enough data,

Loss against data, for large models trained with early stopping on $D$ tokens,

Loss against compute, for the best model at each compute budget,

where $C_{\min}$ is the compute the run would need at a batch size well below the critical batch size (defined below), the most compute-efficient way to train. Here $N_c$, $D_c$, and $C_c^{\min}$ are scale constants with the units of their resource, and $\alpha_N$, $\alpha_D$, $\alpha_C^{\min}$ are exponents. In plain words, each tenfold increase in parameters cuts the loss by a factor $10^{-0.076} = 0.84$, each tenfold increase in data by $10^{-0.095} = 0.80$, and each tenfold increase in compute by $10^{-0.05} = 0.89$.

None of these laws has an irreducible term. Kaplan et al. note that they must eventually flatten, since the loss cannot go below the entropy of text, but over their range a pure power law fitted well. The constants are specific to WebText2 and GPT-2's tokenizer. The exponents, not the constants, are the transferable part.

Kaplan et al. also reported several findings that have held up well.

• Performance depends strongly on scale and weakly on shape. At fixed $N$, varying depth, width, and the number of heads over a wide range changes the loss by only a few percent.
• Transfer follows test loss. Loss on other text distributions tracks loss on the training distribution with a roughly constant offset, so improving one improves the others.
• There is a critical batch size. Beyond a batch of $B_{\text{crit}}$ tokens, larger batches stop reducing the number of steps much. Kaplan found $B_{\text{crit}}(\mathcal{L}) \approx B_\ast/\mathcal{L}^{1/\alpha_B}$ with $B_\ast \approx 2\times10^{8}$ tokens and $\alpha_B \approx 0.21$, so it depends on the loss reached, not on model size. At a loss of 2.5 nats that is about 2.5 million tokens. The idea comes from McCandlish et al. (2018), who related it to the gradient noise scale, and it returns in §8.7.

How data limits a model

The laws above each hold one resource fixed at "plenty". To see how parameters and data interact, Kaplan et al. fit a combined form,

with its own fitted constants $\alpha_N = 0.076$, $\alpha_D = 0.103$, $N_c = 6.4\times10^{13}$, and $D_c = 1.8\times10^{13}$. In plain words, the bracket adds a model-size term and a data term, and the outer power turns the sum back into a loss. When $D$ is huge the data term vanishes and $\mathcal{L}(N, D) \to \mathcal{L}(N)$, and when $N$ is huge the model term vanishes and $\mathcal{L}(N, D) \to \mathcal{L}(D)$.

These runs used early stopping on fixed subsets of WebText2, so a model trained on small $D$ saw its data many times. The law therefore describes overfitting, not single-pass training. The overfitting penalty depends on $N^{\alpha_N/\alpha_D}/D$, and Kaplan derived that keeping it below the run-to-run noise in loss (about 0.02) needs roughly

tokens. Since $8^{0.74} \approx 4.7$, "every time we increase the model size 8x, we only need to increase the data by roughly 5x", in the paper's words. A 1-billion-parameter model needs about 23 billion tokens to avoid overfitting, about as much as WebText2 held.

Large models learn more from each token

The combined law also answers a different question. How many tokens does a model of size $N$ need to reach a target loss $\ell$? Solve $\mathcal{L}(N, D) = \ell$ for $D$,

which is finite only if the denominator is positive, that is, if a model of size $N$ can reach $\ell$ at all. In plain words, the bigger the model, the smaller the subtracted term, so the fewer tokens it needs to reach the same loss. This is sample efficiency, and Kaplan et al. found it in both data and optimization steps. "Large models are more sample-efficient than small models, reaching the same level of performance with fewer optimization steps and using fewer data points."

def kaplan_loss(N, D):
    raise NotImplementedError

def tokens_to_reach(target, N):
    raise NotImplementedError

To reach 2.5 nats, a 1-billion-parameter model needs about 4.8 billion tokens and a 10-billion-parameter model about 2.7 billion, while a 100-million-parameter model never gets there. Bigger models squeeze more out of every token.

The compute-optimal frontier

Now the question the paper set out to answer. With a fixed compute budget $C$, which model size gives the lowest loss? You could train a big model on few tokens or a small model on many, since $C = 6ND$ fixes only the product.

Kaplan et al. answered it from training curves. Each run's loss was recorded throughout training, and each point on a curve corresponds to a compute value $6N \times (\text{tokens so far})$. At each compute value they took the best loss across all model sizes, which traces the compute-efficient frontier, and recorded which size achieved it. Then they fit power laws to the result, finding (in their Table 6, with $C_{\min}$ in PF-days)

for the parameters, batch size in tokens, number of steps, and tokens. In plain words, as compute grows, nearly all of it should go into a bigger model. Ten times the compute means $10^{0.73} \approx 5.4$ times the parameters but only $10^{0.27} \approx 1.9$ times the data, and the number of optimization steps hardly changes.

The same exponent from the learning curves

Kaplan et al. also predicted the frontier from a formula, and the argument is a good warm-up for the derivation in §9.3. With unlimited data, after an early transient, the loss of a model of size $N$ after $S$ steps fits

where $S_{\min} = S/(1 + B_{\text{crit}}/B)$ converts the steps of a run at batch $B$ into the fewest steps the same progress would take with a very large batch. In plain words, the loss is a size penalty plus a penalty for stopping early, and the second one falls quickly, with an exponent ten times larger.

Now spend a budget. Train at about the critical batch size, so the compute is $C \propto N\,B_{\text{crit}}\,S_{\min}$, with $B_{\text{crit}} \propto \mathcal{L}^{-1/\alpha_B}$ from the critical-batch law above. There is a symmetry. Pick any factor $\lambda > 0$, multiply $N$ by $\lambda^{-1/\alpha_N}$ and $S_{\min}$ by $\lambda^{-1/\alpha_S}$, and both terms of the loss, and so the loss itself, are multiplied by $\lambda$, while $B_{\text{crit}}$ is multiplied by $\lambda^{-1/\alpha_B}$. A plan that is optimal at one budget therefore maps to a plan that is optimal at the budget multiplied by $\lambda^{-(1/\alpha_N + 1/\alpha_B + 1/\alpha_S)}$. Every quantity on the frontier is a power of the loss, and eliminating the loss gives

In plain words, each resource grows in inverse proportion to its own exponent. The step term falls fastest, so steps barely need to grow, and the size term falls slowest, so most of the compute goes to parameters. The three allocation exponents add to 1, because $NBS$ is the compute.

The prediction, $N \propto C^{0.68}$ in this rough form and 0.71 in the paper's more careful version, is within 7 percent of the measured 0.73 (within 3 percent for the paper's version), and the batch exponent, 0.25, matches the measured 0.24. Theory and measurement agreed with each other, and both were wrong about the world, because the fitted curves themselves carried the biases discussed below. A derivation is only as good as the curves it starts from.

The paper drew the practical conclusion in a striking phrase, "convergence is inefficient". Compute-optimal training uses very large models and stops "significantly short of convergence". GPT-3 (175B parameters, 300B tokens, 1.7 tokens per parameter) was built in this spirit. Its contemporaries Gopher (280B on 300B tokens) and Megatron-Turing NLG (530B on 270B tokens) went further still.

The cell below asks both rules what to build with GPT-3's compute. Kaplan's line counts non-embedding parameters and Chinchilla's counts all of them, but at this scale the difference is small.

For GPT-3's budget, Kaplan's fit says about 520 billion parameters on 180 billion tokens, a third of a token per parameter. (The two Kaplan formulas were fitted separately, against $C_{\min}$ rather than the actual compute, so their product overshoots the budget, $6ND \approx 5.7\times10^{23}$. Read them as directions, not as a plan that spends exactly $C$.) Chinchilla says 50 billion parameters on a trillion tokens, about 21 per parameter. These differ by a factor of ten in model size. One of them was wrong, and the stakes were enormous, since a model ten times too big for its compute wastes most of the budget.

What Kaplan got wrong, and why

Two years later Hoffmann et al. (2022) repeated the experiment differently and found that parameters and data should grow equally, $N_{\text{opt}} \propto C^{0.50}$ (§9.3). The explanation of the gap became a small research field of its own. Three causes have been identified, and they are worth knowing in detail, because each is a mistake you could make in your own scaling study.

Cause 1, learning-rate schedules and intermediate checkpoints

Kaplan's runs all used the same schedule, 3,000 warmup steps then cosine decay to zero over 250,000 steps, whatever the model size. A point taken from the middle of such a run measures a model whose learning rate is still high. A run tuned to stop at that point would have decayed its learning rate and reached a lower loss. Hoffmann et al. put it this way. With a fixed cosine schedule, "the intermediate loss estimates ... are therefore overestimates of the loss of a model trained with a schedule length matching" the shorter horizon, which "results in underestimating the effectiveness of training models on less data" and pushes the conclusion toward bigger models. They also showed that setting the cosine cycle more than 25 percent longer than the actual run noticeably hurts the final loss.

This was the first explanation offered, and it is a real effect, but later work found it is not the main one. Porian et al. (2024) found that careful learning-rate decay "is not essential" for recovering Chinchilla's law, and Pearce and Song (2024) found that the schedule changed the fitted exponent only slightly.

Cause 2, non-embedding parameters at small scale

Kaplan counted only non-embedding parameters and non-embedding compute, because "this produces significantly cleaner scaling laws" across depths. Chinchilla counted everything. For a large model the two are almost the same. For a small one they are not. A model with $d_{\text{model}} = 256$ and a 50,257-token vocabulary has 12.9 million embedding parameters, which dwarfs a few layers of $12d^2 \approx 0.8$ million each.

Pearce and Song (2024) showed that this choice alone, combined with Kaplan's small model sizes (many runs had fewer than 100 million parameters), reproduces Kaplan's exponent from Chinchilla's own law. Suppose the truth is Chinchilla's law in total parameters $N_T$, and that the embedding parameters grow like the width, $N_T = N_{\setminus E} + \omega N_{\setminus E}^{1/3}$, where $N_{\setminus E}$ is the non-embedding count and $\omega \approx 47{,}491$ is fitted to the shapes of Chinchilla's models, with their 32,000-token vocabulary. The cube root appears because the embedding has $V d$ parameters and, at a fixed depth-to-width ratio, $N_{\setminus E} \propto d^3$, so $d \propto N_{\setminus E}^{1/3}$. Now do what Kaplan did. For each non-embedding compute budget $C_{\setminus E} = 6N_{\setminus E}D$, find the non-embedding size that minimizes the loss, and fit a power law over Kaplan's range of sizes. The cell does exactly that.

The same underlying law gives an exponent of 0.74 when measured Kaplan's way and 0.45 when measured Chinchilla's way. Pearce and Song report 0.74 for this calculation with Chinchilla's published constants and 0.78 with the replication's (§9.3), values they call "close to the Kaplan coefficient of 0.73". Most of the disagreement was bookkeeping, plus curvature at small scale that a single power law over many decades hides.

Cause 3, the output layer, warmup, and optimizer tuning

Porian et al. (2024) reproduced Kaplan's exponent and then removed the discrepancy one fix at a time, over 900 training runs on two datasets. On RefinedWeb the fitted exponent of $N_{\text{opt}}$ moved as follows.

Each row fixes a bias against small models. The output layer is a larger share of a small model's compute, so leaving it out understates their cost. A 3,000-step warmup is a large fraction of a short run, so small models trained on few tokens spend much of their budget warming up. And hyperparameters tuned for one scale are wrong at others, with $\beta_2 = 0.95$ in particular too low at the small batch sizes small models use. Fix all of these and you recover Chinchilla's 0.5.

What survived

Kaplan's allocation exponent did not survive, but most of the paper did, and it changed the field. Loss falls as a smooth power law in parameters, data, and compute over many orders of magnitude. Training compute is $6ND$. Shape matters much less than size. Bigger models are more sample-efficient. There is a critical batch size set by the loss. And, above all, the habit of predicting a large run from small ones before spending the money, which is how every frontier model is now planned.

In PyTorch

PyTorch's FLOP counter checks the $6N$ rule directly. The model below is a four-layer Transformer with explicit attention, so that every matrix multiplication is counted. The forward pass costs exactly $2N + 2d\,n_{\text{vocab}} + 4LTd$ FLOPs per token (weights, output layer, and the full attention core), and the backward pass exactly twice that.

At this toy size the output layer and attention add 16 percent on top of $6N$. At the scale of a real model, with $d$ in the thousands, they shrink to a few percent, which is why $6ND$ is good enough for planning, and why it is not good enough for fitting exponents at small scale.

Read more

• Kaplan et al. (2020). Scaling Laws for Neural Language Models. Read Sections 1 and 2, Table 1 for the FLOP accounting, and Section 6 for the compute-efficient frontier.
• Hoffmann et al. (2022). Training Compute-Optimal Large Language Models. Section 2 explains why Kaplan's schedule choice biased the result.
• Pearce & Song (2024). Reconciling Kaplan and Chinchilla Scaling Laws. Non-embedding counting at small scale reproduces 0.73 from Chinchilla's law.
• Porian et al. (2024). Resolving Discrepancies in Compute-Optimal Scaling of Language Models. The output layer, warmup, and optimizer tuning, removed one at a time.
• McCandlish et al. (2018). An Empirical Model of Large-Batch Training. The gradient noise scale and the critical batch size.
• Brown et al. (2020). Language Models are Few-Shot Learners. GPT-3, the model Kaplan's laws shaped, with its compute in PF-days in Appendix D.
• DeepSeek-AI (2024). DeepSeek LLM. A modern scaling study with a careful FLOPs-per-token measure and hyperparameter scaling laws.

Watch next

▶

▶