How loss falls with parameters, data, and compute, how to fit those curves without fooling yourself, how to spend a training budget, and how to make hyperparameters carry over from a small model to a large one.

You have trained six language models, from 10 million to 300 million parameters, on the same data with the same recipe. Their final losses fall from 3.4 to 2.2 nats per token. Your manager asks the question that every frontier lab asks before every large run. If we spend a hundred times more and train a 30-billion-parameter model, what loss will it reach, and is that worth the money?

You cannot answer by training it, because that is the very thing you are trying to decide. You have to extrapolate, from six points, two orders of magnitude beyond the largest one. The most valuable empirical fact in modern AI is that this extrapolation often works. Loss falls with scale along a curve so regular that a straight line on the right axes predicts it. The second most valuable fact is that it is easy to draw that line wrong.

This section is about drawing it right. We define power laws and see why they are straight on log-log axes, add the irreducible loss that bends them, fit them by least squares in log space, and put honest error bars on extrapolations with the bootstrap. Everything later in the chapter, from Kaplan's laws to Chinchilla's budget rule to the prediction of downstream accuracy, is a power-law fit, and every one of them can fail in the ways shown here.

A function that looks the same at every scale

A power law is a function of the form

where $x$ is the thing we scale (parameters, tokens, or compute), $y$ is what we measure (usually a loss), and $k$ is the exponent. We write the exponent with a minus sign because in this chapter the measured quantity falls as scale grows, so $k > 0$.

The defining property is what happens when you multiply $x$ by a constant $c$,

In plain words, doubling the model shrinks the loss by the same factor, $2^{-k}$, whether you double a tiny model or a giant one. A power law has no built-in scale. Zoom in or out on its graph and it looks the same, which is why physicists call such functions scale-free.

Compare an exponential, $y = a\,e^{-\lambda x}$. There, adding a constant to $x$ multiplies $y$ by a constant, and doubling $x$ squares the remaining fraction. Exponentials die quickly. Power laws decay slowly and forever, with a long tail.

Taking logarithms of the power law turns multiplication into addition,

In plain words, on axes where both $x$ and $y$ are drawn on a logarithmic scale, a power law is a straight line, and its slope is $-k$. An exponential, by contrast, is straight on semi-log axes (only $y$ logarithmic). The cell below draws both on all three kinds of axes.

The power law is the straight line on the right, and the exponential is the straight line in the middle. When someone shows you a scaling plot, the first thing to check is the axes. A "linear trend" on log-log axes is a power law, and its slope is the exponent.

Small exponents, enormous multipliers

The exponents that appear in language modeling are small. Kaplan et al. (2020) measured $k \approx 0.076$ for loss against parameters and $k \approx 0.05$ for loss against compute (§9.2). Small exponents hide a brutal fact. To reduce the loss by a factor $r$ you must multiply scale by

In plain words, a ten percent lower loss costs about eight times the compute, and the next ten percent costs eight times more again. This is the arithmetic behind training budgets that grow by orders of magnitude between model generations.

To halve a pure power-law loss with $k = 0.076$ you would need about 9,000 times more parameters. Real losses do not halve at all, as we will see in a moment, because part of the loss cannot be removed.

Why would loss follow a power law?

Nobody derived scaling laws from first principles and then checked them. They were found in data. Hestness et al. (2017) trained models for translation, language modeling, image classification, and speech recognition on growing subsets of data and found that generalization error fell as a power law in dataset size in every domain, with exponents between about 0.07 and 0.35 (word-level language models sat at the shallow end, near 0.07). They also described three regions of a learning curve. With very little data the model does no better than a best guess, then it enters a power-law region, and finally it flattens against an irreducible error. Kaplan et al. (2020) then showed the same regularity for Transformer language models against parameters, data, and compute, over many orders of magnitude.

Several theories try to explain the regularity. They are hypotheses with supporting evidence, not settled results.

• The data manifold. Sharma and Kaplan (2020) argue that a network fits data lying on a manifold of intrinsic dimension $d_{\text{int}}$, and that the error of a piecewise approximation with $N$ parameters falls like $N^{-4/d_{\text{int}}}$ for a smooth target and cross-entropy loss. Small exponents then mean high intrinsic dimension.
• Variance-limited and resolution-limited regimes. Bahri et al. (2021) separate the regime where one resource is nearly unlimited and the loss approaches its limit like $1/x$ (a variance effect) from the regime where the resolution of the fit limits it, which gives the small exponents seen in practice.
• Quanta. Michaud et al. (2023) propose that a model learns discrete skills ("quanta") whose usefulness follows a Zipf distribution. The derivation below shows how that alone produces a power law.

For the rest of the chapter we treat power laws as what they are, the best empirical description we have, valid over the range where they were measured and no further without evidence.

The floor you cannot get below

Language is not deterministic. After "I had a cup of" the next word might be "tea" or "coffee", and no model, however large, can know which. Recall from §0.5 and §3.1 that the cross-entropy of a model $q$ on data from $p$ splits as $H(p, q) = H(p) + D_{\mathrm{KL}}(p \Vert q)$. Training can drive the KL term toward zero, but the entropy $H(p)$ of the text itself is a floor. A pure power law ignores this. It promises that the loss goes to zero.

The fix is to add a constant. The saturating power law is

Here $\mathcal{L}$ is the loss, $x$ is the scaled resource, $E$ is the irreducible loss, and $A\,x^{-\alpha}$ is the reducible part. In plain words, the loss is a floor that no amount of scale removes, plus an excess that falls as a power law.

$E$ is best read as "the loss this family of models would reach with unlimited scale on this data, with this tokenizer". It estimates the entropy of the text, but it also absorbs any limitation shared by all the models, so it is a fitted parameter, not a measured constant. Chinchilla's fit (§9.3) put it at $E = 1.69$ nats per token on its data.

The floor bends the line

Take logarithms of the saturating law and the straight line is gone. The local slope on log-log axes is

In plain words, the slope you see on a log-log plot is the true exponent multiplied by the fraction of the loss that is still reducible. Early on, when most of the loss is reducible, the slope is close to $-\alpha$. As models approach the floor, the reducible fraction shrinks and the curve flattens.

Here is a worked example with Chinchilla's constants for the parameter term, $E = 1.69$ and $\alpha = 0.34$. At a loss of 3.0 the reducible fraction is $1.31/3.0 = 0.44$, so the local slope is $-0.34 \times 0.44 = -0.15$. At a loss of 2.0 it is $0.31/2.0 = 0.155$, and the slope is $-0.053$. The same law looks three times flatter at the large end than at the small end.

This is the first way a naive fit misleads. Fit a straight line on log-log axes to small models and you measure the steep early slope. Extend that line and it predicts losses far below what large models achieve, eventually below the floor itself.

From models up to 316 million parameters the straight line has slope $-0.123$, and at 30 billion parameters it already predicts 1.24 nats, below the floor of 1.69, when the truth is 1.80. At 300 billion it predicts 0.94 against a truth of 1.74. Every number in this cell is noiseless. The error comes entirely from using the wrong functional form.

import numpy as np

def local_exponent(x, E, A, alpha):
    return -alpha * np.ones_like(np.asarray(x, float))   # placeholder: pretends there is no floor

Fitting a pure power law, and why in log space

Start with the easy case, data that follow $y = a x^{-k}$ up to noise. Taking logs gives a linear model, $\log y_i = \log a - k\log x_i + \varepsilon_i$, so ordinary least squares on the points $(\log x_i, \log y_i)$ gives the answer in closed form. With $u_i = \log x_i$ and $v_i = \log y_i$,

where bars are means. In plain words, the exponent is the slope of the best straight line through the log-log plot, and the prefactor comes from its intercept. With two points the line is exact, and you can do it by hand. Two models of $10^8$ and $10^9$ parameters with losses 3.10 and 2.70 give $\hat k = \log(3.10/2.70)/\log 10 = 0.060$, and a third point is what tells you whether a straight line was the right model at all.

Why fit in log space rather than minimizing $\sum_i (y_i - a x_i^{-k})^2$ directly? There are two reasons, and both matter for scaling laws.

1. The noise is relative. Rerunning a training job with a new seed changes its loss by a roughly fixed percentage, not a fixed amount. Squared error in log space, $(\log y_i - \log\hat y_i)^2 \approx ((y_i - \hat y_i)/y_i)^2$, weights every point by its relative error, which matches that noise.
2. Linear space lets the largest values dominate. The smallest models have the largest losses, so a linear-space fit spends its effort on them and treats the large models, the ones closest to what you want to predict, as afterthoughts.

The cell below fits the same noisy data both ways.

Both fits are right on average, but the linear-space estimate is almost three times as noisy (a spread of 0.009 against 0.003), because it effectively listens only to the few smallest-$x$ points with the largest values. An exponent that wobbles by 0.009 sounds harmless until you extrapolate. Over four decades, it moves the prediction by $10^{4 \times 0.009} - 1 \approx 9$ percent.

import numpy as np

def loglog_fit(x, y):
    raise NotImplementedError

Fitting with an irreducible term

Now the real case, $\mathcal{L}(x) = E + A x^{-\alpha}$, with three unknowns. The model is no longer linear in its parameters, so there is no closed form, and the loss surface has awkward valleys. There are three good ways to fit it, and you should know all of them.

Profile over the floor (variable projection). If you knew $E$, the rest would be easy, since $\log(\mathcal{L} - E) = \log A - \alpha\log x$ is a straight line. So try many values of $E$ between 0 and just below the smallest observed loss, fit the line for each, and keep the $E$ whose fit has the smallest error in log space. This search over one variable is robust, cannot get stuck in a bad local minimum, and gives a profile of the error against $E$ that shows how well determined the floor is.

Nonlinear least squares in log space. Minimize $\sum_i \big(\log y_i - \log(E + A x_i^{-\alpha})\big)^2$ with a general optimizer such as scipy.optimize.least_squares, starting from the profile solution or from several starting points.

The Chinchilla recipe. Hoffmann et al. (2022) fit their two-variable law (§9.3) by minimizing a Huber loss of the log residuals with L-BFGS, from a grid of starting points. They write the model as a log-sum-exp, which keeps $E$ and $A$ positive automatically and makes the problem better conditioned,

In plain words, the logarithm of a sum of two positive terms is the log-sum-exp of their logarithms, so we can optimize $a$ and $e$ freely over the real line. The Huber loss is quadratic for small residuals and linear for large ones, so a single bad run (a diverged job, a mislabeled checkpoint) cannot drag the whole fit. Chinchilla used a threshold of $\delta = 10^{-3}$ on log-loss residuals. We implement this version in PyTorch at the end of the section.

import numpy as np

def fit_saturating(x, y):
    x, y = np.asarray(x, float), np.asarray(y, float)
    slope, intercept = np.polyfit(np.log(x), np.log(y), 1)
    return 0.0, float(np.exp(intercept)), float(-slope)   # placeholder: a pure power law, no floor

How far can you extrapolate?

Even with the right functional form, a fit to a short range of scales cannot pin down all three parameters. The reason is a near-degeneracy. Over a few decades, a law with a lower floor and a smaller exponent looks almost exactly like one with a higher floor and a larger exponent, because both bend by about the same amount inside the range. They separate only beyond it, which is precisely where you want to predict.

You can see this by profiling. Fix $E$, fit the best $A$ and $\alpha$, and plot the remaining error against $E$.

With twelve models from $10^7$ to $10^{8.5}$ parameters (blue), every floor from 1.52 to 1.78 fits within a factor of two of the best error, and those floors predict anything from 1.59 to 1.80 at $10^{12}$ parameters. With two more decades of models (orange), the valley is sharp, only the true floor survives, and the prediction is pinned to within a few thousandths. The lesson is that the range of scales you fit matters more than the number of points. Ten models spread over three decades beat thirty models crowded into one.

Error bars with the bootstrap

A single fitted curve hides how uncertain it is. The simplest honest error bar for an extrapolation is the bootstrap (Efron 1979), which §0.11 used for test-set statistics. Here the resampled unit is a training run rather than a test question. Pretend your $n$ data points are the whole population. Draw $n$ points from them with replacement (some appear twice, some not at all), refit, and record the prediction at the scale you care about. Repeat a few hundred times. The spread of the recorded predictions estimates how much the prediction would change if you had run a different set of models of similar sizes.

Two cautions. The bootstrap measures only the uncertainty that comes from noise in the points you have. It cannot know that the true curve breaks to a different slope beyond your data, a real phenomenon that Caballero et al. (2022) model with broken power laws, smooth joins of several power-law segments. And resampling points requires enough of them for each resample to contain distinct scales, so with six models the band itself is noisy. With so few runs, a common alternative is the residual bootstrap. Fit once, then build each replicate by adding resampled log residuals to the fitted curve at the original scales, which keeps every scale in every replicate at the price of assuming the functional form is right. Besiroglu et al. (2024) turned this reasoning around, asked how many runs an interval as narrow as Chinchilla's would need, and concluded that the published intervals were far too narrow to be true (§9.3).

import numpy as np

def bootstrap_predict(x, y, x_new, fit, n_boot=200, seed=0):
    E, A, alpha = fit(x, y)
    p = E + A * x_new ** -alpha
    return p, p, p          # placeholder: no uncertainty at all

Here is the bootstrap applied to our six small models, and then to ten models reaching two decades further, with each band drawn over the whole range of scales.

With six models up to $3\times10^8$ parameters, the 90 percent band at $10^{12}$ runs from 1.13 to 1.90 nats, far too wide to plan with, and its lower edge sits below the true floor of 1.69. With ten models reaching $10^{10}$, it narrows to 1.70 to 1.75 and contains the truth, 1.72. Most of that gain comes from the wider range, as the profile above showed with the number of models held fixed.

The ways a scaling fit goes wrong

Most failed extrapolations are not caused by the fitting algorithm. They come from the data that went into it. Here are the mistakes that recur, with where in this chapter each one is examined.

In PyTorch

The Chinchilla fitting recipe, in one dimension. The model is written as a log-sum-exp, the objective is a Huber loss on log residuals, and L-BFGS starts from a small grid of initial values, keeping the best result. This is the same procedure §9.3 applies to the two-variable law.

With twelve noisy models over three decades, $A$ comes out 28 percent high, $\alpha$ a little high, and $E$ a little high, errors that largely cancel. The prediction a hundred times past the data is off by 0.012 nats, under 1 percent. That is the near-degeneracy of the previous section in action. Individual constants are poorly determined, while the predictions they make together near the data are much better determined.

Note reduction="sum". Besiroglu et al. (2024) traced part of Chinchilla's fitting problem to averaging the Huber losses instead of summing them, which shrinks the objective so much that the optimizer's stopping tolerance is reached too early (§9.3).

Read more

• Hestness et al. (2017). Deep Learning Scaling is Predictable, Empirically. The first broad evidence of power-law learning curves across domains, and the three regions of a learning curve.
• Kaplan et al. (2020). Scaling Laws for Neural Language Models. Power laws for language models over seven orders of magnitude of compute, the subject of the next section.
• Hoffmann et al. (2022). Training Compute-Optimal Large Language Models. Appendix D describes the Huber and log-sum-exp fitting procedure used here.
• Besiroglu et al. (2024). Chinchilla Scaling: A replication attempt. A case study in how a scaling-law fit and its error bars can go wrong.
• Caballero et al. (2022). Broken Neural Scaling Laws. A functional form for curves whose slope changes, and why single power laws can mislead.
• Michaud et al. (2023). The Quantization Model of Neural Scaling and Sharma & Kaplan (2020), A Neural Scaling Law from the Dimension of the Data Manifold. Two theories of where the exponents come from.
• Bahri et al. (2021). Explaining Neural Scaling Laws. Variance-limited and resolution-limited scaling regimes.

Watch next

▶

▶