You can now compute the gradient of any loss with respect to any parameter. The gradient says which way is downhill. It does not say how far to step, and that is the hard part. Step too timidly and training takes forever. Step too boldly and the loss explodes. And the right step size along one direction can be disastrous along another.

Picture a long, narrow valley. The gradient points mostly across the valley, toward the steep walls, rather than along the gentle floor toward the minimum. Plain gradient descent zigzags from wall to wall, making slow progress along the floor. Almost every idea in this section (momentum, RMSProp, Adam, preconditioning, schedules) is a way of escaping that valley faster. By the end you'll have derived Adam, including the bias correction that interviewers love to ask about, and you'll know why large language models are trained with AdamW, warmup, cosine decay, and gradient clipping.

Loss landscapes and convexity

Think of the loss $\mathcal{L}(\theta)$ as a landscape over parameter space, and training as walking downhill. Two kinds of landscape behave very differently.

A function is convex if the straight line between any two points on its graph lies on or above the graph, $\mathcal{L}(\alpha\theta_1 + (1-\alpha)\theta_2) \le \alpha\mathcal{L}(\theta_1) + (1-\alpha)\mathcal{L}(\theta_2)$ for $\alpha \in [0, 1]$. For twice-differentiable functions this is equivalent to the Hessian (the matrix of second derivatives) being positive semidefinite everywhere, so the landscape curves upward in every direction. Convex problems have a wonderful property. Every local minimum is a global minimum, so going downhill can never get you stuck. Linear regression, logistic regression, and the softmax classifier with fixed features are convex.

Neural networks are not. A quick argument shows why. Swap two hidden units of an MLP (with their incoming and outgoing weights) and you get a different parameter vector computing exactly the same function, with the same loss. If the loss were convex, the average of those two parameter vectors would have loss no higher. But the average makes the two units identical, which usually computes a worse function. So the loss surface has many separate equally good minima and higher ground between them. This is not a theorem about every network, but it is the typical situation.

In practice that matters less than you might fear. In high dimensions, most critical points with high loss are not bad local minima but saddle points, which curve up in some directions and down in others (Dauphin et al., 2014, give theory and experiments). And empirically, large networks trained with gradient methods reliably find minima with low training loss. What remains hard is speed and stability, which is what the rest of this section is about.

Gradient descent and the step size

Gradient descent repeats one update,

with learning rate $\eta > 0$. In plain words, take a step of size proportional to the slope, in the downhill direction. Why should this work, and how large may $\eta$ be? A quadratic answers both questions exactly, and near any minimum, every smooth loss looks like a quadratic.

Take $\mathcal{L}(\theta) = \tfrac12 \theta^\top \mathbf{A}\theta$ with $\mathbf{A}$ symmetric positive definite, eigenvalues $\mu = \lambda_{\min} \le \dots \le \lambda_{\max} = L$, and minimum at $\theta = \mathbf{0}$. The gradient is $\mathbf{A}\theta$, so the update is $\theta_{t+1} = (\mathbf{I} - \eta\mathbf{A})\theta_t$. A symmetric matrix has $n$ perpendicular eigenvectors $\mathbf{q}_i$, directions that it only stretches, $\mathbf{A}\mathbf{q}_i = \lambda_i\mathbf{q}_i$, and positive definite means every $\lambda_i > 0$ (a one-line recap; §0.10 develops eigenvalues properly). Write $\theta$ in that eigenvector basis, $\theta = \sum_i \theta^{(i)}\mathbf{q}_i$. Then $(\mathbf{I} - \eta\mathbf{A})\mathbf{q}_i = (1 - \eta\lambda_i)\mathbf{q}_i$, so each coordinate evolves on its own, multiplied every step by $1 - \eta\lambda_i$,

In plain words, along each eigen-direction the error shrinks by a fixed factor per step. Along a direction of curvature $\lambda_i$, the factor is $1 - \eta\lambda_i$.

Everything follows from that one line.

• Stability. Every direction converges if and only if $|1 - \eta\lambda_i| < 1$ for all $i$, which means $0 < \eta < 2/L$. The largest curvature sets the speed limit. Above it, the steepest direction overshoots further every step and the loss diverges. At exactly $\eta = 1/L$ that direction is solved in one step.
• Slowness. The flattest direction shrinks by $1 - \eta\mu$ per step, and with $\eta < 2/L$ that is at least $1 - 2\mu/L$. If $L/\mu$ is large, progress along the flat direction is glacial.
• The best fixed step. The worst factor is $\max(|1 - \eta\mu|, |1 - \eta L|)$, smallest when the two are equal, at $\eta^\star = 2/(\mu + L)$, which gives a rate of

The ratio $\kappa$ is the condition number, and it controls everything. Take $\kappa = 100$. The best rate is $99/101 = 0.980$, so reducing the error a thousandfold takes $\ln(1000)/\ln(101/99) \approx 345$ steps, even for the two-dimensional problem in the cell below. In general gradient descent needs on the order of $\kappa$ steps per factor of $e$. That is the narrow valley, in numbers.

At $\eta = 2/101$ it takes 346 steps. Increase the step by 1.5% to $0.0201$, just past $2/L = 0.02$, and it diverges. A smaller step is stable but slower. The cliff edge at $2/L$ is real, and it is why learning-rate sweeps in practice often show the loss improving steadily as you raise the rate, right up until it suddenly blows up.

import numpy as np

def gd_quadratic(A, b, theta0, lr, steps):
    raise NotImplementedError

def best_lr(A):
    raise NotImplementedError

Conditioning, and the idea of preconditioning

If the problem is the spread of curvatures, the fix is to rescale the space so all directions curve equally. Newton's method does this exactly, stepping by $-\mathbf{H}^{-1}\nabla\mathcal{L}$, where $\mathbf{H}$ is the Hessian. On a quadratic it lands on the minimum in one step, whatever $\kappa$ is. But for a model with $n$ parameters the Hessian has $n^2$ entries, and inverting it costs $n^3$, which is hopeless at $n = 10^9$.

The practical compromise is a cheap preconditioner, usually a diagonal one, that rescales each parameter by an estimate of its own curvature or gradient scale. That is exactly what RMSProp and Adam do, as we'll see. Simpler forms of the same idea are everywhere. Standardizing input features to unit variance improves the conditioning of linear models, and normalization layers (§0.7) do something similar inside networks.

Stochastic gradients and minibatch noise

The true loss is an average over the whole training set, $\mathcal{L}(\theta) = \frac1N\sum_{i=1}^N \ell_i(\theta)$, and computing its gradient means touching every example. Stochastic gradient descent (SGD) instead uses a random minibatch $\mathcal{B}$ of $B$ examples,

where $\boldsymbol{\Sigma}$ is the covariance of a single example's gradient. In plain words, a minibatch gradient points downhill on average, with noise whose standard deviation shrinks like $1/\sqrt{B}$, the standard error of §0.4.

Each fourfold increase in batch size halves the noise. That has two consequences you will use when training large models.

• Larger batches allow larger learning rates, up to a point. If you multiply the batch size by $k$, multiplying the learning rate by $k$ keeps the expected progress per example roughly constant. That linear scaling rule (Goyal et al., 2017) works remarkably well up to a critical batch size, beyond which extra batch size buys little (McCandlish et al., 2018). Choosing the batch size and learning rate of a pretraining run is the subject of §8.7.
• Noise is not only a nuisance. It helps escape saddle points, and it is widely believed to bias SGD toward flatter minima that generalize better. This is an empirical and partly theoretical claim, not a settled theorem, but it is one reason small-batch SGD often generalizes well.

Batches bigger than memory, by accumulation

The batch size the optimizer wants is often far larger than what fits on a device. GPT-3 trained its largest model with batches of 3.2 million tokens, and Llama 2 with 4 million (Brown et al., 2020, Touvron et al., 2023), while one GPU holds a few dozen sequences. Gradient accumulation closes the gap. Split the batch $\mathcal{B}$ into $k$ micro-batches $\mathcal{B}_1, \dots, \mathcal{B}_k$ of equal size $B/k$, compute each one's mean gradient, and average them before taking one step,

In plain words, the average of $k$ equal-sized averages is the average of everything, so the step is exactly the one the big batch would have taken, with the same $\boldsymbol{\Sigma}/B$ noise. Nothing about the optimizer changes. Its step counter, its schedule, and its moment estimates all advance once per accumulated batch, not once per micro-batch. The cost is time, $k$ forward and backward passes per step, and the total compute per token is unchanged. With data parallelism the effective batch is micro-batch size times $k$ times the number of devices.

Two details decide whether the equality really holds. Micro-batches of unequal size, such as language-model batches with different numbers of real tokens, must be weighted by their sizes rather than averaged equally. And layers whose output depends on the rest of the batch, such as BatchNorm (§0.7), see only the micro-batch, so they break the equivalence. §0.9 implements accumulation in PyTorch, including the weighting and the order of every operation in the step.

Momentum, a heavy ball in the valley

Back to the narrow valley. Gradient descent zigzags because the gradient keeps flipping sign across the valley while staying small and consistent along it. Momentum (Polyak, 1964) fixes this by averaging recent gradients. In the form PyTorch uses,

with $\beta \in [0, 1)$, typically $0.9$, and $\mathbf{v}_0 = \mathbf{0}$. In plain words, the update direction is a running, exponentially decaying sum of past gradients, like a heavy ball that builds up speed rolling downhill and is hard to deflect.

Two effects explain why it helps. Along a direction where the gradient is consistent, $\mathbf{v}$ grows to $\mathbf{g}/(1 - \beta)$, so the effective learning rate becomes $\eta/(1 - \beta)$, ten times larger for $\beta = 0.9$. Across the valley, where the gradient alternates sign, successive gradients cancel inside $\mathbf{v}$ and the oscillation is damped. On a quadratic with the best $\eta$ and $\beta$ (Polyak's choice), the convergence factor improves from $(\kappa - 1)/(\kappa + 1)$ to $(\sqrt\kappa - 1)/(\sqrt\kappa + 1)$. For $\kappa = 100$ that is $9/11 = 0.82$ instead of $0.98$, the difference between $\sqrt{\kappa}$ and $\kappa$ steps.

Nesterov momentum (Nesterov, 1983, introduced it for convex optimization, and Sutskever et al., 2013, popularized it for deep learning) evaluates the gradient at the point the momentum is about to carry you to, $\theta_t - \eta\beta\mathbf{v}_t$, rather than where you are. It corrects course a step earlier and is a little more stable. PyTorch implements an equivalent rearrangement, written in terms of the look-ahead point itself, so the gradient is always taken at the stored parameters, and the update direction becomes $\mathbf{g}_t + \beta\mathbf{v}_{t+1}$.

import numpy as np

def momentum_step(theta, g, buf, lr, beta, nesterov=False):
    raise NotImplementedError

RMSProp, a step size for every parameter

Momentum fixes oscillation but uses one learning rate for all parameters. In a network, gradient scales differ wildly between parameters (embedding rows for rare tokens get tiny, infrequent gradients, while some layers get large ones). A step that is right for one is wrong for another.

RMSProp (proposed by Hinton in his 2012 Coursera lectures, never formally published) divides each parameter's step by a running estimate of that parameter's gradient magnitude,

where squaring, the square root, and the division are elementwise, $\rho$ is around 0.9 to 0.99, and $\epsilon$ is a tiny constant that prevents division by zero. In plain words, each parameter's gradient is measured relative to its own root mean square, so every parameter's step is roughly $\eta$ in size whatever its gradient scale, and parameters with small gradients move as readily as those with large ones. This is the diagonal preconditioner promised above. It uses the root mean square gradient instead of the curvature, a rough but cheap proxy. Its predecessor, AdaGrad (Duchi et al., 2011), summed all past squared gradients instead of averaging them, so its steps shrink forever, which suits convex problems but stalls deep networks.

Adam, derived

Adam (Kingma & Ba, 2015) combines the two ideas. It keeps an exponential moving average of the gradient (momentum, the first moment) and of the squared gradient (RMSProp, the second moment), and corrects both for a bias that comes from starting them at zero. Given the gradient $\mathbf{g}_t$ at step $t = 1, 2, \dots$,

with $\mathbf{m}_0 = \mathbf{v}_0 = \mathbf{0}$ and defaults $\beta_1 = 0.9$, $\beta_2 = 0.999$, $\epsilon = 10^{-8}$. In plain words, step in the direction of the averaged gradient, scaled per parameter by the root of the averaged squared gradient, after undoing the shrinkage caused by starting the averages at zero.

Where the bias correction comes from

Unroll the recursion for $\mathbf{m}_t$. Starting from $\mathbf{m}_0 = \mathbf{0}$,

Suppose the gradients come from a stationary distribution, with $\mathbb{E}[\mathbf{g}_i] = \mathbf{g}$ for all $i$. Taking expectations and summing the geometric series,

So the raw average is biased toward zero by exactly the factor $1 - \beta_1^t$, and dividing by it gives an unbiased estimate. The same argument with $\mathbf{g}_i^2$ gives $\mathbb{E}[\mathbf{v}_t] = \mathbb{E}[\mathbf{g}^2](1 - \beta_2^t)$. In plain words, the zero initialization drags the early averages toward zero, and since the weights of the moving average sum to $1 - \beta^t$ rather than 1, we renormalize by that sum. (If the gradients drift over time, the correction is only approximate. The Adam paper notes that the error stays small because the moving averages give little weight to old gradients.)

Why does it matter? Look at the very first step. With $\mathbf{m}_1 = 0.1\,\mathbf{g}_1$ and $\mathbf{v}_1 = 0.001\,\mathbf{g}_1^2$, the uncorrected ratio is

while the corrected one is exactly $\operatorname{sign}(\mathbf{g}_1)$. Without correction the first step is 3.16 times too large, and it gets worse before it gets better. The cell below holds the gradient constant so you can watch it. $\mathbf{m}$ catches up within a few dozen steps, but $\mathbf{v}$, with its $\beta_2$ close to 1, stays biased toward zero far longer, so the uncorrected step peaks above 6 times the intended size around step 10 and fades only over thousands of steps (since $0.999^{1000} = 0.37$). That is precisely the period when a freshly initialized model is most fragile.

What Adam's update looks like

Three properties are worth knowing well.

• Its steps are bounded and scale-free. The per-parameter step $\eta\,\hat m/\sqrt{\hat v}$ is roughly at most $\eta$ in size when gradients are consistent, and multiplying the loss by a constant does not change it at all (both $\hat m$ and $\sqrt{\hat v}$ scale by the same factor). So $\eta$ in Adam is close to "how far each parameter may move per step", and good values transfer across losses of different scales.
• It shrinks steps for noisy parameters. $\hat m/\sqrt{\hat v}$ is like a signal-to-noise ratio, $\mathbb{E}[g]/\sqrt{\mathbb{E}[g^2]}$. A parameter whose gradient mostly averages out gets small steps. With $\beta_1 = 0$ and very small $\beta_2$, Adam reduces to taking the sign of the gradient (sign descent).
• It costs memory. Adam keeps two extra numbers per parameter. With fp32 weights, gradients, and two moments, that is 16 bytes per parameter before activations, which is why optimizer state is the first thing sharded across GPUs in ZeRO and FSDP (§10.3).

import numpy as np

def adam_step(theta, g, m, v, t, lr=1e-3, beta1=0.9, beta2=0.999, eps=1e-8):
    raise NotImplementedError

AdamW and decoupled weight decay

Weight decay shrinks every weight toward zero a little each step, which regularizes the model (§0.4 showed it is a Gaussian prior). Its strength is $\lambda$, not to be confused with the eigenvalues $\lambda_i$ earlier in this section. With plain SGD there are two equivalent ways to do it. Add $\tfrac\lambda2\lVert\theta\rVert^2$ to the loss, whose gradient adds $\lambda\theta$ to $\mathbf{g}$, or directly multiply the weights by $(1 - \eta\lambda)$ each step. They give identical updates.

With Adam they are not equivalent, and the difference matters (Loshchilov & Hutter, 2019). If you add $\lambda\theta$ to the gradient, it flows through Adam's division by $\sqrt{\hat{\mathbf{v}}}$. Parameters with a large gradient history then get less decay, and parameters with small gradients get more, which is not what a prior on the weights should do. AdamW decouples the decay from the adaptive step,

so every weight shrinks by the same factor $(1 - \eta\lambda)$ per step, regardless of its gradients. PyTorch's AdamW applies the shrink first, theta *= 1 - lr * weight_decay, and then the Adam step. One subtlety. The paper multiplies the decay by the schedule's multiplier only, so $\lambda$ and the learning rate can be tuned separately, while PyTorch multiplies it by the full learning rate, so in PyTorch changing lr also changes the effective decay.

Averaged over 200 runs, the picture is clear. With L2 inside Adam, the weight with large gradients barely feels the decay (it keeps about 94% of its starting value), while the weight with tiny gradients is crushed to almost zero. With AdamW both shrink to about the predicted 0.135. Any single run is noisy, because the Adam steps themselves wander, but the average shows the regularization each scheme really applies. Regularization strength should not depend on how noisy a parameter's gradient happens to be, and with AdamW it does not.

How long the weights remember

Decoupled decay has a clean interpretation that tells you how to set $\lambda$. Unroll the AdamW update. Each step multiplies the weights by $(1 - \eta\lambda)$ and then adds an Adam step $-\eta\,\mathbf{u}_t$, where $\mathbf{u}_t = \hat{\mathbf{m}}_t/(\sqrt{\hat{\mathbf{v}}_t} + \epsilon)$. After $T$ steps, starting from $\theta_0$,

In plain words, the weights are an exponentially weighted sum of past updates, in which an update made $s$ steps ago has been shrunk by $(1 - \eta\lambda)^s \approx e^{-\eta\lambda s}$. So the weights forget on a timescale of about $1/(\eta\lambda)$ steps, and the initialization fades on the same timescale. With $\eta = 3 \times 10^{-4}$ and $\lambda = 0.1$, that is about 33,000 steps, comparable to the length of a real pretraining run, which is no accident. If you double the learning rate, PyTorch's coupled form halves the timescale too, so the two hyperparameters must be tuned together.

The recipe at real scale

AdamW is the default optimizer for Transformers, and published recipes agree closely.

Three choices in that table deserve a reason. The lower $\beta_2 = 0.95$, against Adam's default of 0.999, makes the second-moment estimate average over about $1/(1 - \beta_2) = 20$ steps instead of 1,000, so it reacts quickly when gradients suddenly grow, which practitioners believe reduces loss spikes. The larger $\epsilon$ of Llama 2 puts a floor under the denominator for parameters whose gradients are tiny, so they cannot take full-size steps on noise. And the peak learning rate falls as models grow, by a factor of 10 from 125M to 175B parameters here, because wider layers sum more terms and react more strongly to the same per-parameter step (§9.5 makes this precise). It is also common not to decay biases and normalization gains, which are few and have no reason to be pulled toward zero. Newer optimizers for large models, such as Muon, are compared in §8.6.

import numpy as np

def adamw_step(theta, g, m, v, t, lr, beta1=0.9, beta2=0.999, eps=1e-8, weight_decay=0.01):
    raise NotImplementedError

Learning-rate warmup and decay

No fixed learning rate is right for a whole training run. Early on, a freshly initialized network has large, erratic gradients and Adam's second-moment estimates rest on only a few samples, so large steps are dangerous. Late in training, the noise in minibatch gradients keeps the parameters bouncing around a minimum, and only a smaller step lets them settle. So nearly every large model is trained with a schedule.

Warmup ramps the learning rate up from near zero over the first few hundred or thousand steps. It is close to essential for Transformers, especially with the original post-LayerNorm placement (Xiong et al., 2020), and Liu et al. (2020) traced part of its benefit to the high variance of Adam's early second-moment estimates. Decay then lowers the rate for the rest of training. The most common shape is a cosine curve from the peak $\eta_{\max}$ down to a floor $\eta_{\min}$ (often about 10% of the peak), over the remaining steps. With $w$ warmup steps and $T$ total steps,

In plain words, climb linearly to the peak, then glide down along half a cosine wave to the floor.

How long should warmup be? For Adam there is a principled default. The second-moment estimate averages over roughly $1/(1 - \beta_2)$ steps, so until that many steps have passed it rests on few samples. Ma & Yarats (2021) found that linear warmup over $2/(1 - \beta_2)$ steps works well without tuning, which is 2,000 steps for $\beta_2 = 0.999$ and 40 for $\beta_2 = 0.95$. Large runs use more, for stability of the whole network rather than of Adam's estimates, typically a few hundred to a few thousand steps (the table above), a fraction of a percent of training.

The decay needs to match the run. Hoffmann et al. (2022) found that a cosine whose cycle is much longer than the actual run leaves the model undertrained, because the rate never comes down, so a cosine schedule commits you to the run's length in advance. Linear decay to zero is a common alternative. A newer one, warmup-stable-decay (WSD, Hu et al., 2024), holds the peak rate flat for most of training and decays quickly at the end, which lets you branch off and finish training from any checkpoint. Schedules for pretraining runs are compared in §8.7.

import math

def lr_at(step, total_steps, warmup_steps, max_lr, min_lr):
    raise NotImplementedError

Gradient clipping

Occasionally a batch produces an enormous gradient, from an outlier example, a numerical blow-up, or a cliff in the loss surface (recurrent networks are famous for these, as Pascanu et al., 2013, showed). One such step can undo hours of training. Clipping by global norm caps the size of the whole update while keeping its direction,

where $c$ is the maximum norm, commonly 1.0 for language models. In plain words, if the total gradient is longer than $c$, shrink all of it proportionally so its length is exactly $c$.

The norm is computed over all parameters together, not per tensor, so the relative sizes of different layers' gradients are preserved. Clipping each value separately (clip_grad_value_) changes the direction and is rarely what you want. The pre-clipping norm is also one of the most useful numbers to log, since a rising or spiking gradient norm is often the first sign of an unstable run (§8.8).

Why clip when Adam already normalizes

Adam divides by $\sqrt{\hat{\mathbf{v}}}$, so multiplying every gradient by a constant changes nothing. Why would it need clipping at all? Because a spike is not a constant. Suppose a parameter's gradient has been a steady 1 and one step brings 100. On the spike step itself Adam is self-limiting, since $\mathbf{m}$ grows by $(1 - \beta_1) \cdot 100 = 10$ while $\sqrt{\mathbf{v}}$ grows to about $\sqrt{(1 - \beta_2)\cdot 10^4} = 22$ for $\beta_2 = 0.95$, so the step stays below $\eta$. The damage comes afterward. The spike sits in $\mathbf{v}$ and decays only by a factor of $\beta_2$ per step, so for the next hundred or more steps every update of that parameter is several times too small, and the model effectively stops training it. The cell shows it.

Without clipping, the spike step is 0.47 times normal, and then the updates sink to 0.13 times normal and take about 150 steps to recover. With clipping at twice the usual norm, nothing much happens. With $\beta_2 = 0.999$ the aftermath is milder per step but lasts about 3,000 steps, and late in training, once the bias correction has worn off, the spike step itself approaches $(1 - \beta_1)/\sqrt{1 - \beta_2} \approx 3.2$ times normal. Either way, clipping protects the optimizer's memory as much as the parameters.

Where clipping sits in the step matters too. It must act on the gradient of the whole accumulated batch, after the last micro-batch's backward pass and never on each micro-batch separately. With fp16 loss scaling the gradients must be unscaled first, or you clip a gradient that is thousands of times too large (§0.9). And when parameters are sharded across devices, each device holds only part of $\mathbf{g}$, so the global norm needs one all-reduce of the per-device sums of squares before anyone can scale (§10.3).

import numpy as np

def clip_grad_norm(grads, max_norm):
    raise NotImplementedError

Saddle points

A saddle point is a critical point (zero gradient) where the surface curves up in some directions and down in others, like $\mathcal{L}(x, y) = x^2 - y^2$ at the origin. Gradient descent started exactly on the $x$ axis walks straight into the saddle and stays there. Started a tiny distance $y_0$ off the axis, it escapes, but slowly. The $y$ coordinate grows by a factor of $1 + 2\eta$ per step, so escaping takes about $\log(1/|y_0|)/\log(1 + 2\eta)$ steps, during which the loss looks like a plateau. Long plateaus followed by sudden drops in training curves are often this.

Two things help. Noise in minibatch gradients kicks the parameters off the set of points that flow into the saddle and into its escape direction (Jin et al., 2017, showed that perturbed gradient descent escapes saddles efficiently), and momentum accelerates the escape once it starts. Choose the saddle surface in the race widget above and compare the optimizers.

Which optimizer, when

In PyTorch

Read more

• Kingma & Ba (2015). Adam: A Method for Stochastic Optimization. The algorithm and its bias correction (Section 3).
• Loshchilov & Hutter (2019). Decoupled Weight Decay Regularization. Why L2 and weight decay differ under Adam, and AdamW.
• Gabriel Goh (2017). Why Momentum Really Works. An interactive analysis of momentum on quadratics, with the square-root-of-kappa speedup.
• Sebastian Ruder (2016). An overview of gradient descent optimization algorithms. A compact survey of SGD, momentum, AdaGrad, RMSProp, and Adam.
• Brown et al. (2020), Language Models are Few-Shot Learners, and Touvron et al. (2023), Llama 2: Open Foundation and Fine-Tuned Chat Models. The optimizer settings in the recipe table (GPT-3 Appendix B, Llama 2 Section 2.2).
• Ma & Yarats (2021). On the adequacy of untuned warmup for adaptive optimization. The $2/(1 - \beta_2)$ warmup rule.
• Hoffmann et al. (2022), Training Compute-Optimal Large Language Models, and Hu et al. (2024), MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies. Why the cosine must match the run's length, and the warmup-stable-decay alternative.
• Goyal et al. (2017). Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour. The linear scaling rule and gradual warmup.
• McCandlish, Kaplan, Amodei et al. (2018). An Empirical Model of Large-Batch Training. The gradient noise scale and the critical batch size.
• Pascanu, Mikolov & Bengio (2013). On the difficulty of training recurrent neural networks. Exploding gradients and norm clipping.
• Dauphin et al. (2014). Identifying and attacking the saddle point problem in high-dimensional non-convex optimization. Why saddles, not local minima, dominate.

Watch next

▶

▶

▶

▶