Look at the learning rates in the GPT-3 paper. The 125-million-parameter model was trained with a peak learning rate of $6.0\times10^{-4}$, the 1.3-billion model with $2.0\times10^{-4}$, and the 175-billion model with $0.6\times10^{-4}$, ten times smaller than the first. Every size needed its own value, and nobody could tune the largest one properly, because a single trial run of a 175-billion-parameter model costs millions of dollars. The values for the big models were extrapolated by hand from the small ones. Over that range the width grew sixteenfold, from 768 to 12,288, while the learning rate fell tenfold, close to $d^{-0.83}$, so the drift is not a small correction. By the end of this section you will see why $d^{-1}$ is the natural rate for most of a network's weights and why no single global value can be right for all of them.

Every scaling law in this chapter rests on that step. A scaling study compares models of many sizes, and if the large ones are tuned worse than the small ones, the curve bends for reasons that have nothing to do with scale (§9.1). And every frontier run needs hyperparameters that are right the first time.

This section explains why the best learning rate drifts as a network gets wider, and how to stop it. The fix is a different parameterization, a rule for how initialization, layer multipliers, and per-layer learning rates should change with width, so that one set of hyperparameters means the same thing at every size. It is called the Maximal Update Parametrization, $\mu$P (Yang and Hu 2021), and the practice of tuning a small model and copying its hyperparameters to a large one is $\mu$Transfer (Yang et al. 2022). We derive the rules from a single argument, build the coordinate check that verifies an implementation, and then look at how well transfer works in practice, including across depth.

Two ways a sum can grow

Everything in $\mu$P follows from one fact about sums, so let's be careful with it. In this section $d$ is the width of a layer (its fan-in), and a vector "has coordinates of size $\Theta(1)$" if its typical entry neither grows nor shrinks as $d$ grows. Take a linear layer $\mathbf{h} = \mathbf{W}\mathbf{x}$ with input $\mathbf{x} \in \R^{d}$ whose coordinates are $\Theta(1)$, and look at one output, $h_i = \sum_{j=1}^{d} W_{ij}x_j$, a sum of $d$ terms.

At initialization the terms are independent. With $W_{ij} \sim \mathcal{N}(0, \sigma^2)$ independent of $\mathbf{x}$, the terms have random signs, so their sum grows like $\sqrt d$, by the central limit theorem. To keep $h_i = \Theta(1)$ we need $\sigma^2 = 1/d$. This is the familiar fan-in initialization of §0.7.

After an update the terms are aligned. A gradient step changes $\mathbf{W}$ by something built from $\mathbf{x}$ itself. For one example, $\partial\mathcal{L}/\partial\mathbf{W} = \mathbf{g}\mathbf{x}^\top$, where $\mathbf{g} = \partial\mathcal{L}/\partial\mathbf{h}$, so an SGD step with learning rate $\eta$ changes the output by

The $d$ terms of $\mathbf{x}^\top\mathbf{x}$ are all positive, so they add coherently, and $\lVert\mathbf{x}\rVert^2 \approx d\cdot\overline{x^2}$ grows like $d$, by the law of large numbers.

In plain words, random weights produce outputs that grow like $\sqrt d$, but learned weight changes produce outputs that grow like $d$, because the update is correlated with the very input it multiplies. Any rule that sets the initialization right but scales the learning rate wrong will see updates that grow or shrink with width.

Adam makes this even plainer. On its first step, with bias correction and a negligible $\epsilon$, Adam's update is $-\eta\,\operatorname{sign}(\partial\mathcal{L}/\partial W_{ij})$, the same size $\eta$ for every entry (§0.6). Then

The change in every output coordinate is proportional to $\eta d$. Double the width at a fixed learning rate and every hidden layer's activations move twice as far per step. To keep them moving by the same amount, a hidden layer's Adam learning rate must scale as $1/d$.

import numpy as np

def sgd_delta_h(x, g, eta):
    raise NotImplementedError

def adam_delta_h(x, g, eta):
    raise NotImplementedError

Why one learning rate cannot fit every layer

Now apply the same reasoning to every layer of a network whose hidden width is $d$.

• Input layer. Its fan-in is the input dimension, which does not grow with $d$ (for a Transformer's embedding, the input is a one-hot vector). The update $\Delta\mathbf{h} \propto \eta\cdot(\text{fan-in})$ does not depend on $d$, so its Adam learning rate should stay constant.
• Hidden layers. Fan-in $d$, so the Adam learning rate must shrink like $1/d$.
• Output layer. Fan-in $d$ as well, so the same $1/d$ applies. Its initialization needs a separate argument, below.

The standard parameterization uses one global learning rate. As width grows, either the hidden layers' updates blow up (if the rate is held fixed) or, if it is lowered like $1/d$ to stop the blow-up, the input layer's updates shrink toward zero and the embeddings barely learn. The best global compromise moves with width, which is exactly the drift in the GPT-3 table. This is a structural problem, not a tuning problem.

The abc-parameterization

To reason about all possible rules at once, Yang and Hu (2021) describe each layer $l$ by three exponents. Its weights are written as a fixed multiplier times trainable parameters, those parameters are initialized at a width-dependent scale, and the learning rate also scales with width,

Here $\mathbf{w}^l$ is what the optimizer sees, $d^{-a_l}$ is a constant multiplier applied in the forward pass, $b_l$ sets the initial scale of $\mathbf{w}^l$, and $\eta$ is a width-independent base learning rate. In plain words, every parameterization is a choice of how a layer's multiplier, its initial scale, and its learning rate change as the network widens. The standard parameterization (SP) is $a_l = 0$ everywhere, $b_1 = 0$ for the input layer, $b_l = \tfrac12$ for the rest, and one learning rate.

These three knobs are not independent, and seeing why is a good exercise in the chain rule.

Within this family, Yang and Hu asked which parameterizations stay stable as $d \to \infty$ (activations and outputs neither blow up nor vanish) and which let every layer's features change by $\Theta(1)$ during training. They proved a dynamical dichotomy. Every stable, nontrivial abc-parameterization either learns features or is in the kernel regime, where hidden representations stop moving in the infinite-width limit and the network behaves like a fixed kernel machine, never both. The standard parameterization can only stay stable with a learning rate that shrinks like $1/d$, and then it ends up in the kernel regime, as does the neural tangent parameterization. The one that updates every layer as much as possible without blowing up is $\mu$P, with $a_1 = -\tfrac12$ for the input layer, $a_l = 0$ for hidden layers, $a_{L+1} = \tfrac12$ for the output layer, $b_l = \tfrac12$ for every layer, and $c = 0$ for SGD.

The output layer and its initialization

The last piece is why $\mu$P initializes the output layer smaller than the standard $1/d$. Write the logits as $\mathbf{z} = \mathbf{W}_{\text{out}}\mathbf{h}$, with $\mathbf{W}_{\text{out}}$ initialized to entries of variance $\sigma^2$. When the hidden features change by $\Delta\mathbf{h}$, the logits change by $\mathbf{W}_{\text{out}}\Delta\mathbf{h}$ (plus the output layer's own update). Is that sum of $d$ terms random, growing like $\sqrt d$, or aligned, growing like $d$?

It is aligned, and a short calculation shows how much. The gradient that reaches $\mathbf{h}$ is $\mathbf{W}_{\text{out}}^\top\boldsymbol{\delta}$, where $\boldsymbol{\delta} = \partial\mathcal{L}/\partial\mathbf{z}$ has a fixed, width-independent number of entries. So the features move along $\mathbf{W}_{\text{out}}^\top\boldsymbol{\delta}$, say $\Delta\mathbf{h} = \kappa\,\mathbf{W}_{\text{out}}^\top\boldsymbol{\delta}$ for some scalar $\kappa$. The entries of $\mathbf{W}_{\text{out}}^\top\boldsymbol{\delta}$ have size $\sigma$, so for the feature changes to have coordinates of size $\Theta(1)$, as $\mu$P demands, $\kappa$ must be about $1/\sigma$. Then

because each diagonal entry of $\mathbf{W}_{\text{out}}\mathbf{W}_{\text{out}}^\top$ is a sum of $d$ squares, $\sum_j W_{ij}^2 \approx d\sigma^2$, while the off-diagonal entries are random sums and much smaller. For the logits to change by $\Theta(1)$ we need $d\sigma = \Theta(1)$, that is, $\sigma^2 = 1/d^2$, not $1/d$. A side effect is that the logits start near zero, of size $1/\sqrt d$. Yang et al. (2022) go further and recommend initializing the output layer, and the query projection of attention, exactly at zero.

The muP table

Putting it together, Table 3 of Yang et al. (2022) gives $\mu$P in a form you can implement directly, with no multipliers. Each entry is how the quantity scales with the layer's fan-in or fan-out, up to a constant you tune once. Standard-parameterization values are in parentheses where they differ.

A bias is treated as an input weight with fan-in 1 and fan-out equal to its length. For Transformers there is one more rule. Attention logits use $\mathbf{q}^\top\mathbf{k}/d_k$ instead of $\mathbf{q}^\top\mathbf{k}/\sqrt{d_k}$. The reason is the same law of large numbers. The $1/\sqrt{d_k}$ of §4.3 assumes queries and keys are independent, which holds at initialization, but training makes them correlated, so their dot product grows like $d_k$. The general rule, in Yang et al.'s words, is that "a multiplier of order 1/fan_in should accompany any weight that maps an infinite dimension to a finite one", which covers both the output logits and the attention logits.

In plain words, with Adam, keep the embedding learning rate fixed, divide every matrix's learning rate by its fan-in, shrink the output layer's initialization by an extra factor of the width, and divide attention scores by $d_k$.

In practice nobody writes "$\eta/\text{fan-in}$" with a raw width. You pick a small base model, say width $d_0$, tune it, and express everything through the width multiplier $m = d/d_0$, so that $\mu$P and SP coincide at the base width and the tuned values carry over exactly. A hidden layer of the big model then gets learning rate $\eta_0/m$ and initial standard deviation $\sigma_0/\sqrt m$, the output layer $\eta_0/m$ and $\sigma_0/m$, and the embedding keeps $\eta_0$.

def mup_scale(layer, optimizer, fan_in, fan_out, base_fan_in, base_fan_out, base_lr, base_std):
    return base_lr, base_std * (base_fan_in / fan_in) ** 0.5   # placeholder: standard parameterization

The same table from one condition on norms

The table looks like a list of special cases, but one requirement produces all of it. Yang, Simon, and Bernstein (2023) stated it in terms of the spectral norm $\lVert\mathbf{W}\rVert_{\text{op}}$, the largest singular value, which is the most a matrix can stretch any vector (§0.10). Think in vector lengths. A vector in $\R^n$ whose coordinates are $\Theta(1)$ has length about $\sqrt n$. A layer $\mathbf{W} \in \R^{\text{fan-out}\times\text{fan-in}}$ receives inputs of length about $\sqrt{\text{fan-in}}$ and should produce outputs, and output changes, of length about $\sqrt{\text{fan-out}}$. The spectral condition asks that

In plain words, both a weight matrix and each update to it should stretch vectors by the ratio of the output and input lengths, no more and no less.

Why the largest singular value, rather than the size of the entries? Because updates are aligned with their inputs. The first section showed that a gradient step is built from $\mathbf{x}$ itself, so $\Delta\mathbf{W}$ is low rank and points along $\mathbf{x}$, and then $\lVert\Delta\mathbf{W}\mathbf{x}\rVert$ is close to its upper bound $\lVert\Delta\mathbf{W}\rVert_{\text{op}}\lVert\mathbf{x}\rVert$. A random matrix is not aligned with $\mathbf{x}$, which is the CLT side of the same story. Each half of the table now follows in two lines.

Learning rates. Adam's first step on the rank-one gradient $\mathbf{g}\mathbf{x}^\top$ is $\Delta\mathbf{W} = -\eta\,\operatorname{sign}(\mathbf{g})\operatorname{sign}(\mathbf{x})^\top$, an outer product of two sign vectors, so $\lVert\Delta\mathbf{W}\rVert_{\text{op}} = \eta\sqrt{\text{fan-out}}\sqrt{\text{fan-in}}$. Setting it equal to $\sqrt{\text{fan-out}/\text{fan-in}}$ gives $\eta = 1/\text{fan-in}$, which is the Adam row of the table. Hidden and output matrices have fan-in $d$, so their learning rate falls like $1/d$, and the input layer's fan-in is fixed, so its learning rate is constant.

Initialization. A matrix with independent entries of standard deviation $\sigma$ has $\lVert\mathbf{W}\rVert_{\text{op}} \approx \sigma(\sqrt{\text{fan-in}} + \sqrt{\text{fan-out}})$, a standard result of random matrix theory. The condition then asks for

within a factor of 2. A square hidden layer gets $\sigma^2 \propto 1/d$. The output layer, with fan-in $d$ growing and fan-out fixed, gets $\sigma \approx \sqrt{\text{fan-out}}/d$, so $\sigma^2 \propto 1/d^2$, the $\mu$P output rule. An input layer whose width grows past its fixed fan-in gets $\sigma \to 1/\sqrt{\text{fan-in}}$, a constant. (A token embedding is a special case, because a one-hot input has length 1, not $\sqrt{\text{fan-in}}$. Practitioners keep the table's constant initialization there.)

The cell checks both claims on matrices of several shapes. It reports the spectral norm of a random matrix at that $\sigma$ and of Adam's first update at $\eta = 1/\text{fan-in}$, and how aligned each is with a random input, $\lVert\mathbf{M}\mathbf{x}\rVert/(\lVert\mathbf{M}\rVert_{\text{op}}\lVert\mathbf{x}\rVert)$.

Both norms land on the target for every shape. The alignment column is the point. The update is aligned with its input at about 0.8 whatever the shape (for sign vectors it is $\sqrt{2/\pi}$), so its spectral norm is what the output actually feels. The random initialization's alignment is 0.5 for square layers and shrinks as the fan-in of the output layer grows, which is why $\mu$P logits start near zero.

The condition is also a bridge to the optimizers of §8.6. Muon replaces each matrix's update by an orthogonal factor whose singular values are all 1, so its spectral norm is exactly 1, and the condition says to multiply it by $\sqrt{\text{fan-out}/\text{fan-in}}$. That is the shape scale $\sqrt{\max(1, A/B)}$ of Jordan's Muon for an $A\times B$ matrix (fan-out $A$, fan-in $B$), up to the floor at 1. Muon with that scale is, in this sense, $\mu$P-shaped by construction. The Moonshot scale of §8.6, $0.2\sqrt{\max(A, B)}$, matches AdamW's update size per entry instead, which gives a square $d\times d$ matrix a spectral norm of $0.2\sqrt d$, so by this argument its learning rate still has to shrink with width.

import numpy as np

def spectral_std(fan_in, fan_out):
    return 1 / np.sqrt(fan_in)               # placeholder: fan-in init, ignores the fan-out

def spectral_adam_lr(fan_in, fan_out):
    return 1e-3                              # placeholder: one learning rate for every layer

Coordinate checks

$\mu$P is easy to get wrong. One layer with the wrong multiplier, one learning rate forgotten in an optimizer group, a tied embedding handled as an output layer, or attention left at $1/\sqrt{d_k}$, and transfer quietly fails. The standard test is the coordinate check of Yang et al. (2022). In their words, "one calculates the average coordinate size of every (pre)activation vector in the network over a few steps of training, as width is varied over a large range. An incorrect implementation will see some activation vector blow up or shrink to zero with width."

The cell implements one. A three-layer ReLU network (input of dimension 16, two hidden layers of width $d$, one output) is trained with Adam for one step at five widths, and we measure how far each layer's activations moved, $\overline{\lvert\mathbf{h}_1 - \mathbf{h}_0\rvert}$. Under $\mu$P the movement should be the same at every width. We report the growth factor per doubling of width, which should be about 1.

Under the standard parameterization the first layer is fine, because its fan-in is fixed at 16, but the second layer's activations double with every doubling of width, exactly the $\eta d$ growth derived above, and the output grows even faster, because it compounds the larger feature changes with the larger output-layer update. Under $\mu$P all three stay flat. The widget runs the same check in your browser, with SGD as well as Adam, and over the first four steps.

A coordinate check should be the first thing you run after implementing $\mu$P, and a failing one tells you which layer is wrong. Here is the interview version of the task.

import numpy as np

def growth_slopes(measure, widths):
    raise NotImplementedError

def non_mup_layers(measure, widths, tol=0.25):
    return []                     # placeholder: declares everything correct

The two bugs are the classic ones. The first initializes the output layer with the standard $1/d$ variance, so the logits respond to feature changes like $\sqrt d$. The second forgets to divide the hidden layer's Adam learning rate by the width multiplier, so its features move like $d$.

muTransfer in practice

With a correct $\mu$P implementation, hyperparameter tuning becomes three steps.

1. Parameterize the target model in $\mu$P, relative to a chosen base shape, so that every layer's initialization and learning rate follows the table. Microsoft's mup package does this for PyTorch models, replacing the output layer with a "readout" that applies the $1/m$ multiplier and providing $\mu$P versions of SGD and Adam.
2. Tune a small proxy, the same architecture at a narrow width, with an ordinary hyperparameter search.
3. Copy the tuned values to the full model and train it once.

The torch cell at the end of this section runs a miniature version. Swept over learning rates at widths 64, 256, and 1,024, the best standard-parameterization learning rate drifts down as the model widens, while under $\mu$P it stays put and wider models are better at every learning rate.

Yang et al. (2022) report two headline results. "By transferring pretraining HPs from a model of 13M parameters, we outperform published numbers of BERT-large (350M parameters), with a total tuning cost equivalent to pretraining BERT-large once." And "by transferring from 40M parameters, we outperform published numbers of the 6.7B GPT-3 model, with tuning cost only 7% of total pretraining cost." The 40-million-parameter proxy was the 6.7B architecture shrunk to width 256, 168 times smaller, and the transferred model was comparable to the published 13-billion-parameter GPT-3. (Two caveats from the paper itself. Their own re-run of the baseline "mistakenly used absolute attention", so it is not an exact copy of GPT-3, and the $\mu$P model was trained in float32 after "frequent divergences", while the original and the re-run used float16.)

What transfers, and across what? Yang et al. list it carefully.

The caveats matter. Transfer across the other dimensions held in their experiments only above minimum sizes (for example width 256, depth 4, batch size 32, sequence length 128, and 5,000 training steps), depth transfer worked only for pre-LayerNorm Transformers, and the best initialization scale did not transfer well across depth. A further consequence is attractive. In their experiments, "wider is always better in $\mu$P but not in SP". At any fixed hyperparameters a wider $\mu$P model reached a lower training loss, so a wider model never needed retuning to beat a narrower one.

Weight decay needs a rule of its own. Wang and Aitchison (2024) read AdamW's decoupled decay as making the weights an exponential moving average of recent updates, with a timescale of about $1/(\eta\lambda)$ steps for decay $\lambda$, and found that the best timescale, counted in epochs, stays roughly constant as model and dataset sizes change. Two rules follow. Under $\mu$P a hidden layer's $\eta$ falls like $1/d$, so to keep $\eta\lambda$ fixed its $\lambda$ must grow with width. And on more data, with more steps per epoch, $\lambda$ must shrink. Coupled decay (the default in many codebases, where the decay is multiplied by the learning rate) silently changes the timescale with width, and one of the independent tests below found that it breaks transfer.

Independent tests since then have been mostly positive, with instructive failures.

• Cerebras-GPT (Dey et al. 2023) trained models from 111 million to 2.7 billion parameters with $\mu$P, tuned on a 40-million-parameter proxy, and reported Pile test losses 0.43 percent better and downstream accuracy 1.7 percent higher than their standard-parameterization models, with far smaller deviations from a fitted scaling law.
• MiniCPM (Hu et al. 2024) used $\mu$P-style width and depth scaling and found the best learning rate stayed near 0.01 from 40 million to 500 million parameters, confirmed at 2.1 billion.
• Lingle (2024) tested learning-rate transfer up to 10 billion parameters and found it "worked as promised in most cases", with a slight drift at the largest scale, but not with trainable RMSNorm gains, not with the standard $1/\sqrt{d_k}$ attention scale, and not with the standard form of weight decay. Decoupling weight decay from the learning rate, or leaving it out of the proxy, helped.
• Everett et al. (2024) trained models up to 26.8 billion parameters and found that with the right per-layer learning rates, "all parameterizations, not just muP, can achieve hyperparameter transfer", and that a per-layer prescription for the standard parameterization did even better. They also found that Adam's $\epsilon$ has to be scaled with width, because the gradients of wide layers shrink until a fixed $\epsilon$ dominates the denominator and quietly changes the update, and proposed Adam-atan2, which removes $\epsilon$ altogether.

The lesson from these studies is that the essential content of $\mu$P is the per-layer learning-rate scaling, which is what the coordinate check verifies.

Transfer across depth

$\mu$P fixes width. Depth needs its own rule, because adding blocks adds terms to the residual stream (§5.1), and a stream that is the sum of $L$ branches grows with $L$ unless each branch is scaled down. You saw one version of this in §5.7, GPT-2's $1/\sqrt{2L}$ initialization of the output projections. That fixes the scale at initialization, but, as with width, not the scale of the updates.

Depth-$\mu$P (Yang et al. 2023, and independently Bordelon et al. 2023) multiplies every residual branch by $a/\sqrt L$ in the forward pass,

where $g^{(l)}$ is the block and $a$ a tunable constant, and adjusts the learning rate. For SGD it stays $\eta$, because the multiplier already shrinks each block's gradient by $1/\sqrt L$, and for Adam, which ignores the gradient's scale, it becomes $\eta/\sqrt L$. Bordelon et al. showed that this, combined with $\mu$P, transfers optimal hyperparameters across both width and depth in residual networks and vision Transformers. Yang et al. found a limitation, though. With one layer per residual block the parameterization is optimal, but blocks with two or more layers, like a Transformer's attention and MLP, show fundamental limitations in every infinite-depth limit, and their optimal hyperparameters still shifted with depth.

CompleteP (Dey et al. 2025) revisits the choice and scales branches by $1/L$ instead, $\mathbf{x}^{(l)} = \mathbf{x}^{(l-1)} + L^{-1}g^{(l)}(\mathbf{x}^{(l-1)})$, with matching adjustments to learning rates, LayerNorm parameters, weight decay, and Adam's $\epsilon$. They report that only this choice gave reliable depth-wise transfer, from 2 to 128 layers, and that it enabled 12 to 34 percent compute savings over the previous best parameterization by making deeper, narrower models trainable at their best settings.

A different strategy is to skip transfer and fit hyperparameters directly as functions of scale. DeepSeek's scaling study (DeepSeek-AI 2024) fit the best learning rate and batch size against compute, finding $\eta_{\text{opt}} = 0.3118\,C^{-0.1250}$ and $B_{\text{opt}} = 0.2920\,C^{0.3271}$, and extrapolated them to its large runs. It is the scaling-law approach of this chapter applied to the hyperparameters themselves, and it works with the standard parameterization, at the cost of many small sweeps. Learning-rate and batch-size scaling are taken up again in §8.7.

In PyTorch

A miniature $\mu$Transfer experiment. A student MLP learns a fixed random teacher network, at widths 64, 256, and 1,024, under the standard parameterization and under $\mu$P with Adam. $\mu$P uses per-layer parameter groups, the hidden and output learning rates divided by the width multiplier, and a zero-initialized output layer.

The standard parameterization's best learning rate falls by a factor of about three as the width grows sixteenfold, and its widest model is worse than its narrowest at the largest learning rate. Under $\mu$P the best learning rate is the same at every width, and at every learning rate the wider model wins, the "wider is better" property in miniature.

Read more

• Yang & Hu (2021). Feature Learning in Infinite-Width Neural Networks. Tensor Programs IV, the abc-parameterization, the dichotomy theorem, and the derivation of $\mu$P.
• Yang et al. (2022). Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer. The $\mu$P tables, coordinate checks, and the GPT-3 and BERT transfer results.
• Yang et al. (2023). Tensor Programs VI: Feature Learning in Infinite-Depth Neural Networks. Depth-$\mu$P and the limits for multi-layer blocks.
• Bordelon et al. (2023). Depthwise Hyperparameter Transfer in Residual Networks. $1/\sqrt{L}$ branch scaling combined with $\mu$P.
• Everett et al. (2024). Scaling Exponents Across Parameterizations and Optimizers. Per-layer learning rates for every parameterization, the $\epsilon$ problem, and Adam-atan2.
• Lingle (2024). An Empirical Study of μP Learning Rate Transfer. What breaks transfer in practice, up to 10B parameters.
• Dey et al. (2025). Don't be lazy: CompleteP enables compute-efficient deep transformers. Depth transfer with $1/L$ branch scaling.
• Yang, Simon & Bernstein (2023). A Spectral Condition for Feature Learning. The whole $\mu$P table from one condition on spectral norms.
• Wang & Aitchison (2024). How to set AdamW's weight decay as you scale model and dataset size. Weight decay as an averaging timescale, and how it should scale.
• Microsoft. The mup package. A reference PyTorch implementation, including coordinate-check tools.

Watch next

▶

▶

▶