Chapter 14 · Section 14.5 · 60 min

LoRA

Suppose a hundred customers each want their own fine-tuned version of a 175B model. Full fine-tuning gives you a hundred checkpoints of 350 GB each, 35 TB of weights, and a hundred copies of the model to keep in GPU memory if you want to serve them all. Hu et al. (2021) showed you could do it differently. Freeze the model, and for each customer train a pair of skinny matrices beside some of its weight matrices. On GPT-3 175B, with rank 4 adapters on only the query and value projections, each customer's checkpoint shrank from 350 GB to about 35 MB, a factor of about 10,000, and a hundred of them fit in 354 GB alongside one shared copy of the base. Training memory fell from 1.2 TB to 350 GB, and quality matched full fine-tuning on their benchmarks.

That method is LoRA, low-rank adaptation, and it is now the default way to fine-tune when memory or the number of variants matters. This section explains why a tiny update can be enough, derives the forward and backward passes, explains the two details everyone asks about in interviews (why $\mathbf{B}$ starts at zero and why the update is scaled by $\alpha/r$ ), and covers which layers to adapt, how to merge adapters, and how to build LoRA from scratch.

Why a small update can be enough

Start with an observation about fine-tuning, not about LoRA. Li et al. (2018) measured the intrinsic dimension of a training problem by freezing the weights at their initial values $\theta_0$ and training only a vector $\mathbf{z} \in \R^{k}$ that is mapped into the full space by a fixed random matrix $\mathbf{P}$ ,

\theta = \theta_0 + \mathbf{P}\mathbf{z}, \qquad \mathbf{P} \in \R^{N \times k},\ k \ll N,

and finding the smallest $k$ , called $d_{90}$ , that reaches 90% of the performance of training all $N$ parameters of the model. Aghajanyan et al. (2021) applied this to fine-tuning pretrained language models and found the numbers startlingly small. With RoBERTa-Large, about 200 parameters projected into the full space reached 90% of full fine-tuning on the MRPC paraphrase task. Larger pretrained models had lower intrinsic dimension, and pretraining longer lowered it further. In plain words, pretraining leaves the model in a place from which a task needs only a few independent adjustments.

Why could that be? A random $k$ -dimensional subspace can contain a good solution only if success depends on about $k$ constraints. Here is a linear model with 1,000 weights whose fine-tuning data lives in a 20-dimensional subspace of its inputs, a cartoon of how narrow fine-tuning data is compared with everything a pretrained model can represent. We fit the best $\mathbf{z}$ in random subspaces of growing dimension.

Python · cell 1

At $k = 20$ the random subspace matches full fine-tuning exactly, and adding more dimensions gains nothing, because only 20 combinations of weights affect the task's predictions at all. Below 20 the subspace is too small to reach the target, and how close it gets depends on the luck of the draw (here $k = 10$ happens to do worse than $k = 5$ ). The other 980 directions are free. Real networks are not linear and their "task subspace" is not exact, but the experiments say the effective number is small. LoRA turns this observation into a structured, efficient parameterization.

The low-rank update

Take one weight matrix of the model, in PyTorch's convention $\mathbf{W}_0 \in \R^{d_{\text{out}}\times d_{\text{in}}}$ , so that a layer computes $\mathbf{h} = \mathbf{W}_0\mathbf{x}$ for a column input $\mathbf{x} \in \R^{d_{\text{in}}}$ (and $\mathbf{Y} = \mathbf{X}\mathbf{W}_0^\top$ for a batch $\mathbf{X} \in \R^{n\times d_{\text{in}}}$ of row vectors). Full fine-tuning learns an arbitrary update $\Delta\mathbf{W}$ . LoRA freezes $\mathbf{W}_0$ and restricts the update to a product of two thin matrices,

\mathbf{h} = \mathbf{W}_0\mathbf{x} + \gamma\,\mathbf{B}\mathbf{A}\mathbf{x}, \qquad \mathbf{B} \in \R^{d_{\text{out}}\times r},\quad \mathbf{A} \in \R^{r\times d_{\text{in}}},\quad \gamma = \frac{\alpha}{r},

with rank $r \ll \min(d_{\text{in}}, d_{\text{out}})$ and a constant $\alpha$ . In plain words, $\mathbf{A}$ compresses the input to $r$ numbers, $\mathbf{B}$ expands those back to the output width, and the result is added to what the frozen layer computes. The update $\Delta\mathbf{W} = \gamma\mathbf{B}\mathbf{A}$ has rank at most $r$ . (This $\mathbf{A}$ is not the attention-weight matrix of Chapter 4, and $\mathbf{B}$ is not the batch size $B$ . The names come from the LoRA paper.)

Three things follow from the shapes.

Parameters. LoRA trains $r(d_{\text{in}} + d_{\text{out}})$ numbers instead of $d_{\text{in}}d_{\text{out}}$ . For a $4096\times4096$ projection with $r = 8$ that is 65,536 instead of 16.8 million, 0.39%.
Compute. Never form $\mathbf{B}\mathbf{A}$ during training. Computing $\mathbf{B}(\mathbf{A}\mathbf{x})$ costs $r(d_{\text{in}} + d_{\text{out}})$ multiply-adds per token instead of the $d_{\text{in}}d_{\text{out}}$ of a full matrix, the contraction-order argument of §0.1. The frozen matmul still runs, so LoRA saves memory far more than it saves compute. Count it per token for a model with $N$ parameters. Full fine-tuning costs about $6N$ FLOPs, $2N$ forward, $2N$ for the gradient with respect to each layer's input, and $2N$ for the gradient with respect to its weights (the accounting of §9.2). A frozen matrix still needs the input gradient, so backpropagation can reach earlier adapters, but not the weight gradient, so LoRA costs about $4N$ , a third less. Hu et al. measured a 25% training speedup on GPT-3 175B.
Memory. The frozen $\mathbf{W}_0$ needs no gradient, no master copy, and no optimizer state, so its cost drops from 16 bytes per parameter to 2 (§14.4). Only the adapter pays the full 16.

How good can a rank- $r$ update be? If you knew the update that full fine-tuning would make, the best rank- $r$ approximation in Frobenius norm is its truncated SVD, and the error is set by the singular values left out (the Eckart-Young theorem, proved in §0.10 and used on word counts in §1.4),

\min_{\operatorname{rank}(\mathbf{M}) \le r}\ \big\lVert \Delta\mathbf{W} - \mathbf{M} \big\rVert_F^2 = \sum_{i > r} \sigma_i^2 .

So the question "is LoRA enough" is partly the question "how fast do the singular values of real updates fall". Here are real ones, from full fine-tuning of the small Shakespeare GPT used in §14.4.

The updates are far from random. A rank of 12 to 14 captures 80% of the squared norm of the attention updates, where a random matrix of the same shape needs 50 to 74. But they are not truly low rank either. Their spectra have long tails, and the MLP up-projection of layer 3 needs about 40 directions for 80%. Biderman et al. (2024) found the same at scale. On code and math, full fine-tuning learned perturbations with ranks 10 to 100 times higher than typical LoRA configurations, and LoRA learned less of the target domain (and forgot less). LoRA does not reproduce the full fine-tuning update. It finds its own low-rank solution, which is as good when the task needs little capacity and falls short when it needs a lot, a point Schulman et al. (2025) made quantitatively by comparing LoRA's parameter count with the information content of the training data.

The same study adds two empirical findings that interviewers like to probe. LoRA was less tolerant of large batches than full fine-tuning, with a loss penalty that grew with the batch size and did not shrink at higher rank, so a comparison run at a batch size tuned for full fine-tuning can make LoRA look worse than it is. And in reinforcement learning, LoRA matched full fine-tuning even at rank 1. Their explanation is the capacity argument again. A policy-gradient update learns from one scalar reward per sampled episode, on the order of one bit of information, while a supervised example supplies a whole target sequence, so reinforcement learning needs far less room in the adapter (§15.7 sets up that view of training).

Initialization, and why B starts at zero

LoRA initializes $\mathbf{A}$ randomly and $\mathbf{B}$ to zero (Hu et al. used a Gaussian for $\mathbf{A}$ , and the PEFT library uses a Kaiming-uniform draw). Three reasons, each of which an interviewer will probe.

The model starts exactly at the base. With $\mathbf{B} = 0$ , $\Delta\mathbf{W} = 0$ , so step 0 of fine-tuning is the pretrained model, with its quality intact. Random initialization of both would add a random perturbation to every adapted layer before training even begins.

Both cannot be zero. To see why, we need the gradients. Write the batched layer as $\mathbf{Y} = \mathbf{X}\mathbf{W}_0^\top + \gamma(\mathbf{X}\mathbf{A}^\top)\mathbf{B}^\top$ and let $\mathbf{G} = \partial\mathcal{L}/\partial\mathbf{Y} \in \R^{n\times d_{\text{out}}}$ be the upstream gradient. Using the matrix-calculus rule that for $\mathbf{Y} = \mathbf{U}\mathbf{V}$ the gradients are $\mathbf{G}\mathbf{V}^\top$ and $\mathbf{U}^\top\mathbf{G}$ (§0.2), with $\mathbf{Z} = \mathbf{X}\mathbf{A}^\top \in \R^{n\times r}$ the compressed input,

\frac{\partial\mathcal{L}}{\partial\mathbf{B}} = \gamma\,\mathbf{G}^\top\mathbf{Z} = \gamma\,\mathbf{G}^\top\mathbf{X}\mathbf{A}^\top, \qquad \frac{\partial\mathcal{L}}{\partial\mathbf{A}} = \gamma\,(\mathbf{G}\mathbf{B})^\top\mathbf{X}, \qquad \frac{\partial\mathcal{L}}{\partial\mathbf{X}} = \mathbf{G}\mathbf{W}_0 + \gamma\,(\mathbf{G}\mathbf{B})\mathbf{A}.

In plain words, $\mathbf{B}$ 's gradient is proportional to $\mathbf{A}$ , and $\mathbf{A}$ 's gradient is proportional to $\mathbf{B}$ . If both are zero, both gradients are zero, and training never moves. That point is a stationary point (in general a saddle), and gradient descent started exactly there cannot leave it.

Only B moves on the first step. With $\mathbf{B} = 0$ the gradient of $\mathbf{A}$ is zero, so the first update changes $\mathbf{B}$ only, by $\Delta\mathbf{B} = -\eta\gamma\,\mathbf{G}^\top\mathbf{X}\mathbf{A}^\top$ under plain gradient descent. The resulting weight update is

\Delta\mathbf{W} = \gamma\,\Delta\mathbf{B}\,\mathbf{A} = -\eta\gamma^2\,\underbrace{\mathbf{G}^\top\mathbf{X}}_{\text{full gradient of } \mathbf{W}}\ \mathbf{A}^\top\mathbf{A}.

In plain words, LoRA's first step is the full fine-tuning gradient multiplied on the right by $\mathbf{A}^\top\mathbf{A}$ , a $d_{\text{in}}\times d_{\text{in}}$ matrix of rank $r$ that keeps only the part of each row lying in the $r$ -dimensional row space of the random matrix $\mathbf{A}$ (it is exactly a projection when the rows of $\mathbf{A}$ are orthonormal, and close to a scaled projection for a random $\mathbf{A}$ with $r \ll d_{\text{in}}$ , whose rows are nearly orthogonal). That is exactly the random-subspace training of the intrinsic dimension experiments, applied matrix by matrix. After the first step $\mathbf{A}$ starts to move too, and the subspace rotates toward useful directions, which is what makes LoRA better than a fixed random projection.

Interview

LoRA forward pass, merge, and unmerge

Implement lora_forward(X, W0, A, B, alpha), merge(W0, A, B, alpha), and unmerge(W, A, B, alpha). Shapes follow PyTorch, W0 is (d_out, d_in), A is (r, d_in), B is (d_out, r), and X is (n, d_in) (rows are tokens). The rank $r$ is read from A, and the scale is $\gamma = \alpha/r$ .

lora_forward returns $\mathbf{X}\mathbf{W}_0^\top + \gamma(\mathbf{X}\mathbf{A}^\top)\mathbf{B}^\top$ , without forming $\mathbf{B}\mathbf{A}$ .
merge returns the single matrix $\mathbf{W}_0 + \gamma\mathbf{B}\mathbf{A}$ that a deployed model would use.
unmerge recovers $\mathbf{W}_0$ from a merged matrix.

import numpy as np

def lora_forward(X, W0, A, B, alpha):
    raise NotImplementedError

def merge(W0, A, B, alpha):
    raise NotImplementedError

Hidden tests, hints, and AI help, from $1.99 a month.

Interview

LoRA backward pass, checked numerically

Implement lora_backward(X, W0, A, B, alpha, G) returning (dA, dB, dX), the gradients of a loss with respect to A, B, and X, given G $= \partial\mathcal{L}/\partial\mathbf{Y}$ of shape (n, d_out) for the layer $\mathbf{Y} = \mathbf{X}\mathbf{W}_0^\top + \gamma(\mathbf{X}\mathbf{A}^\top)\mathbf{B}^\top$ . W0 is frozen, so it gets no gradient. Each gradient has the shape of the thing it is the gradient of.

import numpy as np

def lora_backward(X, W0, A, B, alpha, G):
    raise NotImplementedError

Hidden tests, hints, and AI help, from $1.99 a month.

The scale alpha over r, and rsLoRA

Why multiply by $\gamma = \alpha/r$ at all, when $\mathbf{B}$ could absorb any constant? Because the constant changes how fast training moves as you change $r$ , and you want to change $r$ without retuning everything else. Hu et al. set $\alpha$ to the first rank they tried and never tuned it, noting that with Adam, tuning $\alpha$ is roughly the same as tuning the learning rate. Kalajdzievski (2023) argued instead for $\gamma = \alpha/\sqrt{r}$ , which he called rank-stabilized LoRA (rsLoRA), proving for gradient descent that only this choice keeps the adapter's learning neither exploding nor collapsing as $r \to \infty$ , and showing that with $\alpha/r$ larger ranks learned no faster than small ones. Both are right, for different optimizers, and one step of algebra shows why.

∂ How much one step reduces the loss, as a function of r

Start from $\mathbf{B} = 0$ , where only $\mathbf{B}$ has a gradient, $\nabla_{\mathbf{B}} = \gamma\,\mathbf{G}^\top\mathbf{Z}$ with $\mathbf{Z} = \mathbf{X}\mathbf{A}^\top$ . A step $\Delta\mathbf{B}$ changes the output by $\Delta\mathbf{Y} = \gamma\,\mathbf{Z}\,\Delta\mathbf{B}^\top$ and, to first order, the loss by

\Delta\mathcal{L} = \langle \mathbf{G}, \Delta\mathbf{Y}\rangle = \gamma \operatorname{tr}\!\big(\mathbf{G}^\top\mathbf{Z}\,\Delta\mathbf{B}^\top\big) = \langle \gamma\,\mathbf{G}^\top\mathbf{Z},\ \Delta\mathbf{B}\rangle = \langle \nabla_{\mathbf{B}},\ \Delta\mathbf{B}\rangle .

Plain gradient descent, $\Delta\mathbf{B} = -\eta\nabla_{\mathbf{B}}$ . Then $\Delta\mathcal{L} = -\eta\gamma^2\lVert\mathbf{G}^\top\mathbf{X}\mathbf{A}^\top\rVert_F^2$ . Each of the $r$ rows of $\mathbf{A}$ contributes an independent, identically distributed term to this squared norm, so it grows like $r$ , and $\Delta\mathcal{L} \propto \eta\gamma^2 r$ . With $\gamma = \alpha/r$ that is $\eta\alpha^2/r$ , which shrinks as $r$ grows, the collapse rsLoRA describes. With $\gamma = \alpha/\sqrt r$ it is $\eta\alpha^2$ , independent of $r$ .

Adam, whose first steps are close to sign descent because $\hat{\mathbf{m}}/\sqrt{\hat{\mathbf{v}}} \approx \operatorname{sign}(\mathbf{g})$ (§0.6), takes $\Delta\mathbf{B} \approx -\eta\operatorname{sign}(\nabla_{\mathbf{B}})$ . Then $\Delta\mathcal{L} = -\eta\lVert\nabla_{\mathbf{B}}\rVert_1 = -\eta\gamma\lVert\mathbf{G}^\top\mathbf{X}\mathbf{A}^\top\rVert_1$ , a sum of $d_{\text{out}}\cdot r$ absolute values whose typical size does not depend on $r$ , so $\Delta\mathcal{L} \propto \eta\gamma r$ . With $\gamma = \alpha/r$ that is $\eta\alpha$ , independent of $r$ . With $\gamma = \alpha/\sqrt r$ it is $\eta\alpha\sqrt r$ , which grows with the rank, so a larger rank acts like a larger learning rate.

Let's check both predictions numerically on one layer.

Python · cell 2

Under plain gradient descent, $\alpha/\sqrt r$ gives the same progress at every rank and $\alpha/r$ gives 64 times less at rank 256 than at rank 4. Under Adam the roles swap. $\alpha/r$ is the rank-independent choice, and $\alpha/\sqrt r$ makes rank 256 move 8 times faster than rank 4. That reconciles the two papers and matches large-scale practice. Schulman et al. (2025), training with Adam, found that with the standard $\alpha/r$ the optimal learning rate changed by less than a factor of two between ranks 4 and 512, and that early learning curves at the same learning rate were nearly identical across ranks. If you use rsLoRA (PEFT's use_rslora=True), remember that raising $r$ also raises the effective step size, so retune the learning rate when you change the rank.

Two practical consequences. Since $\alpha$ and the learning rate are entangled, fix $\alpha$ (common choices are $\alpha = 16$ , $\alpha = 32$ , or $\alpha = 2r$ ) and tune the learning rate, which is what the QLoRA authors did. And LoRA's best learning rate is higher than full fine-tuning's, by about an order of magnitude (Biderman et al. 2024 and Schulman et al. 2025 both found roughly 10 times), which is why QLoRA used $2\times10^{-4}$ for 7B and 13B models where full fine-tuning would use $2\times10^{-5}$ . LoRA+ (Hayou et al. 2024) goes one step further and gives $\mathbf{B}$ a larger learning rate than $\mathbf{A}$ , recommending a ratio of $2^4 = 16$ , motivated by the same asymmetry between the two factors that the derivation exposes.

Which layers to adapt

The original paper, constrained to a fixed budget of trainable parameters on GPT-3, found adapting both $\mathbf{W}_Q$ and $\mathbf{W}_V$ at rank 4 better than putting all of the budget into $\mathbf{W}_Q$ or $\mathbf{W}_K$ alone at rank 8, and adapting all four attention matrices at rank 2 as good or slightly better (Hu et al. 2021). Spreading rank across matrices beats concentrating it. Later work pushed this to its conclusion.

QLoRA's authors could not match full fine-tuning on LLaMA when adapting only the query and value projections, the paper's default. Adapting all linear layers of every block was the critical choice, and once they did, the rank $r$ hardly mattered (Dettmers et al. 2023).
Schulman et al. (2025) found attention-only LoRA significantly worse than MLP-only LoRA, even at matched parameter counts (attention-only at rank 256 underperformed MLP-only at rank 128), while adding attention to the MLP adapters gave no further gain.

So the modern default is every linear layer of every block, the query, key, value, and output projections and the MLP's gate, up, and down projections, which PEFT spells target_modules="all-linear". Embeddings and the output head are usually left frozen. The exception is when you add tokens, such as new chat-template markers. Their embedding rows start as an untrained mean (§2.5), and an adapter on the linear layers cannot give them distinct meanings, so those rows (or the whole embedding and head, via PEFT's modules_to_save) must be trained too. The small GPT behind this chapter's widgets did exactly that, training the three new token rows alongside its LoRA adapters.

Core

Count LoRA's trainable parameters for a LLaMA config

Implement lora_param_count(cfg, r, modules) for a LLaMA-style decoder. cfg has d_model, d_ff, n_layers, n_heads, and n_kv_heads, with head size d_model // n_heads. The per-block linear layers and their (input, output) widths are

q and o, $d \to d$ ,
k and v, $d \to n_{kv} \cdot d_h$ (grouped-query attention, §4.5),
gate and up, $d \to d_{\text{ff}}$ , and down, $d_{\text{ff}} \to d$ .

LoRA on a $d_{\text{in}} \to d_{\text{out}}$ layer adds $r(d_{\text{in}} + d_{\text{out}})$ parameters. Return the total for the listed modules across all layers, as a Python int.

def lora_param_count(cfg, r, modules):
    raise NotImplementedError

Hidden tests, hints, and AI help, from $1.99 a month.

Merging, swapping, and serving adapters

Because $\Delta\mathbf{W} = \gamma\mathbf{B}\mathbf{A}$ has the same shape as $\mathbf{W}_0$ , you can fold it in after training, $\mathbf{W} = \mathbf{W}_0 + \gamma\mathbf{B}\mathbf{A}$ , and the deployed model has exactly the base architecture and zero added latency, an advantage over adapters that insert new layers (§14.6). You can also keep adapters separate and swap them. Unmerging subtracts the same product, and many adapters can share one copy of the base in memory, which is how a single GPU serves many fine-tuned variants, a technique serving systems for multi-adapter traffic build on by batching requests for different adapters together.

Three cautions. Merging in bfloat16 rounds the sum, and since $\gamma\mathbf{B}\mathbf{A}$ is small relative to $\mathbf{W}_0$ , part of it can round away (the same 8-bit-mantissa issue that forces master weights in mixed precision, §10.1), so merge in float32 and cast once. Merging into a 4-bit quantized base is lossy, because the merged matrix must be requantized and the adapter was trained against the quantized weights, not the merged ones. And adding several adapters together (task arithmetic) works sometimes and interferes other times. Treat merged combinations as new models that need evaluation.

LoRA from scratch

Here is a LoRA layer in NumPy, with the forward and backward passes derived above, trained with Adam toward a target update that is rank 4 plus a little full-rank noise. The inputs are whitened, so minimizing the regression loss is the same as minimizing $\lVert\gamma\mathbf{B}\mathbf{A} - \Delta\mathbf{W}_{\text{true}}\rVert_F$ , and we can compare the result with the Eckart-Young optimum for each rank.

Python · cell 3

At every rank, gradient descent on $\mathbf{A}$ and $\mathbf{B}$ finds essentially the best rank- $r$ update the SVD allows. The error drops sharply up to rank 4, where the true update's structure lives, and slowly after, as the extra ranks fit the noise. This is no accident. For this objective every local minimum of the factored problem is a global one, a classical theorem for linear networks (Baldi and Hornik 1989), and that benign geometry is part of why LoRA trains so reliably.

The real way

A LoRA layer in PyTorch is a wrapper around nn.Linear. Here it is, applied to every linear layer of a small model, with the parameter count, a few training steps, and a merge that must leave the outputs unchanged.

🔥 In PyTorchfor a machine with PyTorch

With Hugging Face PEFT the same thing is a configuration.

⚠️ LoRA pitfalls

Full fine-tuning's learning rate. LoRA wants roughly 10 times more. A LoRA run at $2\times10^{-5}$ looks like it is not learning.
Attention-only adapters. The original default adapts only the query and value projections. Adapt all linear layers, especially the MLP.
New tokens with frozen embeddings. Added special tokens stay identical, untrained vectors. Train their rows.
Changing the rank with rsLoRA and keeping the learning rate. With $\alpha/\sqrt r$ , a larger rank takes larger steps.
Merging in low precision or into a quantized base. Merge in float32, and evaluate the merged model, not only the adapter.
Expecting LoRA to reproduce full fine-tuning on capacity-hungry tasks. Large code or math corpora can exceed what a low-rank adapter learns. Raise the rank, or use full fine-tuning.

🎯 Interview questions on this section

Explain LoRA. What are the trainable parameters, their shapes, and the forward pass?

For a frozen weight $\mathbf{W}_0 \in \R^{d_{\text{out}}\times d_{\text{in}}}$ , LoRA learns $\mathbf{B} \in \R^{d_{\text{out}}\times r}$ and $\mathbf{A} \in \R^{r\times d_{\text{in}}}$ and computes $\mathbf{h} = \mathbf{W}_0\mathbf{x} + \frac{\alpha}{r}\mathbf{B}\mathbf{A}\mathbf{x}$ , evaluated as $\mathbf{B}(\mathbf{A}\mathbf{x})$ without forming $\mathbf{B}\mathbf{A}$ . It trains $r(d_{\text{in}}+d_{\text{out}})$ parameters per matrix instead of $d_{\text{in}}d_{\text{out}}$ , needs no optimizer state for the base, and can be merged into $\mathbf{W}_0$ after training for zero inference overhead.

Why is B initialized to zero and A randomly? What happens if both are zero, or both random?

With $\mathbf{B} = 0$ the update is zero, so training starts exactly at the pretrained model. The gradients are $\partial\mathcal{L}/\partial\mathbf{B} = \gamma\mathbf{G}^\top\mathbf{X}\mathbf{A}^\top$ and $\partial\mathcal{L}/\partial\mathbf{A} = \gamma(\mathbf{G}\mathbf{B})^\top\mathbf{X}$ , so if both are zero both gradients vanish and training is stuck at a saddle. If both are random the model starts with a random perturbation of every adapted layer. With $\mathbf{B} = 0$ , the first step moves only $\mathbf{B}$ , and the resulting update is the full gradient projected onto the random row space of $\mathbf{A}$ .

Why scale by alpha over r? How does rsLoRA differ, and which should you use?

The scale controls how fast the adapter changes the model as $r$ changes. Under Adam, whose early steps are sign-like, the first-step loss decrease is proportional to $\gamma r$ , so $\gamma = \alpha/r$ makes it rank-independent, and the optimal learning rate barely moves with rank. Under SGD it is proportional to $\gamma^2 r$ , so $\alpha/\sqrt r$ is the rank-stable choice, which is rsLoRA's analysis. With Adam, rsLoRA's $\alpha/\sqrt r$ makes larger ranks take larger steps, which can help high ranks but means the learning rate must be retuned with $r$ . Fix $\alpha$ and tune the learning rate either way.

Which layers should LoRA adapt?

All linear layers of every block, the attention projections and especially the MLP. The original paper adapted only $\mathbf{W}_Q$ and $\mathbf{W}_V$ , but QLoRA found all linear layers necessary to match full fine-tuning, after which the rank hardly mattered, and Schulman et al. (2025) found attention-only adapters clearly worse than MLP-only ones at the same parameter count. Train embedding rows too if you add tokens.

Count the LoRA parameters for rank 16 on all linear layers of Llama 3 8B.

Per block, q and o are $4096\to4096$ ( $16\cdot8192 = 131{,}072$ each), k and v are $4096\to1024$ because of 8 key-value heads ( $16\cdot5120 = 81{,}920$ each), and gate, up, down are $4096\leftrightarrow14336$ ( $16\cdot18432 = 294{,}912$ each). That is $1{,}310{,}720$ per block, times 32 blocks, $41{,}943{,}040$ , about 0.5% of the model.

Does LoRA make training faster? By how much, and why not more?

Somewhat. Per token, full fine-tuning costs about $6N$ FLOPs, $2N$ forward, $2N$ for input gradients, and $2N$ for weight gradients. LoRA keeps the forward pass and the input gradients through every frozen matrix, because backpropagation must reach earlier adapters, and drops only the frozen weight gradients, so it costs about $4N$ plus a small adapter term, roughly a third less (Hu et al. measured 25% on GPT-3 175B). The big savings are memory, no gradients, master weights, or Adam states for the base, which is what lets larger batches or smaller GPUs be used.

What is the intrinsic dimension of fine-tuning, and how does it motivate LoRA?

Train only $\mathbf{z} \in \R^k$ in $\theta = \theta_0 + \mathbf{P}\mathbf{z}$ for a fixed random $\mathbf{P}$ , and find the smallest $k$ that reaches 90% of full fine-tuning. Aghajanyan et al. (2021) found it small, about 200 for RoBERTa-Large on MRPC, and smaller for larger models. If task-relevant changes live in a low-dimensional subspace, the update to each matrix can be low rank, which LoRA imposes directly and learns the subspace rather than fixing it at random.

Is a full fine-tuning update actually low rank? What does that imply for LoRA?

Not exactly. Its singular values fall much faster than noise's, so a modest rank captures most of its norm, but there is a long tail, and Biderman et al. (2024) found full fine-tuning updates on code and math with ranks 10 to 100 times higher than typical LoRA settings. LoRA finds its own low-rank solution rather than approximating the full update. It matches full fine-tuning when the data's information content fits in the adapter, as for most instruction tuning, and falls short on capacity-hungry training such as continued pretraining on a new domain.

🧭 What to remember

LoRA freezes $\mathbf{W}_0$ and trains $\Delta\mathbf{W} = \frac{\alpha}{r}\mathbf{B}\mathbf{A}$ , with $r(d_{\text{in}}+d_{\text{out}})$ parameters per matrix, motivated by the small intrinsic dimension of fine-tuning.
$\mathbf{B} = 0$ starts training at the base model, and only $\mathbf{B}$ moves on the first step, which projects the full gradient onto the random row space of $\mathbf{A}$ . Both zero is a saddle.
With Adam, $\alpha/r$ keeps early progress independent of rank. With SGD, $\alpha/\sqrt r$ (rsLoRA) does. Fix $\alpha$ , tune the learning rate, and expect about 10 times full fine-tuning's.
Adapt every linear layer, especially the MLP, and train new token rows if you added tokens.
Real fine-tuning updates are compressible but have long spectral tails, so LoRA matches full fine-tuning when the task needs little capacity.
Adapters merge into the base for zero latency, or stay separate to share one base across many variants. Next, §14.6 quantizes the frozen base to 4 bits and surveys the rest of the PEFT family.

Hu et al. (2021). LoRA: Low-Rank Adaptation of Large Language Models. The method, the GPT-3 results, and the analysis of which matrices to adapt and how the update relates to the pretrained weight.
Aghajanyan et al. (2021). Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning. Random-subspace fine-tuning and why bigger, longer-pretrained models need fewer dimensions.
Kalajdzievski (2023). A Rank Stabilization Scaling Factor for Fine-Tuning with LoRA. The case for alpha over the square root of r, with a proof.
Biderman et al. (2024). LoRA Learns Less and Forgets Less. LoRA versus full fine-tuning on code and math, and the rank of full fine-tuning updates.
Schulman et al. (2025). LoRA Without Regret. When LoRA matches full fine-tuning, the importance of MLP layers, and learning rates across ranks.
Hayou et al. (2024). LoRA+: Efficient Low Rank Adaptation of Large Models. Different learning rates for A and B.
Li et al. (2018). Measuring the Intrinsic Dimension of Objective Landscapes. The original random-subspace method.

Zero to Frontier is made for larger screens

LoRA

Why a small update can be enough

The low-rank update

Initialization, and why B starts at zero

LoRA forward pass, merge, and unmerge

LoRA backward pass, checked numerically

The scale alpha over r, and rsLoRA

Which layers to adapt

Count LoRA's trainable parameters for a LLaMA config

Merging, swapping, and serving adapters

LoRA from scratch

The real way

Read more

Watch next