A 65B-parameter model in bfloat16 is 130 GB of weights, more than any single GPU holds. Full fine-tuning it with Adam needs about a terabyte of model states. In May 2023, Dettmers et al. fine-tuned 65B models on one 48 GB GPU in about a day, and their best model, Guanaco 65B, reached 99.3% of ChatGPT's score on the Vicuna benchmark as judged by GPT-4. The method, QLoRA, combined LoRA (§14.5) with three memory tricks, a 4-bit data type designed for normally distributed weights, a second round of quantization for the quantization constants themselves, and an optimizer whose states can spill to CPU memory.

This section derives each of those tricks, then steps back to survey the rest of the parameter-efficient fine-tuning (PEFT) family, adapters, prefix and prompt tuning, IA3, DoRA, and the LoRA variants, with what each costs and when to choose it.

Block-wise quantization in one page

Quantization stores each number with fewer bits by mapping it to one of a few levels. The workhorse is absmax scaling. Split a weight tensor into blocks of $b$ consecutive values. For each block store one constant, its largest magnitude $c = \max_i |w_i|$, divide the block by it so every value lands in $[-1, 1]$, and replace each normalized value by the index of the nearest level $q_j$ of a fixed codebook of $2^k$ levels. To use a weight, look up its level and multiply back,

In plain words, every weight becomes a $k$-bit index plus a share of its block's constant. With $k = 4$ that is 16 levels. Small blocks track local scale better, since one large weight forces its whole block onto a coarse grid, but the constants cost more per weight. Choosing levels, block sizes, and which weights to keep in higher precision is a deep subject in its own right, central to quantization for fast inference. For fine-tuning we need only one codebook, chosen for a specific distribution.

NF4, derived from normal quantiles

Pretrained weights are close to normally distributed with mean zero. The QLoRA authors tested the weights of every hidden unit of LLaMA 7B and found only 7.5% failing a normality test at the 5% level, about 2.5 points above the rate expected by chance. A uniform grid of levels, as in ordinary int4, spends as many levels on the sparse tails as on the crowded center, so the center is quantized coarsely and many tail levels are almost never used. Better is to place levels so that each is used equally often. If $X \sim \mathcal{N}(0, 1)$ has CDF $\Phi$, the quantile function $\Phi^{-1}$ maps evenly spaced probabilities to values with equal probability mass between them. QLoRA's NormalFloat type takes the midpoints of adjacent quantiles,

and rescales them into $[-1, 1]$ to match absmax normalization. In plain words, the levels are dense near zero, where most weights are, and sparse in the tails. A code that is used with equal probability carries the full $k$ bits of information per weight, which is the sense in which the paper calls NF4 information-theoretically optimal for normal data.

There is a catch. With 16 levels symmetric around zero, no level is exactly 0, and zero matters (padding, pruned weights, exact zeros in a matrix should stay zero). So NF4 is built asymmetrically. It takes 8 quantile levels for the positive half and 7 for the negative half, adds an exact 0, and normalizes so the largest magnitude is 1. The reference implementation in bitsandbytes evaluates the quantiles from a probability of $p = 0.9677$ down to $0.5$ (the code does not explain the constant, but it equals the average of $1 - \frac{1}{30}$ and $1 - \frac{1}{32}$, the centers of the outermost bins when probability is cut into 15 and into 16 equal bins), which avoids the infinite $\Phi^{-1}(1)$. Let's build it and compare with the 16 values published in the paper's appendix.

Our construction reproduces the published table to float32 precision. Notice the asymmetry. The positive side has one more level, so its steps are slightly finer.

How much better is it than a uniform grid? Quantize a large matrix of Gaussian weights in blocks of 64 with three codebooks, the symmetric int4 grid $\{-7, \dots, 7\}/7$, a 16-level grid $\{-8, \dots, 7\}/8$, and NF4, and also compute the codebook that minimizes squared error (Lloyd's algorithm, k-means in one dimension).

NF4 cuts the RMS error by about 15% relative to symmetric int4 and uses its codes far more evenly (3.88 of 4 possible bits of entropy, against 3.50). The MSE-optimal codebook is only about 5% better still. Equal-mass levels are not exactly the levels that minimize squared error, because Lloyd's optimum also weighs how far values fall from their level, but NF4 comes close, and it has two practical advantages, a fixed table known in advance and an exact zero. In QLoRA's experiments, NF4 gave better perplexity and zero-shot accuracy than 4-bit floats or integers at the same size.

import numpy as np

def nf4_quantize(w, block=64):
    raise NotImplementedError

def nf4_dequantize(codes, absmax, shape):
    raise NotImplementedError

Double quantization

Each block of 64 weights carries one float32 constant, 32 bits shared by 64 weights, which adds $32/64 = 0.5$ bits per parameter, an eighth of the 4-bit budget. QLoRA's double quantization quantizes those constants too. Treat the float32 absmax values $c_1$ of the first quantization as data, subtract their mean (they are all positive, and centering lets a symmetric code use its full range), and quantize them to 8-bit floats in blocks of 256, each with its own float32 second-level constant $c_2$. The cost per parameter becomes

down from 0.5, a saving of 0.373 bits per parameter, about 3 GB for a 65B model. The total storage of an NF4 weight with double quantization is $4.127$ bits, or about 0.516 bytes.

import numpy as np

def double_quantize(c1, block=256):
    raise NotImplementedError

def double_dequantize(q, c2, mean):
    raise NotImplementedError

Paged optimizers, and QLoRA end to end

The third trick handles memory spikes. With gradient checkpointing (§10.1), a long sequence can briefly need far more activation memory than a typical one, and a run that fits on average crashes on its longest batch. Paged optimizers allocate the optimizer states in NVIDIA's unified memory, which the driver pages automatically between GPU and CPU RAM. When the GPU runs out, optimizer pages are evicted to the CPU, and they are paged back when the optimizer step needs them. With LoRA the optimizer states are small, so paging them costs little, and the QLoRA authors found that at batch size 16 paged optimizers ran as fast as regular ones on 65B models.

Put the pieces together. The base weights are stored as NF4 with double-quantized constants and never receive gradients. In every forward pass each frozen matrix is dequantized to bfloat16 on the fly, block by block, and multiplied in 16-bit precision, and a LoRA adapter in bfloat16 is added to its output,

In plain words, storage is 4-bit, compute is 16-bit, and learning happens only in the adapters. Gradients flow backward through the dequantized weights to reach earlier layers' adapters, but the 4-bit weights themselves are constants. For 65B, the base costs $65\times10^9 \times 4.127/8 \approx 33.5$ GB, the adapters and their Adam states a few GB, and activations with checkpointing the rest of a 48 GB GPU. The QLoRA paper also found, as §14.5 discussed, that adapting every linear layer was essential to match 16-bit full fine-tuning, and it used $r = 64$, $\alpha = 16$, and learning rates of $2\times10^{-4}$ for 7B and 13B models and $10^{-4}$ for 33B and 65B.

def finetune_state_bytes(n_params, n_trainable, method, double_quant=True):
    raise NotImplementedError

def fits(n_params, n_trainable, method, gpu_gb, overhead_gb):
    raise NotImplementedError

Does QLoRA lose anything? On MMLU, the paper found 4-bit QLoRA matching 16-bit LoRA and full fine-tuning. Tulu 2's authors, evaluating more broadly, found QLoRA close on classic NLP tasks but behind full fine-tuning on open-ended generation, by about 20% on AlpacaEval on average, with the overall gap shrinking from about 10% to about 3% as the model grew (Ivison et al. 2023). And Guanaco's 99.3% of ChatGPT was a GPT-4-judged score on 80 Vicuna questions, which the paper itself analyzes with care. A sensible reading is that QLoRA is excellent value when memory is the constraint, and that the quantization error of the frozen base is a real but modest cost. LoftQ (Li et al. 2023) reduces it by initializing the adapter so that $\operatorname{dequant}(\mathbf{W}^{\text{NF4}}) + \gamma\mathbf{B}\mathbf{A}$ starts close to the original $\mathbf{W}_0$ rather than to its quantized copy.

The rest of the PEFT family

LoRA and QLoRA dominate practice, but they are two points in a larger design space. Every PEFT method answers the same question differently. Which small set of new or existing parameters, inserted where, can steer a frozen model?

Adapters (Houlsby et al. 2019) insert a small bottleneck MLP after the attention and feed-forward sublayers of every block, $\mathbf{h} \leftarrow \mathbf{h} + f(\mathbf{h}\mathbf{W}_{\text{down}})\mathbf{W}_{\text{up}}$, with $\mathbf{W}_{\text{down}} \in \R^{d\times m}$, $\mathbf{W}_{\text{up}} \in \R^{m\times d}$, and the up-projection initialized near zero so the block starts as the identity. On GLUE they came within 0.4% of full fine-tuning of BERT while adding 3.6% parameters per task. Their cost is latency. They are new sequential layers that cannot be merged into existing weights, so every token pays for them at inference.

Prefix tuning (Li and Liang 2021) prepends $n_p$ learned key and value vectors to the attention of every layer, so every real token can attend to a learned "virtual context". It trains about 0.1% of the parameters, matched fine-tuning on table-to-text and summarization with full data, and beat it in low-data settings. Optimizing the prefixes directly was unstable, so during training they are produced by a small MLP from a smaller matrix, and only the resulting prefixes are kept.

Prompt tuning (Lester et al. 2021) is the minimal version. Learn $n_p$ embedding vectors $\mathbf{E}_{\text{soft}} \in \R^{n_p\times d}$ prepended to the input only, and nothing else. For a 7B model with 20 virtual tokens that is 81,920 parameters. Its striking finding is about scale. Prompt tuning lags full fine-tuning badly on small T5 models, but the gap closes as models grow, and at about 11B parameters (T5-XXL) it matched full model tuning on SuperGLUE. Both prefix and prompt tuning spend context length on the virtual tokens and change nothing about the weights, so many tasks can share one batch, each row with its own prompt.

IA3 (Liu et al. 2022) learns three vectors per block that rescale activations elementwise, $\mathbf{l}_k$ on the attention keys, $\mathbf{l}_v$ on the values, and $\mathbf{l}_{\text{ff}}$ on the feed-forward hidden layer, all initialized to ones. That is about 0.01% of the parameters. Because scaling the outputs of a linear map is the same as scaling the rows of its weight matrix, IA3 can be merged, $\mathbf{W}_K' = \operatorname{diag}(\mathbf{l}_k)\mathbf{W}_K$ in PyTorch's layout. Its paper argued that few-shot PEFT can beat in-context learning on accuracy and cost, and its T-Few recipe was the first to exceed human performance on the RAFT benchmark.

BitFit (Ben Zaken et al. 2022) trains only the bias terms. It works surprisingly well for BERT-style models, and does not apply to LLaMA-style models, which have no biases.

DoRA (Liu et al. 2024) starts from an observation about how weights change. Write each row of a weight matrix (one output unit's weights, in PyTorch's layout) as a magnitude times a unit direction. Analyzing these two parts over training, the authors found that full fine-tuning's magnitude and direction changes were negatively correlated (correlation $-0.62$), while LoRA's were positively correlated ($0.83$), as if LoRA could not adjust one without the other. DoRA decouples them,

where the norm is taken row by row and $\mathbf{m} \in \R^{d_{\text{out}}}$ is a trainable magnitude vector initialized to the row norms of $\mathbf{W}_0$, so that $\mathbf{W}' = \mathbf{W}_0$ at the start. LoRA updates the direction, and $\mathbf{m}$ the magnitude. (The paper writes the matrix transposed and calls these columns. Its official code and Hugging Face PEFT both normalize each output unit's weights.) DoRA's correlation came out at $-0.31$, closer to full fine-tuning, and it improved commonsense reasoning accuracy over LoRA by 3.7 points on LLaMA-7B and 1.0 on LLaMA-13B. It merges like LoRA, at the cost of computing the norms during training, which the paper reduces by treating the norm as a constant in the backward pass.

LoRA variants change one design choice each. rsLoRA changes the scale (§14.5). LoRA+ gives $\mathbf{B}$ a larger learning rate. AdaLoRA (Zhang et al. 2023) parameterizes the update in an SVD-like form and prunes singular values to spend rank where it helps. VeRA (Kopiczko et al. 2023) shares one pair of frozen random matrices across all layers and trains only small scaling vectors, for far fewer parameters. PiSSA (Meng et al. 2024) initializes $\mathbf{A}$ and $\mathbf{B}$ from the top singular vectors of $\mathbf{W}_0$ and freezes the residual, so training starts on the principal directions. LoftQ, above, initializes for a quantized base.

import numpy as np

def prompt_tuning_inputs(E, ids, attn, labels, P):
    raise NotImplementedError

def prompt_grad(G, n_p):
    raise NotImplementedError

Choosing a method

Here are the trainable parameter counts of each method on Llama 2 7B, computed from its configuration (32 blocks, width 4,096, MLP width 11,008, 6.74B parameters in total).

A practical decision procedure follows from the table and from the evidence in this chapter.

1. If you can afford it and need the most capability, especially for large datasets that teach new skills, use full fine-tuning (§14.4).
2. Otherwise use LoRA on all linear layers, with a learning rate about ten times full fine-tuning's. Try DoRA if you are at low rank and want a few more points.
3. If the model does not fit in 16-bit, use QLoRA, and evaluate open-ended generation carefully.
4. Use prompt or prefix tuning when you need very many tasks served from one batch on a very large model, and IA3 when the budget is truly tiny.
5. Whatever you choose, re-run the base-capability and safety evaluations of §14.7 afterward.

import numpy as np

def ia3_merge(W, l):
    raise NotImplementedError

def dora_init(W0):
    raise NotImplementedError

The real way

Here is QLoRA's forward and backward path in PyTorch, with NF4 storage built from the exact table, double-quantization left out for brevity, and a LoRA adapter trained on top. The frozen weight lives as 4-bit codes and float32 block constants and is dequantized to bfloat16 for each matrix multiply.

The real stack uses bitsandbytes kernels that keep two codes per byte, double-quantize the constants, and fuse dequantization into the matrix multiply.

Read more

• Dettmers et al. (2023). QLoRA: Efficient Finetuning of Quantized LLMs. NF4, double quantization, paged optimizers, and Guanaco, with an appendix of hyperparameters.
• Houlsby et al. (2019). Parameter-Efficient Transfer Learning for NLP. Bottleneck adapters, the start of parameter-efficient fine-tuning.
• Li and Liang (2021). Prefix-Tuning: Optimizing Continuous Prompts for Generation. Learned key and value prefixes in every layer.
• Lester et al. (2021). The Power of Scale for Parameter-Efficient Prompt Tuning. Soft prompts, and how the gap to fine-tuning closes with scale.
• Liu et al. (2022). Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning. IA3 and the T-Few recipe.
• Liu et al. (2024). DoRA: Weight-Decomposed Low-Rank Adaptation. The magnitude and direction analysis, and the method.
• Ivison et al. (2023). Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2. A broad comparison of QLoRA and full fine-tuning.

Watch next

▶

▶