Quantization stores the numbers of a neural network in fewer bits. A weight trained as a 16-bit floating-point number is stored instead as an 8-bit or 4-bit integer. A group of such integers shares one scale factor that turns them back into real numbers. Done well, the model computes almost the same outputs as before. Done carelessly, it falls apart. This section is about the difference.

It matters because serving a language model is paid for in bytes. Recall from §20.1 that each decoding step, the forward pass that writes one new token for every sequence in the batch, reads every weight of the model from GPU memory once. At small batch sizes the time spent reading those bytes, not the arithmetic, sets the speed. A model stored in 4 bits instead of 16 reads a quarter of the bytes. It can produce tokens up to four times faster, and it fits on fewer GPUs.

Where rounding hurts is not where most people expect. We measure the damage with perplexity, the exponential of the average next-token loss, so lower is better. Take a language model with 13 billion parameters. Store its weights, and also its activations (the inputs to every matrix product), as 8-bit integers with one scale factor for each whole matrix. Its perplexity on C4, a held-out set of web text, rises from 12.45 to 19.08 (Dettmers et al. 2022, Table 1). That is worse than an unquantized model ten times smaller, which scores 15.91. Now give every token and every output column its own scale. Also take out the handful of activation columns that hold huge values, and multiply those in 16 bits instead of 8. Perplexity is back to 12.45. The weights were never the problem. A few activation columns were.

The reason is easiest to see on a ruler. A quantizer replaces each number by the nearest of a few evenly spaced marks, and the spacing is set by the largest number the marks must reach. One huge value stretches the spacing, so all the ordinary values crowd onto the same few marks. Large language models grow a few such huge values in their activations, in the same few coordinates at most tokens. Beyond about 6.7 billion parameters these values appear in every layer, and every 8-bit scheme that does not treat them specially loses accuracy.

This section builds quantizers from scratch and works out how much error they add and where it goes. It shows how a matrix product runs on integers and proves which scale factors can be pulled out of it. That proof explains why the large activation values are so hard to handle, and it leads to the fix above. The section then separates quantizing only the weights from quantizing weights and activations, by what each one speeds up, and it ends with training a model through its quantizer. §20.3 builds the methods that real LLMs ship with on these foundations.

A note on symbols. $b$ is a bit width, $\Delta$ the step between neighboring marks, and $g$ a group size. As in §11.2, $\pi$ is a GPU's peak arithmetic rate in FLOPs per second and $\beta$ its memory bandwidth in bytes per second. Weights use the book's (input, output) layout, $\mathbf{W} \in \R^{d_{\text{in}}\times d_{\text{out}}}$ and $\mathbf{Y} = \mathbf{X}\mathbf{W}$, with one token per row of $\mathbf{X}$. So an output channel, the weights that produce one output feature, is a column of $\mathbf{W}$. An input channel is a row of $\mathbf{W}$, and it multiplies one column of $\mathbf{X}$.

One large value sets the step for everyone

Here is the whole problem in four numbers. We store each one in 4 bits, as a signed integer from $-7$ to $7$. We choose the step between marks so that the largest magnitude lands exactly on the top mark, and every other number goes to its nearest mark.

One large value set the step for everyone. The rest of this section is about keeping that from happening, or paying for it as little as possible. There are two levers, how many numbers share one step and what is done with the rare large values.

A uniform quantizer divides, rounds, and clamps

A uniform quantizer turns a real number into a small integer, its code, and back again, using a fixed step. To store a number, divide it by the step, round to the nearest integer, and clamp the result into the range the bits allow. To read it back, multiply the code by the step.

There are two standard versions. The symmetric one centers its marks on zero. The asymmetric one shifts its marks by an integer called the zero point, so that it can cover a lopsided range. The outputs of a ReLU, the activation function $\max(0, x)$, are the usual example, since they are never negative.

In plain words, divide by the step, round, keep the result inside the integer range, and multiply back. The symmetric version stores one extra number for each group of values, the step. The asymmetric version stores two, the step and the zero point.

Three details are easy to get wrong, and each has a reason.

The symmetric range leaves out $-2^{b-1}$. An int8 can hold $-128$, but the symmetric grid uses only $[-127, 127]$. That way $x$ and $-x$ always get codes of the same size, so the grid really is symmetric about zero.

The zero point is an integer, and the range contains 0. With an integer $z$, the real value 0 gets the code $\operatorname{round}(0) + z = z$, and it comes back as $Q(0) = \Delta(z - z) = 0$ exactly. Zeros are everywhere in a network, in padding, masked positions, ReLU outputs, and pruned weights. A quantizer that turned every zero into $\pm 0.003$ would add the same small bias to every sum. If $z$ were not an integer, no code would map to 0. If the range did not contain 0 (say every value lies in $[2, 5]$), the zero point would be negative and fall outside the codes.

The clamp is needed even without clipping. In the asymmetric version both $x_{\max}/\Delta$ and $z$ are rounded. When both land exactly halfway between two integers and both round up, the code of $x_{\max}$ is $2^b$, one past the end. With $x = (-1.5, 1.5)$ and $b = 2$, the step is $\Delta = 1$ and $z = \operatorname{round}(1.5) = 2$. The code of 1.5 would be $\operatorname{round}(1.5) + 2 = 4$, outside $[0, 3]$.

Which version goes where? A symmetric quantizer wastes half its codes on a ReLU output, because no value is negative. Take $(0, 0.5, 1.0, 2.55)$ in 8 bits. The asymmetric quantizer uses the step $2.55/255 = 0.01$ and the codes $(0, 50, 100, 255)$. The symmetric one needs the step $2.55/127 = 0.020$, twice as coarse. But a zero point costs extra work inside a matrix product, as we derive below. So practice settles on symmetric weights and, where the hardware allows, asymmetric activations. The int8 embeddings of §7.3 use a close cousin of the asymmetric quantizer, with one range per dimension and the range's minimum itself as the shift instead of an integer zero point.

Not every grid is uniform. Take the NF4 format of §14.6, which stores frozen weights while small adapters are fine-tuned on top. It keeps one absmax scale per block, but it places its 16 levels at quantiles of a normal distribution. That is a non-uniform grid. The floating-point formats of §8.5 are non-uniform too, with marks that grow sparser as numbers grow. This section is about uniform grids, the ones that integer matrix units multiply directly.

The cell writes both quantizers and replays the hand example at 4 and 8 bits.

With 8 bits, the outlier still makes the total error of the three small numbers about eight times larger (0.0169 against 0.0020). But each error here is about 18 times smaller than at 4 bits, because the step shrinks by $127/7 = 18.1$. That is why one scale for a whole tensor is often fine at 8 bits and rarely at 4. An interviewer who asks why int8 tolerates coarse scales and int4 does not wants this ratio.

import numpy as np

def quantize(x, b, mode="symmetric"):
    raise NotImplementedError

def dequantize(x_q, delta, z=0):
    raise NotImplementedError

How much error rounding adds

Inside the clipping range, the quantizer moves each number to its nearest mark. So the error $e = Q(x) - x$ is never larger than half a step, $|e| \le \Delta/2$. That is the worst case. For the typical error we need a model of where $x$ falls between two marks. The simplest model says anywhere, with equal chance.

This is a model, not a law. It fails when the values span only a few steps. In the 2-bit Gaussian of the next cell, most values round to 0, and the error of each is $-x$ itself. The same $\Delta^2/12$ sets the noise floor of digital audio in §35.1.

To compare quantizers we measure the error against the size of the signal. The signal-to-quantization-noise ratio (SQNR) says how many times larger the signal's power is than the error's. It is usually quoted in decibels (dB), ten times its base-10 logarithm. So 10 dB means ten times, 20 dB means a hundred times, and every 3 dB is about a doubling.

In plain words, each extra bit roughly doubles the number of marks. So it halves the step, divides the noise power by about 4, and adds about $20\log_{10}2 = 6.02$ dB. When an interviewer asks what one more bit buys, this is the answer.

The cell checks both results by simulation. A full-scale uniform signal matches $\Delta^2/12$ to within 0.3% at every width. A Gaussian clipped at four standard deviations matches it from 3 bits up. At 2 bits the Gaussian's step is 4 standard deviations, most values round to 0, and the model is off by a third.

Large activations amplify weight error

A network never uses one weight by itself. It computes dot products, $y = \sum_{k=1}^{d_{\text{in}}} x_k w_k$, so what matters is the error in $y$. Quantize the weights with step $\Delta_w$, keep the input $\mathbf{x}$ exact, and ask how far the quantized output $\hat y$ lands from $y$.

Proof. The output error is $\hat y - y = \sum_k x_k e_k$. Its mean is 0 because each $e_k$ has mean 0. The terms are uncorrelated, so the variance of the sum is the sum of the variances, $x_k^2\Delta_w^2/12$ each. ∎

In plain words, each weight's rounding error is multiplied by the activation it meets. A channel with a large activation therefore amplifies the error of every weight in its row of $\mathbf{W}$. So the weights that matter most are not the largest weights. They are the ones that meet large activations. §20.3 builds a method on exactly this, which scales those rows up before rounding so that their relative error shrinks (activation-aware weight quantization, Lin et al. 2023).

If the activations are quantized too, with step $\Delta_x$ and independent errors $e'_k$, the output error gains the terms $\sum_k w_ke'_k$ and $\sum_k e_ke'_k$. Their variances add, which gives $\operatorname{Var}(\hat y - y) \approx (\Delta_w^2/12)\lVert\mathbf{x}\rVert^2 + (\Delta_x^2/12)\lVert\mathbf{w}\rVert^2 + d_{\text{in}}\Delta_x^2\Delta_w^2/144$.

The cell builds an activation vector of length 4,096 in which four channels are large. It quantizes a thousand random weight columns to 4 bits and compares the measured error variance with the formula.

Four channels out of 4,096 carry 63% of the output error, because they carry 63% of $\lVert\mathbf{x}\rVert^2$. An interviewer who asks why a method for quantizing weights needs to look at activations wants this formula.

Clipping trades rounding error for tail error

The absmax rule puts the top mark at the largest value, so nothing is ever clipped. But the step grows with the threshold, and a heavy-tailed distribution spends most of its range on a few rare values. Lowering the threshold to $c < \max|x|$ shrinks the step for everyone. Only the values beyond $c$ pay, because they are all mapped to $\pm c$.

The error splits into two parts. Values inside the range suffer rounding error, which grows with $c$ because the step does. Values outside suffer clipping error, which shrinks as $c$ grows.

The two terms pull in opposite directions, and that has a sharp consequence.

Proof. At $c = c_{\max}$ there is no mass beyond $c$, so the clipping term and its derivative, $-4\int_c^\infty (x - c)p(x)\,\mathrm{d}x$, are both 0. The derivative of the rounding term is $\tfrac{c}{6q_{\max}^2}\mathbb{P}(|x| \le c) + \tfrac{c^2}{6q_{\max}^2}p(c)$. At $c = c_{\max}$ its first part is $c_{\max}/(6q_{\max}^2) > 0$, and its second part is never negative. ∎

So the question is never whether to clip, only how much. The cell minimizes the closed form for a Gaussian and a Laplace input at several bit widths. It checks each answer against a simulation of the real quantizer.

The gain per bit also falls short of 6 dB for a Gaussian. From 4 to 8 bits the optimal SQNR rises from 18.9 to 40.5 dB, about 5.4 dB per bit. The best threshold creeps outward as the step shrinks, because the Gaussian's tail never ends. So each bit buys a little less than a halving of the step. The plot shows the trade for each bit width, with the minimum marked.

In practice the threshold is chosen per tensor or per channel on calibration data, a few hundred typical inputs run through the model to record the values each layer sees. Three rules are common. Minimize the MSE, as here. Take a high percentile of $|x|$, such as the 99.99th. Or minimize the KL divergence between the histograms of $x$ and $Q(x)$. Wu et al. (2020) compared the percentile and KL rules with absmax on many networks and found that no single rule was best for all of them. Nagel et al. (2021) recommend the MSE criterion as the default for weights and for most activations.

import numpy as np
from scipy.stats import norm

def expected_mse_gaussian(c, b):
    raise NotImplementedError

def best_clip(x, b, n_grid=200):
    raise NotImplementedError

Granularity decides how many numbers share a scale

The hand example showed the cure for an outlier, which is to give it its own scale. So the question is how many numbers should share each scale, and what the extra scales cost. A weight matrix offers natural groupings, and so does a matrix of activations.

In plain words, smaller groups confine each outlier's damage to fewer neighbors and follow the local size of the weights more closely. They cost a few hundredths to a few tenths of a bit per weight. Per-channel scales for weights cost almost nothing, $16/d_{\text{in}}$ bits per weight, about 0.004 bits for $d_{\text{in}} = 4{,}096$.

The cell builds a synthetic layer whose output columns differ in size (a lognormal spread) and whose entries have heavier tails than a Gaussian. It quantizes the layer at four granularities and reports the SQNR of each.

The widget quantizes 2,048 weights in 16 channels and can plant one outlier in channel 5. It draws every weight over its group's levels, with the step of every group in a strip below.

import numpy as np

def quantize_weight(W, b, granularity, g=None):
    raise NotImplementedError

A matrix product can run on integers

Storing numbers in 8 bits saves memory. Multiplying them in 8 bits saves time as well. A GPU's tensor cores, its dedicated matrix-multiply units, run faster at lower precision. On an H100 SXM they do 1,979 trillion dense int8 (or FP8) operations per second, twice the 989 trillion of BF16, the 16-bit float format models are usually served in (§11.1). Spec sheets also quote twice these numbers, but only for sparse matrices. To use the int8 rate, the whole matrix product must run on integer codes. Kernel writers call a matrix product a GEMM (general matrix multiply). Every step and zero point has to be applied outside its inner loop, the sum of products that the tensor cores run. A tiny example shows how.

The same steps work for whole matrices. Quantize the activations $\mathbf{X} \in \R^{n\times K}$ asymmetrically, with one step $\Delta_{x,i}$ and zero point $z_{x,i}$ per token. Quantize the weights $\mathbf{W} \in \R^{K\times m}$ symmetrically, with one step $\Delta_{w,j}$ per output column. Here $K = d_{\text{in}}$ is the length of each dot product, called the reduction dimension, and $m = d_{\text{out}}$.

Proof. Expand $(X_{q,ik} - z_{x,i})W_{q,kj}$ and sum over $k$. The steps $\Delta_{x,i}$ and $\Delta_{w,j}$ do not depend on $k$, so they leave the sum. ∎

In plain words, multiply the integer codes, subtract one correction per output entry built from the precomputed column sums, and only then multiply by the two steps. Everything inside the parentheses is integer arithmetic, and $\mathbf{u}$ is computed once, when the model is loaded.

Three practical points follow from the formula.

The accumulator must be wide. A uint8 code times an int8 code is at most $255\times 127 = 32{,}385$ in magnitude, just inside the int16 limit of 32,767, and a dot product adds $K$ of them. With $K = 4{,}096$ the sum can reach $1.3\times 10^8$. So tensor cores add the products in an int32 accumulator, the running total of the dot product. It is safe up to $K \approx 2^{31}/32{,}385 \approx 66{,}000$.

Zero points on both sides cost a term that depends on the input. If the weights also had zero points $z_{w,j}$, the expansion would be

The row sums $v_i$ of the activation codes depend on the input, so they must be recomputed for every batch. Nagel et al. (2021) note that this extra pass over $\mathbf{X}_q$ costs about as much as adding an extra channel to the layer. That is the reason for the convention of symmetric weights and, optionally, asymmetric activations. Interviewers ask for exactly this expansion.

The output is rescaled with one multiplier. If the next layer wants int8 inputs with step $\Delta_y$, the int32 accumulator is multiplied by $M = \Delta_x\Delta_w/\Delta_y$. It depends only on the steps. With static per-tensor steps it is one number fixed before inference. With dynamic per-token steps it is the outer product of the token and column steps divided by $\Delta_y$, formed at run time.

The cell runs the integer path on 4 tokens of length 4,096. It checks the result against the product of the dequantized matrices and against the float product.

The naive NumPy product of a uint8 and an int8 array is computed in int16, while the true sums reach 445,752, so it silently wraps around. Widened to 64 bits, the integer path equals the product of the dequantized matrices to rounding. The whole 8-bit product is within about 1% of the float one.

Only some scales can leave the sum

The derivation pulled $\Delta_{x,i}$ and $\Delta_{w,j}$ out of the sum over $k$. That worked because of where those steps live, one per token and one per output column. The rule is worth stating as a theorem, because it decides how every quantized kernel is designed. We call it the factoring theorem, and later sections use that name.

Proof. In case 1, $a_i$ and $b_j$ do not depend on the summation index, so they factor out, $\sum_k a_i\tilde X_{ik}\tilde W_{kj}b_j = a_ib_j\sum_k\tilde X_{ik}\tilde W_{kj}$. In case 2, take $K = 2$, $\tilde X_{i1} = \tilde X_{i2} = 1$, and two weight columns, $(\tilde W_{1j}, \tilde W_{2j}) = (1, 0)$ and $(0, 1)$. Both have the integer product 1, but the outputs are $c_1$ and $c_2$. If $c_1 \ne c_2$, no function of the integer products can produce both. ∎

Three consequences shape all of LLM quantization.

• Activations are quantized per token and weights per output channel, because those are exactly the scales that factor out. Per tensor is the special case of constant scales.
• Activations are never quantized per input channel inside an integer GEMM, even though, as we are about to see, that is where their outliers live. A per-input-channel scale can instead be moved into the weights, since $\mathbf{X}\mathbf{W} = (\mathbf{X}\operatorname{diag}(\mathbf{c})^{-1})(\operatorname{diag}(\mathbf{c})\mathbf{W})$. The rescaled weights are then quantized per output channel as usual. §20.3 builds a method on this identity that divides each large activation channel down and multiplies the matching weight row up (Xiao et al. 2022).
• Group-wise weight scales vary along $k$, so they break case 1. They still work in two ways. A kernel can add up each group of $g$ products separately and rescale the partial sums, as kernels for 8-bit floats (FP8) with one scale per block do. Or, when only the weights are quantized, it can turn the weights back into 16-bit numbers in registers and multiply in 16-bit arithmetic, where any scale is allowed.

import numpy as np

def int_gemm(Xq, zx, dx, Wq, dw):
    raise NotImplementedError

Outliers are why LLM activations resist

Back to the opening puzzle. The weights of trained networks are roughly bell-shaped, and per-channel scales absorb the differences in size between their columns. So 8-bit weights have been standard practice since the integer-only inference work of Jacob et al. (2018). Wu et al. (2020) brought every network they studied, BERT included, within 1% of its float accuracy in int8, though a few needed some layers left in float or a short fine-tune. The activations of large language models are different.

Dettmers et al. (2022) measured how. They speak of feature dimensions, the coordinates of the hidden vector that enters a matrix product. A feature dimension is one column of $\mathbf{X}$, and it meets one input channel of $\mathbf{W}$.

Now the failure follows from the hand example. Most tokens carry a value near $-60$ in a few columns. A per-tensor step is $60/127 \approx 0.47$, and so is the per-token step of every token that carries one, about three quarters of them. The ordinary values, of size about 1, are rounded with a step almost half their own size.

The factoring theorem rules out the easy fixes. Per-token scales factor out of the GEMM, but they cannot help, because the outliers live in columns and these scales vary by row. A per-column scale for $\mathbf{X}$, one per input channel, would fix the problem, but it varies along the sum, so an integer GEMM cannot apply it. That is the trap. An interviewer who asks why 8-bit activations are harder than 8-bit weights is asking for this argument.

The widget shows the trap on a small activation matrix, 32 tokens by 64 channels. Two of the channels carry outliers near $-40$, in a share of the tokens that you set. The left map is the size of each entry, and the right map is its quantization error under the scheme you pick. The LLM.int8 split, explained just below the widget, keeps the outlier columns in 16 bits.

The fix multiplies the outlier columns in 16 bits

Dettmers et al. escaped the trap by using the structure of the outliers. They live in a handful of columns of $\mathbf{X}$. So take those columns out and multiply them, with the matching rows of $\mathbf{W}$, in 16 bits. Multiply everything else in 8 bits with the scales that factor out. The paper calls this method LLM.int8().

In plain words, the columns of $\mathbf{X}$ that contain an outlier, and the matching rows of $\mathbf{W}$, are multiplied in 16 bits. With 7 outlier columns out of 5,120, that is about 0.1% of the work. The other 99.9% runs in int8 with scales that the factoring theorem allows.

The cell builds a realistic activation and measures the error of $\mathbf{X}\mathbf{W}$ under four schemes. The activation has 4,096 channels and six one-sided outlier dimensions near $-58$ (the median the paper reports for its 13B model), present in 75% of the tokens.

The decomposition is not free. The outlier columns change with the input, so they are found at run time, and the 16-bit and 8-bit products are computed separately and added. Dettmers et al. measured the cost in their appendix. They timed the first feed-forward matrix multiply at the shapes of GPT-3 models of every size. With the decomposition it ran slower than in 16 bits at every size up to 6.7B, and only the 13B and 175B shapes ran faster. End-to-end generation with BLOOM-176B took 247 ms per token at batch 1 on three A100 GPUs, against 239 ms in 16 bits on eight. LLM.int8 bought memory, not speed.

Getting 8-bit activations and speed together needed new ideas. One is to move each outlier channel's scale into the weights, dividing the channel down and multiplying the matching row of $\mathbf{W}$ up, by the identity $\mathbf{X}\mathbf{W} = (\mathbf{X}\operatorname{diag}(\mathbf{c})^{-1})(\operatorname{diag}(\mathbf{c})\mathbf{W})$. Another is to rotate the channels so that no single one is large. Both are in §20.3.

import numpy as np

def llm_int8_matmul(X, W, thr=6.0):
    raise NotImplementedError

Weight-only and weight-activation quantization speed up different things

So far we have asked how accurate a quantizer is. The other question is what it buys, and the answer depends on what limits the GPU. Recall the roofline of §11.2. A piece of work that does $F$ floating-point operations needs at least $F/\pi$ seconds of arithmetic. The bytes it moves need at least their number divided by $\beta$ seconds of memory traffic. The work takes at least the larger of the two times.

§20.1 applied this to one decode step of one model on one GPU. A dense model with $N$ parameters does about $2N$ FLOPs per token, plus attention. The step reads the weights, $M_W = s_WN$ bytes with $s_W$ bytes per weight, plus every sequence's KV cache, the stored keys and values of all its earlier tokens (§6.5). Quantization can change two of these numbers. It can shrink the bytes per weight $s_W$ (and the cache's bytes, in §20.3). And if the multiply itself runs in low precision, it raises the peak $\pi$.

The roofline then predicts where each one helps. Decode at small batch is memory-bound, so it runs as fast as the bytes allow. There, weight-only quantization speeds it up almost in proportion to the bytes saved. Prefill, the single forward pass over the whole prompt, is compute-bound, and so is decode at large batch. There, fewer weight bytes do nothing, and only a higher $\pi$ helps.

The cell evaluates the step model for Llama 3.1 8B on one H100. Like §20.1, it counts all $N = 8.03$ billion weights as read each step. The input embedding table is really read only one row per token, and it holds 0.53 billion of those weights, so a careful count reads 6.5% less.

So weight-only quantization is the tool for small-batch serving, where latency matters most, and for fitting a model in memory. Weight-activation quantization is the tool for high-throughput serving and for prefill. An interviewer will often ask why W4A16 speeds up decode but not prefill, and this table is the answer.

Real W4A16 kernels have one more subtlety. Turning codes back into 16-bit numbers costs instructions. As the batch grows the kernel becomes compute-bound, and a naive kernel loses its advantage. Kernels built to hide that work inside the matrix-multiply pipeline do much better. One of them, MARLIN, keeps close to the ideal 3.87x speedup of 4-bit weights with group 128 up to batches of about 16 to 32 (Frantar et al. 2024). §20.3 returns to them.

The table collects the common formats. E4M3 is the 8-bit float with 4 exponent bits and 3 mantissa bits (§8.5). NVFP4 and MXFP4 are 4-bit float formats in which each small block of values shares one scale.

Training through the quantizer

Everything so far quantizes a model that has already been trained. That is the cheap option, and for 8 bits it is usually enough. At 4 bits and below, the model can instead be taught to live with its quantizer, by training with the rounding in the forward pass.

In plain words, the forward pass sees the rounded weights, and the backward pass pretends the rounding was not there, except that clipped weights get no gradient. Bengio et al. (2013) named the estimator. They credited the idea to a 2012 lecture by Geoffrey Hinton, which passed gradients through a hard threshold as if it were the identity, and they noted that it is clearly biased.

The bias is easy to state. The STE gradient is the gradient of the loss at the quantized point $Q(w)$, applied to $w$ as if the quantizer were not there. The true gradient of $\mathcal{L}(Q(w))$ with respect to $w$ is 0 almost everywhere, so the STE is not an estimate of it at all. What it gives is a useful search direction. Gradient steps add up in the latent weight until it crosses a rounding boundary, and then its quantized value moves.

§0.3 wrote the STE as one line with a stop-gradient, w + (q(w) - w).detach(), and §0.9 wrote it as a custom autograd.Function. The clip mask is the one addition QAT needs.

Two cautions come with it. First, a latent weight near a rounding boundary can flip between two grid points every step. That adds noise, and in networks with batch normalization it corrupts the running statistics the normalization keeps. Nagel et al. (2022) measured these oscillations and proposed damping them or freezing the weights that oscillate. A decaying learning rate also lets the latent weights settle. Second, a weight pushed beyond $c_{\text{clip}}$ receives no gradient and stays clipped. Methods that learn the step $\Delta$ itself, such as learned step size quantization (Esser et al. 2020), apply the STE to $\Delta$ as well.

How much can QAT gain? On a small problem we can see the best possible answer exactly. The cell fits a linear model with eight correlated inputs and rounds the least-squares weights to a 2-bit grid $\{-\Delta, 0, \Delta\}$. Then it compares that with the best of all $3^8 = 6{,}561$ grid choices.

Choosing each rounding with its neighbors in mind is the idea that §20.3 develops into a whole algorithm, which rounds one input channel at a time and corrects the rest for its error (Frantar et al. 2022). QAT finds such choices by gradient descent through the quantizer, as the exercise shows.

import numpy as np

def fake_quant(w, delta, b):
    raise NotImplementedError

def ste_grad(w, X, y, delta, b):
    raise NotImplementedError

In PyTorch

PyTorch ships fake quantization as built-in operators, torch.fake_quantize_per_tensor_affine and torch.fake_quantize_per_channel_affine. Their backward pass is the masked STE above with one small difference. The mask tests the rounded code rather than $w$, so a value up to half a step beyond $c_{\text{clip}}$, which still rounds to $q_{\max}$, keeps its gradient. The block first shows the gradient mask. Then it runs QAT on a linear layer with 3-bit weights and one step per output column, and compares the result with rounding the float solution.

The two values outside $[-1.75, 1.75]$, $-2$ and 3, lie more than half a step beyond it, so they are clipped and get zero gradient. QAT lowers the error of the 3-bit layer from 0.0621 to 0.0389. For real models, QAT costs a training run (or a long fine-tune) and the training data, which PTQ does not need. That is why PTQ dominates for 8 bits and for most 4-bit weight-only deployments. QAT is used where the last fraction of quality at 4 bits or below is worth a training run, as in the natively low-precision releases that §20.3 surveys.

Where it breaks

Read more

• Jacob et al. (2018). Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference. The scale and zero-point scheme, why zero must be exact, the integer GEMM with precomputed sums, int32 biases, and fixed-point requantization.
• Nagel et al. (2021). A White Paper on Neural Network Quantization. The clearest single treatment of uniform quantizers, granularity, why asymmetric weights cost a data-dependent term, range setting, PTQ, and QAT with the STE.
• Dettmers et al. (2022). LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale. The measurement of outlier features and their emergence near 6.7B parameters, vector-wise quantization, and the mixed-precision decomposition.
• Sun et al. (2024). Massive Activations in Large Language Models. The few huge scalars in the residual stream, where they sit, and their link to attention sinks.
• Banner et al. (2019). Post-training 4-bit quantization of convolution networks for rapid-deployment. Analytical clipping (ACIQ), the rounding-plus-clipping error model for Gaussian and Laplace inputs.
• Wu et al. (2020). Integer Quantization for Deep Learning Inference: Principles and Empirical Evaluation. A careful empirical study of int8 choices (per-channel weights, calibration methods) with a workflow that keeps every network studied within 1% of float accuracy.
• Bengio et al. (2013). Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation. Where the straight-through estimator was named and its bias discussed.
• Gholami et al. (2021). A Survey of Quantization Methods for Efficient Neural Network Inference. A broad map of the field (uniform and non-uniform grids, mixed precision, QAT variants) for going beyond this section.

Watch next

▶

▶

▶