Quantizing a language model means storing its numbers in fewer bits, usually 8 or 4 instead of 16. Each step of generation then reads fewer bytes from memory, so it runs faster, and the model fits on fewer GPUs. The simplest way is to round each weight to the nearest of a few evenly spaced levels, and for large models that often fails badly. This section builds the methods that make 8-bit and 4-bit models work, the ones frontier models ship with. They share one idea. A weight matters only through the outputs it helps compute, so a good method preserves each layer's outputs on real inputs, not each weight on its own.

Here is the gap that idea closes. OPT-175B is an open language model from Meta with 175 billion parameters. Round each of its weights to the nearest level of a 4-bit grid, with one grid per output channel. This is called round to nearest, or RTN. The model's perplexity on WikiText-2, a set of Wikipedia articles, rises from 8.34 to 10.54. (Perplexity is the exponential of the average next-token loss, so lower is better.) That is worse than the 16-bit OPT-13B, a model thirteen times smaller, which scores 10.13.

Now quantize the same model to the same grid in a different way. First run 128 passages of ordinary text through the model, to see what each layer's inputs look like. Then pick each weight's level with that knowledge. This method is GPTQ (Frantar et al. 2022), and it brings the perplexity to 8.37. At 3 bits the gap is enormous, about 7,300 for rounding against 8.68 for GPTQ (Table 3 of the paper). Both models use the same bits, the same levels, and the same storage format. A GPU runs them with the same kernel, the program that does the multiply. Only the choice of level for each weight differs.

What GPTQ does differently is look at the layer's output instead of its weights. Every weight multiplies an input, and the inputs of a layer are correlated. So when one weight is rounded up, a weight that sees a similar input can be nudged down to cancel most of the damage. This section derives that idea exactly, as a formula called the optimal brain surgeon update, and turns it into GPTQ.

Three more methods then reshape a layer, without changing what it computes, so that plain rounding becomes accurate. One moves the activations' outliers into the weights. One gives finer levels to the weights that meet large inputs. One rotates the numbers so that no coordinate stands out. After them come the 8-bit and 4-bit floating-point formats, whose levels keep the same relative precision at every size, and whose 4-bit versions give every small block of numbers its own scale in hardware. Then comes quantization of the KV cache, the other large stream of bytes a decode step reads. The section ends with how to measure honestly what all of this costs in quality, and with the models that are now trained in their low-precision format from the start.

What this section builds on

§20.2 built the basic tools. Here are the ones we need, restated.

A quantizer with $b$ bits replaces each real number by one of a few evenly spaced levels. The distance between neighboring levels is the step $\Delta$. The symmetric quantizer uses the integer codes from $-q_{\max}$ to $q_{\max}$, with $q_{\max} = 2^{b-1} - 1$, which is 7 at 4 bits and 127 at 8 bits. It stores $x$ as the code $x_q = \operatorname{clamp}(\operatorname{round}(x/\Delta), -q_{\max}, q_{\max})$ and reads it back as $Q(x) = \Delta\,x_q$. The usual step is the largest magnitude divided by $q_{\max}$, so that the largest number lands on the top level. For example, the weights $(0.3, -0.9, 0.4)$ at 4 bits get the step $0.9/7 = 0.129$ and come back as $(0.257, -0.9, 0.386)$. The zero-point quantizer shifts the codes to $0, \dots, 2^b - 1$ with an integer offset $z$, as $Q(x) = \Delta(x_q - z)$, so that it can cover a lopsided range such as $[0, 5]$.

Which numbers share one step is the granularity. Weights usually get one step per output channel, or one per group of $g$ consecutive input channels within an output channel. Activations usually get one step per token. Each step is stored in 16 bits, so $b$-bit codes in groups of $g$ cost $\bar b = b + 16/g$ bits per weight. At $b = 4$ and $g = 128$ that is 4.125 bits.

Two more facts from §20.2 carry most of this section. The first is about which scales a matrix product can carry. In $\mathbf{X}\mathbf{W}$, a scale that depends only on the token or only on the output channel can be applied once, after the integer sum. A scale that depends on the input channel, the index being summed over, cannot. It would have to be applied to every product inside the sum. We call this the factoring theorem. The second fact is that LLM activations have outlier features. These are a few input channels whose values are 3 to 20 times larger than everything else. In models of about 6.7 billion parameters and more they appear in every layer and in about 75% of the tokens (Dettmers et al. 2022). Together the two facts explain why 8-bit activations are hard. The outliers live in input channels, and an input-channel scale is exactly the one an integer matrix product cannot carry.

§20.2 also separated two kinds of quantization by what they speed up. Weight-only quantization, written W4A16 for 4-bit weights and 16-bit activations, shrinks the weight bytes that each decode step reads. That helps when the step waits on memory, as decode does at small batch. Weight-activation quantization, written W8A8 for 8-bit weights and activations, also runs the multiply itself in low precision. It uses the GPU's low-precision tensor cores, its matrix-multiply units, which raises the peak FLOP rate.

Weights use the book's (input, output) layout, $\mathbf{W} \in \R^{d_{\text{in}}\times d_{\text{out}}}$ and $\mathbf{Y} = \mathbf{X}\mathbf{W}$, with one token per row of $\mathbf{X} \in \R^{n\times d_{\text{in}}}$. So an output channel is a column of $\mathbf{W}$, and an input channel is a row of $\mathbf{W}$ and a column of $\mathbf{X}$. As in §20.1, $\pi$ is a GPU's peak FLOP rate and $\beta$ its memory bandwidth.

The quantity to protect is the layer's output

Round to nearest makes each stored weight as close as possible to its original value. That sounds like the right goal, but the network never uses a weight by itself. It computes $\mathbf{X}\mathbf{W}$, and each output is a dot product. §20.2 showed that the error of a dot product is the sum of each weight's error times the input it meets. So the errors of different weights add up in the output. Whether they reinforce or cancel depends on the inputs. A tiny example shows how much that matters.

The cell checks every number. The line marked OBS computes the compensation with a formula that the next part derives.

A real layer has thousands of inputs, so we need the general version of this calculation. Post-training quantization (PTQ) quantizes a trained model without training it further. It works one linear layer at a time. It runs a small calibration set, typically about a hundred passages of ordinary text, through the model and records each layer's inputs $\mathbf{X}$. Then it looks for the quantized weights that best reproduce the layer's outputs on those inputs.

In plain words, a quantized layer is good if it gives the same outputs on real inputs. How much a weight error hurts depends on how strongly the inputs excite that weight and on how the inputs move together. Two properties of this objective carry the rest of the section. It splits into one independent problem per output column, so the columns can be quantized in parallel. And all columns share one Hessian, which depends only on the inputs, never on the weights. The two-input example is one column whose error is averaged over tokens, so its Hessian is $\mathbf{H} = 2\boldsymbol\Sigma$ with $\boldsymbol\Sigma = \mathbf{X}^\top\mathbf{X}/n$.

Finding the best grid point for a quadratic objective is a hard combinatorial problem. It is the closest vector problem on a lattice, which is NP-hard in general. So no method solves it exactly for a layer with millions of weights. What we can do cheaply is the move from the example, made exact and repeated one weight at a time. An interviewer who asks what GPTQ optimizes wants this objective, and the fact that every output column shares the same Hessian.

The optimal brain surgeon update

The example made two moves. It put one weight on the grid. Then it shifted the weights that were still free to make up for it. Here is that pair of moves in general.

Take one output column, drop its index, and write $\mathbf{w} \in \R^{d_{\text{in}}}$ for its current weights. Suppose $\mathbf{w}$ is the best point we can reach with the coordinates we are still free to change. Then a change $\boldsymbol\delta$ of those free coordinates raises the error by exactly $\tfrac12\boldsymbol\delta^\top\mathbf{H}\boldsymbol\delta$. The reason is that the error is quadratic and its slope along the free coordinates is zero at the best point. At the start this holds trivially, since $\hat{\mathbf{w}} = \mathbf{w}$ gives zero error. Now we must move coordinate $q$ onto the grid. Which change $\boldsymbol\delta$ of the free coordinates does that most cheaply?

In plain words, the cheapest way to put one weight on the grid moves every other weight along column $q$ of $\mathbf{H}^{-1}$. The move is scaled so that weight $q$ lands exactly on its level. That column, scaled so its $q$-th entry is 1, is the way of changing $w_q$ by one unit that changes the layer's outputs least.

The cost has a meaning worth knowing. It is the squared rounding gap times the squared length of the part of input $q$ that the other inputs cannot reproduce. If input $q$ is almost a copy of the others, the other weights can absorb nearly all of its error, and the cost is small. If input $q$ is unlike any other, nothing can absorb its error, and the cost is what rounding alone would pay.

Check it on the two-input example. The part of $x_1$ that $x_2$ cannot predict has mean square $1 - 0.8^2 = 0.36$ per token. So the cost is $0.04\times0.36 = 0.0144$, as the cell printed, a tenth of what rounding paid. The update is $0.2\,(1, -0.8)$, because $[\mathbf{H}^{-1}]_{11} = 1.389$ and $[\mathbf{H}^{-1}]_{21} = -1.111$. It moves $w_1$ up by 0.2 onto its level and $w_2$ down by 0.16.

When the inputs are uncorrelated, $\mathbf{H}$ is diagonal. Then the update touches only $w_q$, and the method is plain rounding. That limit says exactly when GPTQ helps and when it cannot. It is the first thing an interviewer probes after the derivation.

Repeating the step

One step is not enough, since every weight must end on the grid. So quantize one coordinate, update the others, and repeat on the coordinates still free. Two facts make the repetition exact rather than a rule of thumb.

In plain words, each step puts one more weight on the grid and leaves the rest at their best values given the ones already fixed. So the greedy sweep never has to look back, and its total cost is just the sum of what each step reports.

Each step needs the inverse of the free set's Hessian, $(\mathbf{H}_{F\setminus\{q\}})^{-1}$, and that inverse changes every step. It does not have to be recomputed from scratch. Removing one coordinate from an inverse is one step of Gaussian elimination, as the next derivation shows.

import numpy as np

def obs_quantize_one(w, H, q, grid):
    raise NotImplementedError

GPTQ, the same update made cheap

Optimal brain quantization handles each output column separately, always choosing the cheapest weight next. A column of length $d_{\text{in}}$ takes $d_{\text{in}}$ steps of $O(d_{\text{in}}^2)$ each, so the layer costs $O(d_{\text{out}}\,d_{\text{in}}^3)$. A Llama-scale feed-forward down projection has $d_{\text{in}} = 14{,}336$ and $d_{\text{out}} = 4{,}096$. That is about $10^{16}$ operations for one matrix. OBQ took about an hour for ResNet-50, an image model with 25 million parameters. Its cubic cost puts models with billions of parameters out of reach. GPTQ keeps the update and removes the cost in three steps.

Step 1, one order for every column. The greedy order differs from column to column, because it depends on the weights. But the free set's Hessian $\mathbf{H}_F$ depends only on which coordinates are free and on the inputs. Suppose every column quantizes its coordinates in the same fixed order, row 1, then row 2, and so on. Then every column has the same free set at every step. The inverse is updated once per row for the whole layer instead of once per weight. The cost falls to $O(\max(d_{\text{out}}d_{\text{in}}^2,\, d_{\text{in}}^3))$, a saving of a factor $\min(d_{\text{in}}, d_{\text{out}})$. The authors found that on large layers an arbitrary order is nearly as good as the greedy one. Their explanation is that the greedy order saves its few large-error weights for last. By then few free weights remain to absorb their errors, which cancels most of the greedy order's advantage.

Step 2, lazy batches. Quantizing row $q$ updates every later row. That is a rank-one update of a huge matrix, with a couple of arithmetic operations for each number read from memory. So the GPU spends its time waiting on memory, which the roofline model of §11.2 calls memory-bound. But the rounding of row $q$ depends only on the updates row $q$ has already received, so updates to rows far ahead can wait. GPTQ processes 128 rows at a time. Inside the block it applies each row's update to the block's own rows at once. It also stores the scaled errors of the block's rows. At the end of the block it applies them to every later row in one matrix product. The arithmetic is identical, and the GPU does one large matrix multiply per block instead of 128 rank-one updates.

Step 3, Cholesky. Repeated elimination steps accumulate rounding error. On models beyond a few billion parameters the inverse $\mathbf{H}_F^{-1}$ became indefinite in some layers, meaning it lost the positive definiteness the update relies on. That sent the updates in wrong directions and ruined those layers. The fix uses the Cholesky factorization of §0.10. Every symmetric positive definite matrix $\mathbf{A}$ factors as $\mathbf{A} = \mathbf{U}^\top\mathbf{U}$, with $\mathbf{U}$ upper triangular with a positive diagonal. (That section writes it as $\mathbf{L}\mathbf{L}^\top$ with $\mathbf{L} = \mathbf{U}^\top$.) A fast, numerically stable algorithm computes it. Step $q$ only uses row $q$ of $\mathbf{H}_F^{-1}$ from the diagonal onward. For a fixed order those rows are exactly the rows of one Cholesky factor.

So GPTQ computes one Cholesky factorization of the inverse at the start and never updates an inverse again. It also adds dampening, a small number $\lambda_{\text{damp}} = 0.01\cdot\operatorname{mean}\operatorname{diag}(\mathbf{H})$ added to the diagonal of $\mathbf{H}$. Without it, $\mathbf{X}^\top\mathbf{X}$ is singular when the calibration set is small or two features are copies of each other. Dampening keeps $\mathbf{H}$ positive definite in both cases. It is also a regularizer, because $\tfrac12\boldsymbol\delta^\top(\mathbf{H} + \lambda_{\text{damp}}\mathbf{I})\boldsymbol\delta = \lVert\mathbf{X}\boldsymbol\delta\rVert^2 + \tfrac{\lambda_{\text{damp}}}{2}\lVert\boldsymbol\delta\rVert^2$. The extra term is a small penalty on moving weights far from where training put them.

The first cell below implements GPTQ without lazy batches, which is shorter and computes the same numbers. The second compares it with rounding on a synthetic layer of 512 inputs and 256 outputs. The layer's inputs are driven by 32 shared hidden factors, so they are strongly correlated, as LLM activations are. Four of its input channels are also eight times louder than the rest. The cell checks the second part of the greedy theorem too, that the per-step costs add up to the final error.

The widget runs the same sweep on a layer small enough to see, 10 inputs and 6 outputs, on a 3-bit grid. Each step rounds one input row, and the weights not yet rounded change to absorb its error. The plot tracks the relative output error of plain rounding (RTN) and of GPTQ as the rows are rounded. Press Play to run the whole sweep, or Next row to take one step at a time. The slider sets $\rho$, the correlation between neighboring inputs. Inputs $i$ and $j$ have correlation $\rho^{\lvert i - j\rvert}$.

Two questions decide whether you can trust this method in production. How much calibration data does it need? And what happens when the inputs carry no correlation to exploit? The next cell answers both on the same simulated layer.

With 128 calibration tokens, fewer than the layer's 512 inputs, GPTQ fits the calibration set far better (error 0.024) than it does new inputs (0.058). That is overfitting. With fewer tokens than inputs, $\mathbf{X}^\top\mathbf{X}$ is singular, and dampening is what keeps the solve finite. The gap closes as the calibration set grows, to 0.0445 against 0.0457 at 8,192 tokens. Real GPTQ runs use 128 sequences of 2,048 tokens, about 262,000 tokens, far more than the width of any layer.

With uncorrelated inputs GPTQ is slightly worse than rounding on new data, 0.145 against 0.135. There is no true correlation to exploit, so the method fits the noise in its estimate of $\mathbf{H}$. GPTQ's gain is a property of the inputs, not of the weights.

import numpy as np

def gptq(W, X, b, g=None, block=128, damp=0.01):
    q_max = 2 ** (b - 1) - 1
    delta = np.abs(W).max(axis=0) / q_max
    return delta * np.clip(np.round(W / delta), -q_max, q_max)    # placeholder: plain rounding (RTN)

In practice two refinements matter. The first is groups, usually $g = 128$ for $\bar b = 4.125$ bits, which keep each column's outliers from setting the step for the whole column. GPTQ with groups of 128 is one of the most common 4-bit formats for published checkpoints, and every major serving engine runs it. The second is that a weight-only 4-bit model is fast only with kernels built for it. MARLIN (Frantar et al. 2024) is a set of GPU kernels for 4-bit weights times 16-bit activations. It overlaps memory access, unpacking, and 16-bit tensor-core work, and so keeps close to the full 4 times speedup over FP16 for batches up to 16 to 32. Inside the vLLM serving engine it gives up to 2.8 times faster generation end to end.

An interviewer who asks about GPTQ usually follows up with when it fails. The answer is in the calibration cell. It needs correlated inputs, and it needs enough calibration data that looks like production.

Moving the difficulty instead of absorbing it

GPTQ changes which levels the weights land on. A second family of methods changes the layer first, without changing what it computes, so that plain rounding becomes accurate. All of it rests on one identity. You may insert any invertible matrix and its inverse between the activations and the weights. §20.2 met its diagonal case right after the factoring theorem.

In plain words, dividing an input channel by some number and multiplying the matching weight row by the same number leaves every output unchanged. The same holds for rotating the inputs and rotating the weights back.

The identity is free only if computing $\mathbf{X}\mathbf{T}^{-1}$ costs nothing at inference. For a diagonal $\mathbf{T}$ that holds whenever the operation that produced $\mathbf{X}$ scales each channel separately. Then $1/c_k$ can be folded into that operation once, offline. Three places in a Transformer block allow this.

• After a normalization. An RMSNorm (§5.2) ends by multiplying channel $k$ by a learned gain $\gamma_k$. Replace $\gamma_k$ by $\gamma_k/c_k$. The query, key, value, gate, and up projections that read the normalized vector then absorb $c_k$ in their rows.
• Between the value and output projections. The values reach $\mathbf{W}_O$ through attention weights that mix tokens but not channels. So column $k$ of $\mathbf{W}_V$ can absorb $1/c_k$ for row $k$ of $\mathbf{W}_O$.
• Inside the feed-forward block. The down projection reads $\operatorname{silu}(\mathbf{x}\mathbf{W}_{\text{gate}})\odot(\mathbf{x}\mathbf{W}_{\text{up}})$, which is linear in the up branch. So column $k$ of $\mathbf{W}_{\text{up}}$ absorbs $1/c_k$ for row $k$ of $\mathbf{W}_{\text{down}}$.

An orthogonal $\mathbf{T}$ needs more care, and we come back to it with rotations.

SmoothQuant moves activation outliers into the weights

W8A8 needs activation scales per tensor or per token, the only ones that factor out of an integer matrix product. But LLM activations put their outliers in a few fixed input channels, and those channels are large in about three quarters of the tokens. So even a per-token scale is set by the outliers, and the ordinary values of every token land on a few levels. The weights, on the other hand, are tame. The fix is to use the diagonal identity to move part of each channel's range from the activations to the weights. This method is SmoothQuant (Xiao et al. 2022).

Here is the idea by hand. Take a layer whose ordinary input channels have activations and weights up to about 1. One outlier channel has activations that reach 50, while its weights reach only 0.1. Measure the spread of a side as its largest channel over its typical one. The activations then have a spread of 50, and the weights a spread of 1. Divide the outlier channel by $c = \sqrt{50/0.1} = 22.4$, and multiply its weight row by the same number, so the product is unchanged. Its activations now reach $50/22.4 = 2.24$, and its weights $0.1\times22.4 = 2.24$. Both sides have a spread of about 2, which an 8-bit grid handles easily.

In plain words, the product of the two ranges, $m_kv_k$, is fixed by the layer, and $\alpha$ decides how to share it. At $\alpha = 1$ the activations become perfectly flat and the weights inherit the outliers. At $\alpha = 0$ it is the reverse. At $\alpha = \tfrac12$ both sides of channel $k$ carry the same range. Activation channels are far more uneven than weight rows, so sharing the unevenness helps both.

The cell below tests this with the simulated layer's weights and new activations, in which eight channels are 50 times larger than the rest. It sweeps $\alpha$ and prints the spread of each side and the W8A8 error.

Before smoothing, the spread is about 58 in the activations and 1.5 in the weights. An 8-bit grid handles a spread of 8 easily, but a spread of 58 leaves the typical activations on a handful of levels. The spreads then trade places exactly as the theorem says. They are 62.2 on the activations and 1.0 on the weights at $\alpha = 0$, the reverse at $\alpha = 1$, and 7.9 on both at $\alpha = \tfrac12$. The W8A8 error, with one scale for the activations and one per column for the weights, falls from 0.0704 without smoothing to 0.0143 at $\alpha = \tfrac12$. That is about five times less. Either extreme is worse, because it hands all of the difficulty to one side.

The widget lets you move $\alpha$ and the size of two outlier channels. It draws each channel's largest activation and largest weight, before and after smoothing, beside the int8 output error at every $\alpha$.

In the paper, $\alpha = 0.5$ worked for every OPT and BLOOM model. GLM-130B, a 130-billion-parameter bilingual model whose activations have about 30% outliers, needed 0.75. The int8 models ran inside FasterTransformer, NVIDIA's inference library, up to 1.56 times faster than FP16 and with half the memory. That put the 530-billion-parameter MT-NLG on one node of eight A100 GPUs, where FP16 needed sixteen. In an interview, the usual question is what $\alpha$ does at its extremes, and the theorem above is the whole answer.

import numpy as np

def smoothquant(X, W, alpha):
    c = np.ones(W.shape[0])                     # placeholder: no smoothing
    X_h, W_h = X / c, c[:, None] * W
    Y = X @ W
    err = np.linalg.norm(quant8_tensor(X_h) @ quant8_columns(W_h) - Y) / np.linalg.norm(Y)
    return c, X_h, W_h, err

AWQ gives finer levels to the weights that meet large inputs

In weight-only quantization the activations stay in 16 bits, so their outliers do no harm by themselves. But they still decide which weights matter. Recall from §20.2 the error of a dot product with rounded weights. If each weight's rounding error is like independent noise with variance $\Delta_w^2/12$, where $\Delta_w$ is the weight step, the output error has variance $(\Delta_w^2/12)\lVert\mathbf{x}\rVert^2$. Each weight's error is multiplied by the activation it meets. So the weights in rows that meet large activations matter most.

Lin et al. (2023) measured this on OPT-6.7B at 3 bits with groups of 128. Rounding gives a WikiText-2 perplexity of 23.54 there, against 10.86 in FP16. Keeping just 1% of the weight rows in FP16, chosen by the average size of the activations they meet, brought it to 11.39. Keeping 1% chosen by weight magnitude gave 22.37, and a random 1% gave 24.23 (their Table 1).

Mixing precisions inside one matrix is awkward for kernels. So AWQ, activation-aware weight quantization, protects those rows by scaling instead. Scale row $k$ of $\mathbf{W}$ by $c_k \gt 1$ and the activation channel by $1/c_k$. Under the same noise model, the scaled weight's rounding error has variance $\Delta'^2/12$, where $\Delta'$ is the step of its group after scaling. That error now meets the activation $x_k/c_k$. Its contribution to the output error therefore has standard deviation $\Delta'\lvert x_k\rvert/(\sqrt{12}\,c_k)$. Before scaling it was $\Delta\lvert x_k\rvert/\sqrt{12}$. The ratio is

In plain words, scaling a row up gives its weights finer levels relative to their size, as long as the group's step does not grow with it. Suppose the scaled weights of row $k$ stay below the largest magnitude in their group. Then the group's step is unchanged, $\Delta' = \Delta$, and that channel's error falls by the factor $c_k$. The cost appears when $c_k$ grows so large that row $k$ holds the group maximum. Then $\Delta'$ grows with $c_k$, and every other weight in the group gets a coarser grid.

AWQ's Table 2 shows the trade-off on OPT-6.7B at 3 bits with $g = 128$, with the 1% salient channels (those that meet the largest activations) scaled by $s$. Perplexity goes from 23.54 at $s = 1$ to 11.92 at $s = 2$, then back up to 12.36 at $s = 4$, where 21.2% of the groups' steps had changed.

$\alpha = 0$ means no scaling, which is plain rounding. $\alpha = 1$ scales each channel in proportion to its activation size. The exponent turns a search over $d_{\text{in}}$ scales into a search over one number. And it takes only $\bar{\mathbf{a}}$, one average per channel, from the calibration data.

That is the design point. GPTQ fits a full $d_{\text{in}}\times d_{\text{in}}$ Hessian and can overfit its calibration set, while AWQ estimates only $d_{\text{in}}$ averages. The paper tested this on OPT-6.7B at 3 bits, calibrating on PubMed abstracts and evaluating on Enron emails, and the reverse. The mismatch cost AWQ 0.5 to 0.6 perplexity, against 2.3 to 4.9 for GPTQ. With 16 calibration sequences, AWQ beat GPTQ calibrated on 192.

The cell below repeats the 1% experiment on our simulated layer, at 4 bits with groups of 128, and then runs AWQ's search over $\alpha$.

On our simulated layer, keeping the five rows with the loudest inputs in full precision cuts the error from 0.118 to 0.090. Keeping five rows chosen by weight norm or at random does nothing. AWQ's scaling gets part of the way there with every weight still in 4 bits, 0.104 at its best $\alpha = 0.25$. At $\alpha = 0.75$ and above it is worse than plain rounding, because the loud rows take over their groups' steps.

AWQ's speedups come from the bytes, about 4 times fewer. Its inference engine, TinyChat, turned them into 3.2 to 3.3 times the speed of Hugging Face's FP16 implementation on desktop, laptop, and mobile GPUs. AWQ changes the layer and GPTQ chooses the levels, so the two can be applied one after the other. An interviewer comparing them wants this split, and the calibration-robustness numbers above.

import numpy as np

def awq_search(X, W, b=4, g=128, grid=21):
    c = np.ones(W.shape[0])
    Y = X @ W
    err = np.linalg.norm(X @ quant_groups(W, b, g) - Y) / np.linalg.norm(Y)
    return 0.0, c, err                           # placeholder: never scales

Rotations spread outliers over every coordinate

Smoothing flattens channels one at a time, and it needs a weight side able to take the range. At 4 bits even smoothed activations are too uneven. A per-token step of $\max/7$ leaves the typical values of a token on two or three levels. A rotation does something no diagonal scaling can. It spreads each outlier over every coordinate.

A tiny example shows the effect. The vector $(8, 0, 0, 0)$ has one spike. Multiply it by a $4\times4$ matrix whose entries are all $\pm\tfrac12$, in the pattern of a Hadamard matrix (defined below), and it becomes $(4, 4, 4, 4)$, up to signs. Its length is still 8, but now no entry stands out, and a 4-bit grid with step $4/7$ stores it exactly. §12.4 used the same trick for FP8 attention, since $(\mathbf{Q}\mathbf{R})(\mathbf{K}\mathbf{R})^\top = \mathbf{Q}\mathbf{K}^\top$ for any orthogonal $\mathbf{R}$. The idea, called incoherence processing, came from a method for quantizing weights to 2 bits, QuIP (Chee et al. 2023). QuIP multiplied by random orthogonal matrices built as Kronecker products of small ones, and its successor QuIP# switched to the random-sign Hadamard rotation used here (Tseng et al. 2024). Here we make the effect precise and then place rotations in a whole network.

Measure how spiky a vector is by its peak-to-RMS ratio, the largest magnitude over the root mean square,

which lies between 1 (all entries equal in size) and $\sqrt d$ (one nonzero entry). A uniform quantizer's step is set by the peak. What we care about is its noise compared with the typical value, so $\mu$ is the number that hurts. The spike above has $\mu = 2$, and its rotation has $\mu = 1$.

Now the rotation. Let $\mathbf{H}_d$ be the Sylvester Hadamard matrix of size $d$, a power of two. It is built by $\mathbf{H}_1 = [1]$ and $\mathbf{H}_{2d} = \begin{bmatrix}\mathbf{H}_d & \mathbf{H}_d\\ \mathbf{H}_d & -\mathbf{H}_d\end{bmatrix}$. Its entries are $\pm1$, and $\mathbf{H}_d^\top\mathbf{H}_d = d\,\mathbf{I}$. We always write its size as a subscript, to keep it apart from the Hessian $\mathbf{H}$. Let $\mathbf{R} = \mathbf{D}\,\mathbf{H}_d/\sqrt d$, where $\mathbf{D}$ is diagonal with random signs $D_{jj} \in \{\pm1\}$. Then $\mathbf{R}$ is orthogonal. Because of the recursive structure, applying it costs $O(d\log d)$ instead of the $O(d^2)$ of a general matrix.

In plain words, after a random-sign Hadamard rotation no coordinate can stand out by more than a logarithmic factor. Every output coordinate is a sign-scrambled average of all the inputs, so one spike is shared out among all of them. The same rotation applied to $\mathbf{W}$, as $\mathbf{R}^\top\mathbf{W}$, spreads the weights' outliers too. QuIP calls this making the weights incoherent, and its guarantees rest on it.

Where can $\mathbf{R}$ live in a Transformer at no cost? The natural place is the residual stream, the running sum of vectors that every block reads from and adds to (§5.1). It works because an RMSNorm commutes with rotations once its gain is moved out of the way. RMSNorm divides a vector by its root mean square, and a rotation does not change the root mean square.

In plain words, store the residual stream rotated, give every layer that reads it the inverse rotation inside its weights, and give every layer that writes it the rotation. Every layer then computes what it computed before, while the activations it sees, rotated, have no outliers.

Rotating the residual stream and folding the rotation into the weights is QuaRot (Ashkboos et al. 2024). It builds on the same invariance that SliceGPT, a pruning method by largely the same authors (Ashkboos et al. 2024), used to delete rows and columns of weight matrices. The residual rotation is folded into the weights offline and costs nothing at run time.

A few activations are not on the residual stream, and there the rotation cannot be folded away. One is the input of the down projection. Another is the input of the attention output projection. The others are the queries and keys after the rotary position encoding (RoPE), which rotates pairs of their channels by angles that depend on the token's position (§5.5). These need fast Hadamard transforms during inference, each $O(d\log d)$ per token. QuaRot counts the cost as one and a half transforms per layer, where rotating every weight matrix without folding would need two per matrix.

The cell below takes activations with six outlier channels 25 times larger than the rest. It compares W4A4 errors, 4-bit weights and 4-bit activations, with and without a random-sign Hadamard rotation.

The rotation takes the activations' peak-to-RMS ratio from 34.0 to 4.5. The theorem bounds each token's ratio. Every token here has about the same norm, so a union bound over all 1,024 tokens with $\delta = 0.01$ promises at most about $\sqrt{2\ln(2\cdot512\cdot1024/0.01)} = 6.1$ for the whole matrix. The rotation roughly halves the W4A4 output error, from 0.361 to 0.172, while $\mathbf{X}\mathbf{W}$ is unchanged in exact arithmetic. What remains is mostly the rounding noise of 15-level grids on both operands. QuaRot's default, GPTQ on the rotated weights, reduces it further. In an interview, expect to explain both halves, why a rotation removes spikes and why it costs nothing on the residual stream.

import numpy as np

def fwht(x):
    raise NotImplementedError

def incoherence(W, signs):
    raise NotImplementedError

Low-precision floats, FP8 and FP4

Integers space their levels evenly, so one large value makes every small value coarse. Floating-point numbers space their levels geometrically. They have the same number of levels between every two consecutive powers of two, a range called a binade. So a small value keeps its relative precision however large its neighbors are.

§8.5 built the 8-bit float formats. E4M3 has 1 sign bit, 4 exponent bits, and 3 mantissa bits. With 3 mantissa bits, every normal number is stored to a relative error of at most $2^{-4} = 6.25\%$. The normal range spans a factor of about 28,700, from $2^{-6}$ to 448. Below $2^{-6}$ come the subnormals, evenly spaced values that fill the gap down to zero with less relative precision. Suppose one scale per tensor maps the largest magnitude to 448. Then an outlier 50 times the typical value still leaves the typical values well inside the normal range.

The cell compares three 8-bit schemes on the activations of the SmoothQuant cell, which have eight outlier channels 50 times the rest. It reports the signal-to-quantization-noise ratio (SQNR) of §20.2. That is the signal's power over the error's power, in decibels, where every 10 dB is ten times less noise.

Overall, all three look tolerable, because the outliers carry most of the energy. The 504 ordinary channels tell the real story. INT8 with one scale per tensor leaves them at 6.9 dB, and per-token INT8 at 13.4 dB. FP8 with one scale per tensor gives them 31.5 dB, the same as everywhere else in the tensor. This is why FP8 W8A8 is the low-risk 8-bit format, and why it usually needs no smoothing. On an H100 it also doubles the dense peak, from 989 to 1,979 TFLOP/s, so it speeds up compute-bound prefill as well as memory-bound decode.

FP4, and two ways to scale it

At 4 bits a float has room for 1 sign bit, 2 exponent bits, and 1 mantissa bit, the format E2M1. Its values follow from the bit layout of §8.5 with exponent bias 1. Exponent field 0 gives the subnormals $2^{1-1}\cdot m/2 \in \{0, 0.5\}$. Exponent fields 1, 2, and 3 give $2^{e-1}(1 + m/2)$, that is $\{1, 1.5\}$, $\{2, 3\}$, and $\{4, 6\}$. E2M1 has no infinities or NaNs, so all 16 codes are numbers. They make 15 distinct values, counting $+0$ and $-0$ as one. The cell lists them.

Its nonzero magnitudes, 0.5 to 6, cover a factor of only 12, far too little for a whole tensor. So FP4 works only with a scale for every small block of values. Two such block formats have hardware support on NVIDIA's Blackwell GPUs, the generation after the H100 (B200 and its successors). §8.5 defined both for 4-bit training and showed how each scale places a block's largest value. Here we restate them next to the 8-bit formats a server chooses among, and compare them on the question that matters for serving, the error they leave in fixed weights and activations.

The two FP4 formats differ in exactly two ways, and both favor NVFP4. First, a block of 16 values has a smaller spread than a block of 32. Second, a power-of-two scale cannot map the block maximum onto the top level 6. With the floor rule above, the block maximum lands somewhere in $[4, 8)$ after scaling, and maxima above 6 are clipped. Some training recipes instead round the scale up to avoid clipping. Then the maximum can land as low as 3, and the levels 4 and 6 go unused (NVIDIA 2025, appendix B.4). NVFP4's E4M3 scale has three mantissa bits, so it maps each block maximum to within about 6% of the top level. The price is 0.25 more bits per value than MXFP4.

The cell below implements both formats, plus INT4 with one FP16 scale per 32 values for comparison, and measures their SQNR on two kinds of data. One is Gaussian. The other is heavy-tailed, a Student-t distribution with 3 degrees of freedom, which has rare values far larger than the typical ones.

At the same 4.5 bits per value, NVFP4 and INT4 with an FP16 scale per 32 values are within 0.2 dB on Gaussian data. On heavy-tailed data NVFP4 wins by 4 dB. Most values there are small next to their block's maximum, and the float grid's fine levels near zero pay off. MXFP4 is 1.5 to 1.7 dB behind both on Gaussian data, the cost of its coarse scale. These are synthetic distributions, so measure real weights and activations the same way before choosing. An interviewer comparing MXFP4 and NVFP4 wants the two differences above, block size and scale precision, and their cost in bits.

The table collects the formats of this part.

Bits per value turn directly into weight bytes. A model with $N$ parameters at $\bar b$ bits each stores $M_W = \bar bN/8$ bytes, the weight bytes of §20.1's model of a decode step. The cell computes what each format does to a 70B model and checks one published case to the decimal.

A dense 70B model makes the same point. In NVFP4 its weights are 39.7 GB, so a single 80 GB GPU has about 40 GB left. That is room for about 123,000 tokens of BF16 cache, or 246,000 in FP8, before counting activations and workspace. The cache's share of memory is the subject of the next part.

Quantizing the KV cache

The KV cache stores, for every token in every sequence being served, the keys and values that attention reads at each later step (§6.5). It costs $m_{\text{KV}} = 2Ln_{kv}d_ks_{\text{KV}}$ bytes per token. Here $L$ is the number of layers, $n_{kv}$ the number of key-value heads, $d_k$ their width, and $s_{\text{KV}}$ the bytes per number. For Llama 3.1 8B that is $2\times32\times8\times128\times2 = 131{,}072$ bytes per token in BF16. So one user with 8,192 tokens of context holds 1.07 GB.

§20.1 asked how many users one H100 can serve with this model, at 50 ms per output token and 8,192 tokens of context each. Memory ran out at 59 users, long before the latency target. Quantizing the weights from 16 to 4 bits frees only 12 GB. Quantizing the cache shrinks every user. The cell redoes that count, the users that fit in memory and the users the latency target allows, for five formats.

Which axis to quantize along

Activations are quantized per token because per-token scales factor out of a matrix product with the weights. The cache is different. The attention kernel dequantizes it to 16 bits as it reads it, or folds a per-channel key step into the query, which stays in 16 bits. So the cache can have one scale per token or one per channel. The right choice depends on where the outliers are, and keys and values differ.

Keys have outlier channels, a few coordinates that are large in almost every token. Two methods for low-bit caches both found this, KIVI (Liu et al. 2024) and KVQuant (Hooper et al. 2024). With per-token quantization, those channels set every token's step, and every other coordinate of every key is rounded coarsely. Per-channel quantization instead gives each coordinate a step set by its own range across tokens. With a zero point, which shifts the levels to cover the channel's own minimum to maximum, the outlier channel's large typical value then costs nothing, because the step depends only on how much the channel varies.

Two more facts make keys the delicate half. A channel that is nearly the same in every token adds nearly the same amount to all of a query's scores, and the softmax ignores a shift shared by all scores. And key errors enter the softmax's exponent. A score error $\delta s_j = \mathbf{q}^\top\delta\mathbf{k}_j/\sqrt{d_k}$ multiplies the weight of key $j$ by about $e^{\delta s_j}$, and it grows with the size of the query.

Values have no fixed outlier channels, and they enter the output linearly, $\mathbf{o} = \sum_ja_j\mathbf{v}_j$. The output is a weighted mix of a few tokens' values, so each of those tokens must be accurate relative to its own size. Per-token quantization keeps each value's error inside that token. Per-channel steps would be set by the largest token in the context and would crush the small ones.

The cell builds keys with two outlier channels and values whose tokens differ in size. Then it measures the attention-output error for all four combinations of axes.

At 4 bits, keys per channel and values per token give a relative attention-output error of 0.148, against 0.374 for per-token keys and values. The order of the four choices is the same at 3 bits. Interviewers ask for exactly this rule, keys per channel and values per token, and for the reason behind each half.

Per-channel keys raise a systems problem. The cache grows one token at a time, and a per-channel step needs a group of tokens to compute. So KIVI keeps the newest tokens, up to 128, in full precision and quantizes keys only in completed groups of 32 tokens. At 2 bits it reported 2.6 times less peak memory for Llama-2-7B, weights included. That allowed up to 4 times larger batches and 2.35 to 3.47 times the throughput.

In production the common choice is simpler. It is an FP8 cache with one scale per tensor, which costs little quality and halves $m_{\text{KV}}$. In the vLLM serving engine it is the flag --kv-cache-dtype fp8. vLLM's published recipe for NVIDIA's Nemotron 3 Ultra (vLLM 2026) and its benchmarks of DeepSeek-V4-Pro (vLLM 2026) both run with an FP8 cache.

import numpy as np

def quantize_kv(K, V, b, k_mode, v_mode):
    raise NotImplementedError

def attn_error(Q, K, V, K_hat, V_hat):
    raise NotImplementedError

Measuring what quantization costs

A quantized model can match its parent's perplexity and benchmark average and still be a different model. Perplexity averages the loss of the true next token only, so it misses changes in the rest of the distribution. An accuracy average can stay put while many individual answers change in both directions. Three measurements do better.

Divergence from the original. Run both models on the same real prompts. At each position, compare the original's next-token distribution $\mathbf{p}$ with the quantized model's $\mathbf{q}$. The comparison is the KL divergence $D_{\mathrm{KL}}(\mathbf{p}\,\Vert\,\mathbf{q}) = \sum_v p_v\log(p_v/q_v)$ (§0.5), averaged over positions. This needs no labels, and it compares every token's whole distribution. It also has an exact link to the error in the logits, the scores the softmax turns into probabilities.

In plain words, the KL counts only the spread of the logit error, weighted by where the model puts its probability. An error that shifts every logit equally costs nothing, and an error on tokens the model never considers costs almost nothing. Those are exactly the errors a quality metric should ignore.

Flips. On a labeled benchmark, compare the answers item by item. Out of $n$ items, let $n_{10}$ count the ones the original gets right and the quantized model gets wrong, and $n_{01}$ the reverse. The accuracy changes by $(n_{01} - n_{10})/n$. The fraction of answers that flipped is $(n_{01} + n_{10})/n$, which is never smaller and can be much larger.

Dutta et al. (2024) measured both on seven benchmarks, among them MMLU, a multiple-choice knowledge benchmark. Accuracy changed by only 0 to 2% under every quantization scheme they tried. Yet only 8-bit GPTQ weights kept the flips negligible. The bitsandbytes library's 4-bit weights and activations (W4A4) flipped 8 to 16% of MMLU answers, and even Llama 2 70B flipped 3 to 8% under 4-bit quantization. They also found flips well correlated with KL.

Because both models answer the same items, the right test of whether accuracy changed is a paired one (§0.11). McNemar's exact test looks only at the $n_{01} + n_{10}$ items where the models disagree. It asks whether $n_{01}$, the count that improved, is consistent with a fair coin flip on each of them.

The cell checks the KL expansion on random logits and then runs the flip arithmetic on a made-up benchmark of 1,000 questions.

The quadratic approximation matches the exact KL to three digits once the logit error is small. The benchmark example is the trap in miniature. Accuracy moved by 0.4 points, and McNemar's test finds no evidence of a change ($p = 0.70$). Yet 6% of the answers are different. Both statements are true. The models are about equally accurate on this benchmark, and they are not the same model.

Long and hard tasks. Errors compound over long generations and long contexts, so evaluate on those directly, not only on short multiple-choice questions. Liu et al. (2025) quantized reasoning models, which write long chains of thought before answering. They called an accuracy drop of at most 1% lossless and one of 3% or more risky. Weights, activations, and KV cache all in 8 bits were lossless across model sizes and tasks. 4-bit weight-only and 4-bit KV-cache quantization were nearly lossless on most models, though 4-bit weights cost the smallest, 1.5B, model 2.1 points. 3-bit weights were risky. Mildly quantized models did not reason for longer. But aggressive low-bit settings made small models produce longer outputs, a cost the accuracy table does not show.

A sound evaluation of a quantized model follows from all this. It is also the answer to the most common interview question on quantization, how you would know a quantized model is safe to ship.

1. Compute the KL to the original on a sample of real production prompts, and per layer group if a number looks high.
2. Report accuracy changes with paired statistics, and report flips, on the benchmarks that matter for the product.
3. Run long-context retrieval and multi-step reasoning tasks, and compare output lengths.
4. Sample many long generations and compare a quality signal over them (a reward model or a judge), as Meta did for FP8, because rare corrupted outputs do not move averages.
5. Calibrate on data that looks like production (other languages, code, tool outputs), and test on data the calibration did not see.

How to put error bars on benchmark scores in general is the subject of §22.2.

Models born in low precision

Post-training quantization asks a finished model to tolerate rounding. Since 2025 several labs have instead trained or post-trained models in their deployment format. One way is quantization-aware training (QAT). It runs the quantizer in the forward pass during training, and it passes gradients through the rounding as if the rounding were the identity, the straight-through estimator of §20.2. The other way is to train in low precision from the start. Each release below is a result reported by its developer.

• Gemma 3 QAT (Google, April 2025). Int4 checkpoints trained with QAT cut Gemma 3 27B from 54 GB in BF16 to 14.1 GB, small enough for a 24 GB consumer GPU. Q4_0, a common 4-bit format of the llama.cpp library, loses some perplexity when applied to the BF16 model directly. QAT cut that loss by 54% (Google 2025).
• gpt-oss (OpenAI, August 2025). The MoE weights were post-trained in MXFP4, as in the example above.
• NVFP4 pretraining (NVIDIA, September 2025). A 12-billion-parameter model was trained on 10 trillion tokens with NVFP4 matrix multiplications (NVIDIA 2025). It reached 62.58% on MMLU-Pro, a harder version of MMLU, against 62.62% for the same model trained in FP8. It used random Hadamard transforms on the inputs of the weight-gradient products, two-dimensional $16\times16$ block scales for the weights, stochastic rounding for gradients, and 16% of the linear layers (those of the first two and the last eight blocks) kept in BF16. The pretraining recipe is taught in §8.5.
• Kimi K2 Thinking (Moonshot AI, November 2025). QAT during post-training made the MoE weights INT4, weight-only. The model card reports roughly twice the generation speed in low-latency serving, with all benchmark results reported at INT4 (Moonshot AI 2025).
• 2026 releases. Checkpoints in NVFP4, MXFP4, and MXFP8 are now often published alongside or instead of BF16 ones. vLLM's posts of June 2026 describe serving NVIDIA's Nemotron 3 Ultra from an NVFP4 checkpoint (vLLM 2026) and MiniMax M3 from MXFP8 expert weights (vLLM 2026).

DeepSeek-V3 is the earlier case. It was pretrained in FP8 with $1\times128$ and $128\times128$ block scales (§8.5), and its released weights are in that format. Training in the other direction, small adapters on top of frozen 4-bit weights, is QLoRA (§14.6). Its NF4 format is a non-uniform 4-bit code whose 16 levels sit at quantiles of a normal distribution, the usual shape of pretrained weights, with a scale per block.

Choosing a method

The table puts the methods of this section side by side, by what each changes, what data it needs, and where it is used.

In PyTorch

The block below is GPTQ as its authors run it, with lazy blocks of rows and the Cholesky factor of the inverse Hessian computed once with PyTorch's kernels. It then stores FP8 E4M3 weights with one scale per output column, using PyTorch's native float8 type. Production tools (GPTQModel, llm-compressor, AutoAWQ, TensorRT Model Optimizer) wrap the same steps in calibration loops over the model's blocks and pack the codes for their kernels.

Where it breaks

Read more

• Hassibi & Stork (1993). Second order derivatives for network pruning: Optimal Brain Surgeon. The inverse-Hessian update and its cost, for pruning, the root of every method in this section. Frantar, Singh & Alistarh (2022), Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruning, extend it to quantization.
• Frantar et al. (2022). GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers. The fixed order, lazy batches, and Cholesky reformulation, with the OPT and BLOOM results quoted here. Frantar et al. (2024), MARLIN: Mixed-Precision Auto-Regressive Parallel Inference on Large Language Models, make its 4-bit format fast at the batch sizes servers use.
• Lin et al. (2023). AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration. Salient channels found from activations, the scaling argument, and the evidence on calibration robustness.
• Xiao et al. (2022). SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models. Migration of activation outliers into the weights, the O1 to O3 settings, and INT8 serving up to 530B parameters.
• Ashkboos et al. (2024). QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs. Where rotations go in a whole Transformer and how to fold them. Read with Liu et al. (2024), SpinQuant: LLM quantization with learned rotations, and Chee et al. (2023), QuIP: 2-Bit Quantization of Large Language Models With Guarantees, for the theory of incoherence.
• Rouhani et al. (2023). Microscaling Data Formats for Deep Learning. An evaluation of the OCP MX formats, with their conversion rule. Read with NVIDIA's Introducing NVFP4 for Efficient and Accurate Low-Precision Inference and Pretraining Large Language Models with NVFP4 for NVFP4's two-level scales.
• Liu et al. (2024). KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache. Keys per channel, values per token, and the streaming residual. Hooper et al. (2024), KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization, add pre-RoPE keys and non-uniform levels. The vLLM team's The State of FP8 KV-Cache and Attention Quantization in vLLM (2026) is the production view, with the long-context accumulation failure and when to stay in BF16.
• Kurtic et al. (2024). "Give Me BF16 or Give Me Death"? Accuracy-Performance Trade-Offs in LLM Quantization. The largest evaluation of FP8, INT8, and INT4 formats, with deployment advice. Read with Dutta et al. (2024), Accuracy is Not All You Need, on flips and KL, and Liu et al. (2025), Quantization Hurts Reasoning? An Empirical Study on Quantized Reasoning Models, on long reasoning.

Watch next

▶

▶

▶