In bfloat16, $1 + 2^{-9} = 1$. Add a thousand updates of size $0.001$ to a weight of 1 in bfloat16, and the weight never moves. Store a gradient of $10^{-8}$ in float16 without scaling it first, and it becomes zero. Store 500 in the 8-bit format E4M3 and it comes back as 448, the largest number the format has.

These are not exotic corner cases. They are what happens every step in a modern training run, which does almost all of its arithmetic in 16 or 8 bits because that is where the speed is. An H100 runs dense matrix products at about 989 TFLOP/s in bfloat16 and 1,979 TFLOP/s in FP8, against 67 TFLOP/s in plain float32 without tensor cores. A recipe that cannot survive low precision gives up most of the hardware.

Sections 8.2 to 8.4 built the data. From here to the end of the chapter we train on it, and we start with the number formats because every later section leans on them. The optimizer's memory bill in §8.6 is counted in bytes per parameter, and which copy lives in which format is decided here. Adam's $\epsilon$ problem at scale, the logit drift that z-loss fights in §8.8, and the FP8 recipe of DeepSeek-V3 in §8.9 are all, at bottom, questions about what a few bits can represent. Chapter 0 introduced the formats, autocast, and a dynamic loss scaler (§0.9), and our GPT trained in bfloat16 (§6.3). This section goes all the way down, bit by bit, through what each format can and cannot represent and the techniques that make low-precision training work (fp32 accumulation, master weights, compensated summation, loss scaling, stochastic rounding, and fine-grained scaling).

How a floating-point number is stored

A binary floating-point format with $E$ exponent bits and $M$ mantissa bits stores three fields, a sign bit $s$, an exponent field $e$ (an unsigned integer from 0 to $2^E - 1$), and a mantissa field $m$ (an unsigned integer from 0 to $2^M - 1$). The exponent is stored with a bias $b = 2^{E-1} - 1$ so that it can represent negative powers. For a normal number, $1 \le e \le 2^E - 2$, the value is

In plain words, the exponent picks a power of two, and the mantissa picks one of $2^M$ evenly spaced points between that power and the next. The leading 1 is not stored (it is implicit), which buys one free bit of precision.

Two exponent patterns are reserved. When $e = 0$, the number is subnormal, $x = (-1)^s\,2^{1-b}\,(m/2^M)$, with no implicit 1. Subnormals fill the gap between zero and the smallest normal number with evenly spaced values, so precision degrades gradually instead of falling off a cliff. When $e = 2^E - 1$ (all ones), IEEE formats store infinity (if $m = 0$) or NaN (otherwise).

Everything else follows from these rules. Working them out once is the best way to remember the numbers.

• Largest normal number. The largest exponent is $e = 2^E - 2$, so $e - b = 2^{E-1} - 1 = b$, and the largest mantissa gives $1 + (2^M - 1)/2^M = 2 - 2^{-M}$. So $x_{\max} = (2 - 2^{-M})\,2^{b}$.
• Smallest normal number, $2^{1-b}$, from $e = 1$ and $m = 0$.
• Smallest subnormal, $2^{1-b-M}$, from $e = 0$ and $m = 1$.
• Machine epsilon, the gap between 1 and the next number, $2^{-M}$. Rounding to nearest makes the relative error at most half of that, $2^{-M-1}$, anywhere in the normal range.

For FP16, with $E = 5$ and $M = 10$, the bias is 15, so $x_{\max} = (2 - 2^{-10})\cdot 2^{15} = 65{,}504$, the smallest normal is $2^{-14} \approx 6.1\times10^{-5}$, the smallest subnormal is $2^{-24} \approx 5.96\times10^{-8}$, and epsilon is $2^{-10} \approx 9.8\times10^{-4}$.

The FP8 format E4M3 bends the rules to gain range (Micikevicius et al. 2022). It has no infinities, and it uses the all-ones exponent for ordinary numbers, reserving only the single pattern S.1111.111 for NaN. Its largest number is therefore $2^{15 - 7}\cdot(1 + 6/8) = 1.75\times 2^8 = 448$ rather than the 240 that IEEE rules would give. E5M2 follows the IEEE conventions, so its maximum is $1.75\times 2^{15} = 57{,}344$, with infinities and NaNs.

TF32 is not a storage format. It is the internal format NVIDIA tensor cores (A100 and later) use when asked to multiply float32 matrices fast. Inputs are rounded to 10 mantissa bits, keeping float32's 8-bit exponent, and products are accumulated in float32. PyTorch enables it for matrix products with torch.backends.cuda.matmul.allow_tf32 = True or torch.set_float32_matmul_precision("high").

def decode(bits, E, M, ieee=True):
    raise NotImplementedError

Range against precision

The exponent bits buy range, the mantissa bits buy precision, and with 16 or 8 bits you cannot have both. The two 16-bit formats sit at opposite ends.

FP16 has 10 mantissa bits, relative precision $2^{-11} \approx 0.05$ percent, but its normal numbers span only from $6.1\times10^{-5}$ to 65,504. Gradients routinely fall below that floor, and activations can exceed the ceiling.

BF16 ("brain float", from Google Brain) keeps float32's 8-bit exponent and so its entire range, and gives up mantissa bits, keeping 7. Its relative precision is $2^{-8} \approx 0.4$ percent, about two to three significant decimal digits. Kalamkar et al. (2019) showed that BF16 training matches FP32 across many workloads with no change to hyperparameters and no loss scaling. Because BF16 is the top half of an FP32 number, converting between them is a matter of rounding off the lower 16 bits, which also makes it trivial to simulate.

A useful way to see the trade is that relative precision is constant across a format's range, while the absolute spacing between neighbors (one unit in the last place, ulp) grows with magnitude. In BF16 the spacing is $2^{-7}$ between 1 and 2, $0.5$ between 64 and 128, and $2$ between 256 and 512. So in BF16, the integers above 256 are not all representable, which is one reason logits should not drift to large values (§8.8) and why token counts, positions, and loss sums are kept in higher precision.

Rounding and overflow, simulated

To experiment with any format in NumPy, round each value to the nearest representable number. For a value $x$ in the normal range, its exponent is $\lfloor\log_2|x|\rfloor$, and the spacing between neighbors there is $q = 2^{\lfloor\log_2|x|\rfloor - M}$. Below the smallest normal, the spacing stays at $2^{1 - b - M}$ (that is what subnormals are). So

where $\operatorname{rint}$ rounds to the nearest integer with ties to even, the IEEE default. Anything that rounds above the largest value overflows, to infinity in IEEE formats and, by the usual convention for E4M3, to the largest value (saturation).

import numpy as np

def round_to_format(x, E, M, ieee=True, overflow="inf"):
    return np.asarray(x, dtype=np.float64)          # placeholder: no rounding at all

Why accumulation needs more bits

Low precision is safest for storing numbers and riskiest for adding many of them. When you add a small number to a large one, the small one is rounded to the large one's spacing, and if it is less than half a spacing, it disappears. This is called swamping. Summing $n$ numbers in a low-precision accumulator, the running sum grows until each new term is below half its ulp, and the sum stops growing.

A matrix product $\mathbf{C} = \mathbf{A}\mathbf{B}$ with inner dimension $K$ sums $K$ products per output. With $K = 4{,}096$ or more, accumulating in BF16 would lose most of the low-order terms, so tensor cores multiply in 16 or 8 bits but accumulate in FP32. That one design decision is what makes 16-bit training work. It is also where FP8 needed extra care. DeepSeek-V3's authors found that the FP8 tensor cores of their H800 GPUs accumulate with only about 14 bits of precision, which gave relative errors of up to about 2 percent for $K = 4{,}096$. Their fix was to copy partial sums out to FP32 registers every 128 elements of $K$ (DeepSeek-AI 2024).

The BF16 sum stalls at 16, far short of the true 100. At 16, BF16's spacing is $2^{4-7} = 0.125$, and every term (at most 0.04) is below half of that, so each addition rounds back to 16. FP16, with three more mantissa bits, reaches 88 and is still short, because above 64 its spacing is $1/16$ and every term below $1/32$ rounds away. The last line is the same effect on a weight update, the "BF16 weight that never moves", and it is the reason for the master weights of the next part.

Two more facts about summation come up in interviews. First, floating-point addition is not associative. In BF16, $(256 + 1) + 1 = 256$, because each 1 is half an ulp of 256 and rounds away (ties go to the even neighbor), while $256 + (1 + 1) = 258$. So the same reduction computed in a different order, for example on a different number of GPUs or with a different kernel tiling, gives slightly different bits, which is one reason two runs with identical seeds can diverge. Second, the order can be chosen to shrink the error. Adding $n$ terms one after another has a worst-case relative error that grows like $n\cdot 2^{-M-1}$, while pairwise summation, which adds neighbors and then sums the partial sums up a tree, grows like $\log_2 n\cdot 2^{-M-1}$. GPU reductions and the tile-by-tile accumulation inside matrix products are close to the pairwise case, which is part of why FP32 accumulators are enough even for very long sums.

Mixed precision and master weights

The standard recipe, mixed-precision training (Micikevicius et al. 2018), keeps three things in different precisions.

1. Master weights in FP32. The optimizer updates an FP32 copy of every weight. A BF16 or FP16 copy is made from it each step for the forward and backward passes.
2. Matrix products in 16 bits with FP32 accumulation, as above.
3. Precision-sensitive operations in FP32. Softmax, normalization statistics, the loss, and large reductions run in FP32 (autocast does this per operation, §0.9).

Why are master weights necessary? A weight $w$ updated by $\Delta$ changes only if $|\Delta|$ is at least half an ulp of $w$. In a format with $M$ mantissa bits, a weight between $2^e$ and $2^{e+1}$ has ulp $2^{e-M}$, which lies between $2^{-M-1}|w|$ (just below the next power of two) and $2^{-M}|w|$ (at a power of two). Half an ulp is therefore between $2^{-M-2}|w|$ and $2^{-M-1}|w|$, so updates with

can vanish entirely, and those below $2^{-M-2}$ always do. For FP16 that ratio is $2^{-11} \approx 1/2048$, exactly the threshold Micikevicius et al. reported, and for BF16 it is $2^{-8} \approx 1/256$. That is the demo of the cell above, where every update of 0.001 to a weight of 1.0 is below $2^{-8} \approx 0.0039$ and is lost. Typical updates in a long training run are $10^{-4}$ to $10^{-3}$ of the weight's size, so in BF16 a large fraction of all updates would round away. In FP32, with $2^{-24}$, they never do.

The cost is memory. With Adam, mixed-precision training stores about 16 bytes per parameter (two each for the 16-bit weights and gradients, four for the FP32 master copy, and four for each of Adam's two moments), and §8.6 counts them in detail. The master copy is a quarter of that. Some recipes cut further. DeepSeek-V3 kept the master weights and gradients in FP32 but stored AdamW's two moments in BF16. The same threshold explains why that is safe. A moving average with decay $\beta$ moves by a fraction of about $1 - \beta$ of its value each step, 10 percent for $\beta_1 = 0.9$ and 5 percent for $\beta_2 = 0.95$, far above BF16's $1/256$, so its updates are not rounded away. With Adam's default $\beta_2 = 0.999$, the second moment would move by 0.1 percent per step, below the threshold, so many of a BF16 second moment's updates would be rounded away.

Compensated summation, a cheaper fix

There is a way to keep weights in BF16 and still lose almost nothing, by remembering what each rounding threw away. Kahan summation (Kahan 1965) keeps a second, small number $c$ per weight, the error of the last addition, and subtracts it from the next update,

In plain words, $t - w$ is what the rounded addition actually added, $y$ is what we wanted to add, and their difference is the error, which gets paid back on the next step. A run of updates that are each too small to move $w$ piles up in $c$ until their sum is big enough to round $w$ up by one ulp, at which point $c$ is reset by the same formula. The compensation is a small number, so it can itself be stored in BF16 with little relative error. Zamirai et al. (2020) showed that BF16 weights with a Kahan-compensated update closed most of the gap to FP32 master weights, and several BF16 optimizer implementations offer it. The weight then costs 4 bytes (a BF16 value and a BF16 compensation) instead of the 6 of a BF16 working copy plus an FP32 master, and the update stays deterministic, unlike stochastic rounding below.

import numpy as np

def kahan_update(w, c, delta):
    return to_bf16(np.asarray(w, np.float32) + np.asarray(delta, np.float32)), c   # placeholder: plain BF16 update

Loss scaling for FP16

FP16 has another problem, the range. Its smallest subnormal is $2^{-24} \approx 6\times10^{-8}$, and gradients of activations are often smaller. In one network Micikevicius et al. studied, the SSD object detector, a large share of the nonzero activation gradients fell below FP16's smallest subnormal and became zero, and training without a fix diverged from the FP32 result. The values that mattered lay between $2^{-27}$ and $2^{-24}$.

The fix is loss scaling. Multiply the loss by a factor $S$ before backpropagation. By the chain rule every gradient is multiplied by $S$ too, which shifts the whole gradient distribution up into FP16's range. Then divide the gradients by $S$ in FP32, before clipping and the optimizer step. Shifting by a power of two is exact, so if nothing overflows, the result equals an unscaled computation done in a format with more range. For the SSD network a factor of 8 was enough, and the paper chose constant factors between 8 and 32,768 for different networks.

Choosing $S$ by hand is fragile, so frameworks scale dynamically. Start with a large $S$, such as $2^{16}$. If any gradient overflows to infinity or NaN, skip that step and halve $S$. After a run of good steps (2,000 in PyTorch's GradScaler), double $S$. The scale settles just below the overflow point and tracks the gradient magnitudes as they change during training. You built exactly this scaler in Exercise 0.9d (§0.9). Two details matter at scale. Unscale before clipping, or the clipping threshold is applied to a gradient $S$ times too large. And every skipped step is a step the schedule still counts, so a run that skips often is silently training with less data than planned, which is worth logging. BF16, with float32's range, needs none of this, which is the main reason it replaced FP16 for training once hardware supported it.

Stochastic rounding

Round-to-nearest has a systematic flaw for accumulation. An update smaller than half an ulp is lost every time, so a thousand small pushes in the same direction add up to nothing. Stochastic rounding fixes the bias. Round $x$, which lies between neighbors $x_\downarrow$ and $x_\uparrow = x_\downarrow + q$, up with probability proportional to how close it is,

Then

with $p = (x - x_\downarrow)/q$. In plain words, stochastic rounding is unbiased. A tiny update is usually lost, but occasionally it moves the weight a whole ulp, and on average it lands exactly where it should. The cost is noise of at most half an ulp per operation. Gupta et al. (2015) showed that with stochastic rounding, networks could be trained in 16-bit fixed point with little or no loss, in settings where round-to-nearest stopped learning because small updates rounded to zero.

In practice, stochastic rounding lets the weights themselves live in BF16 without an FP32 master copy or a compensation buffer, saving all 4 bytes per parameter of the master copy, and it is supported in hardware on some accelerators. For BF16 it has a neat implementation. Add a random 16-bit integer to the lower half of the FP32 bit pattern, then truncate those 16 bits. The carry into the upper half happens with exactly the probability above.

import numpy as np

def sr_bf16(x, rng):
    raise NotImplementedError

FP8 training

On an H100, FP8 matrix products run at twice the BF16 rate, and FP8 tensors take half the memory and bandwidth. With only 3 or 2 mantissa bits and a narrow range, though, FP8 cannot be used naively. The recipes that work have four ingredients.

Two formats for two jobs. Micikevicius et al. (2022) proposed E4M3 for weights and activations in the forward pass, where precision matters more, and E5M2 for gradients in the backward pass, where values span a wider range. They trained models up to GPT-3 175B in FP8 with this split, matching the 16-bit baselines.

Scaling factors. An FP8 tensor is stored together with a higher-precision scale $s$, so the represented values are $s\cdot q$ for FP8 numbers $q$. The scale is chosen so the tensor's largest magnitude, its amax, maps to the top of the format, $s = \operatorname{amax}/x_{\max}$ (448 for E4M3). With one scale per tensor, NVIDIA's Transformer Engine uses delayed scaling. It computes each tensor's scale from the maximum amax over a history of recent steps (1,024 by default) rather than from the current tensor, which avoids an extra pass over the data but can be wrong when values suddenly grow.

Fine-grained scales for outliers. One per-tensor scale is held hostage by the tensor's largest value. E4M3's normal numbers span a factor of about 28,700, from 0.0156 to 448, so as long as the largest value is within a few thousand times the typical ones, a per-tensor scale costs little. But LLMs have rare massive activations, up to 100,000 times larger than the rest (Sun et al. 2024), and with a scale set by one of those, every ordinary value is pushed into E4M3's subnormal range or to zero. DeepSeek-V3 used a scale for every $1\times128$ tile of activations (128 consecutive channels of one token) and every $128\times128$ block of weights, computed on the fly from the current values. That let them use E4M3 for all tensors, gradients included. Blackwell GPUs support this idea in hardware with microscaling formats, in which every block of 32 values shares a power-of-two scale.

High precision where it matters. DeepSeek-V3 kept the embedding, the output head, the MoE gating, normalization, and attention in BF16 or FP32, promoted FP8 accumulations to FP32 every 128 elements, and kept master weights and gradients in FP32. In validation runs on smaller models, at the scale of their earlier V2 models trained for about a trillion tokens, the loss of FP8 training stayed within 0.25 percent of BF16. Then they trained the 671B-parameter V3 this way (DeepSeek-AI 2024).

Long runs, not short ones, are the test. Fishman et al. (2024) trained a 7B model in FP8 on up to 2 trillion tokens and found instabilities that shorter runs had never shown. They traced them to SwiGLU (§5.3), whose product of two linear projections slowly develops aligned weights during training and amplifies rare activations into outliers that FP8 cannot hold. A rescaled variant, Smooth-SwiGLU, fixed it without changing what the layer computes. The lesson generalizes. A low-precision recipe validated on a few hundred billion tokens has not yet been validated.

import numpy as np

def fp8_quant_dequant(X, block=None):
    raise NotImplementedError

Low precision at inference time, where weights are fixed and calibration is possible, has its own, larger toolbox (8-bit and 4-bit integer quantization, outlier handling, KV-cache quantization), which is how most deployed models are run (quantizing a trained model's weights and activations).

In PyTorch

PyTorch has real FP8 dtypes (torch.float8_e4m3fn, the "finite" E4M3 variant without infinities, and torch.float8_e5m2), so we can check our simulator against actual casts, and show the autocast and FP8 scaled-matmul pieces used in practice.

The simulator agrees with PyTorch's casts on every value, for all four formats. On a GPU with FP8 support, torch._scaled_mm(a_fp8, b_fp8, scale_a, scale_b, out_dtype=torch.bfloat16) multiplies FP8 matrices with their scales and accumulates in higher precision, and libraries such as Transformer Engine and torchao wrap it into drop-in FP8 linear layers with the scaling recipes described above.

Read more

• Micikevicius et al. (2018). Mixed Precision Training. FP32 master weights, loss scaling, and FP32 accumulation, with the gradient histograms that motivated them.
• Micikevicius et al. (2022). FP8 Formats for Deep Learning. The definitions of E4M3 and E5M2 and FP8 training up to 175B parameters.
• Kalamkar et al. (2019). A Study of BFLOAT16 for Deep Learning Training. Why BF16 matches FP32 without loss scaling.
• Zamirai et al. (2020). Revisiting BFloat16 Training. Pure BF16 training with stochastic rounding or Kahan summation in place of FP32 master weights.
• Fishman et al. (2024). Scaling FP8 training to trillion-token LLMs. The late-appearing FP8 instability from SwiGLU outliers, and its fix.
• Gupta et al. (2015). Deep Learning with Limited Numerical Precision. Stochastic rounding makes 16-bit fixed-point training work.
• DeepSeek-AI (2024). DeepSeek-V3 Technical Report. Section 3.3 is the most detailed public account of FP8 training at frontier scale.
• Goldberg (1991). What Every Computer Scientist Should Know About Floating-Point Arithmetic. The classic reference on rounding, ulps, and error.

Watch next

▶

▶

▶