Your profiler shows two kernels from the same Transformer layer on an H100. The matrix multiplication in the MLP runs at 700 TFLOP/s, about 70 percent of the chip's bfloat16 peak. The LayerNorm right after it runs at 6 TFLOP/s, well under 1 percent. A colleague proposes spending a week rewriting the LayerNorm to make it "as efficient as the matmul". Should they?

No, and by the end of this section you will be able to prove it in one line. The LayerNorm is already running within about 10 percent of the fastest speed any LayerNorm can run on that GPU. Its limit is not arithmetic, it is the rate at which the GPU can read and write memory, and a LayerNorm does only a handful of operations per byte. The tool that makes this kind of judgment quick and reliable is the roofline model (Williams, Waterman & Patterson, 2009). It needs two numbers about the hardware and two numbers about the kernel, and it tells you which resource limits the kernel, how fast it could possibly go, and therefore whether optimizing it is worth your time.

Two limits on every kernel

Recall from §11.1 that a GPU is a fast calculator fed by a comparatively narrow pipe. Describe the hardware by two numbers.

• $\pi$, the peak compute throughput in FLOP/s. For an H100 SXM in dense bfloat16 on tensor cores, $\pi = 989$ TFLOP/s.
• $\beta$, the peak memory bandwidth in bytes per second between HBM and the chip. For the same GPU, $\beta = 3.35$ TB/s.

Describe a kernel by two numbers as well.

• $F$, the number of floating-point operations it performs.
• $Q$, the number of bytes it moves between HBM and the chip, reads plus writes.

The kernel cannot finish before its arithmetic is done, which takes at least $F/\pi$, and it cannot finish before its data has moved, which takes at least $Q/\beta$. GPUs overlap the two (that was the point of latency hiding in §11.1), so the best possible time is the larger of the two,

In plain words, a kernel is limited by whichever of its two jobs, computing or moving data, takes longer on this machine.

Now divide $F$ by both sides. The achieved throughput $P = F/t$ obeys

is the kernel's arithmetic intensity, its FLOPs per byte of memory traffic. The two limits meet at the ridge point

A kernel with $I < I^\star$ is memory-bound. Its best possible throughput is $\beta I$, which grows in proportion to its intensity, and making its arithmetic faster changes nothing. A kernel with $I > I^\star$ is compute-bound. Its ceiling is $\pi$, and reducing its memory traffic changes nothing.

In plain words, arithmetic intensity is how much work you do with each byte you fetch. Below the ridge point, bytes are the bottleneck and the speed you can reach is proportional to intensity. Above it, FLOPs are.

Back to the puzzle. A LayerNorm does about 8 FLOPs per element (we count them below) and, in bfloat16, reads 2 bytes and writes 2 bytes per element, so $I \approx 8/4 = 2$ FLOPs per byte. Its ceiling is $\beta I = 3.35 \times 2 = 6.7$ TFLOP/s. At 6 TFLOP/s it is at 90 percent of its roof. The only way to make it substantially faster is to move fewer bytes, for example by fusing it into a neighboring kernel so that its input never makes the round trip to HBM (§11.7).

The roofline picture

Plot attainable throughput against intensity, both on logarithmic axes. The bound $\beta I$ is a straight line of slope 1 (the slanted roof), the bound $\pi$ is a horizontal line (the flat roof), and they meet at $I^\star$. Every kernel is a point under this roof. Its horizontal position is its intensity, a property of the algorithm, and the gap between the point and the roof above it is how much an implementation could still gain.

The widget draws the roofline for several GPUs and number formats and places the operations of a Transformer on it. Change the hardware and watch the ridge point move. Change the matrix size or the batch size and watch matmuls slide along the slanted roof until they hit the flat one.

The ridge points of the GPUs you are likely to use, in dense bfloat16 unless stated, all from NVIDIA's datasheets.

The trend matters more than any one entry. Compute has grown faster than bandwidth for decades (the A100 to H100 step tripled $\pi$ but less than doubled $\beta$), so ridge points drift upward, and every generation turns more kernels memory-bound. The H200 shows the same pressure from the other side. It is an H100 with more and faster memory (141 GB at 4.8 TB/s), so on the roofline its only effect is to lower the ridge point and speed up memory-bound work. Its larger capacity matters too, for bigger batches and longer KV caches, but that is a different limit from the ones the roofline draws.

The intensity of a Transformer's operations

Let's derive the intensity of each operation in a Transformer layer. Write $s$ for the bytes per element (2 in bfloat16), and count ideal traffic, each input read once and each output written once, which is what a well-written kernel achieves.

Elementwise operations

Adding two tensors of $n$ elements does $F = n$ FLOPs and moves $Q = 3ns$ bytes (two reads, one write), so

That is about 1,800 times below the H100's ridge point, so an elementwise add runs at about 0.06 percent of peak FLOP/s, and that is the best case. Activation functions are no different. GELU costs perhaps 8 to 20 FLOPs per element depending on how you count tanh or erf, still an intensity of a few FLOPs per byte. Every elementwise operation is memory-bound on every modern GPU, and its time is set by the bytes alone.

Softmax and LayerNorm

A row softmax over $n$ entries finds the maximum ($n$ comparisons), subtracts it ($n$), exponentiates ($n$), sums ($n$), and divides ($n$), about $5n$ FLOPs, while a fused kernel reads the row once and writes it once, $2ns$ bytes. So $I_{\text{softmax}} = 5/(2s) = 1.25$ in bfloat16. (We count exp as one FLOP, although it runs on the slower special-function units of §11.1. The kernel is memory-bound either way.)

A LayerNorm over a row of $d$ features (§5.2) computes the mean (about $d$ FLOPs), the variance (subtract, square, add, about $3d$), the normalization (subtract and multiply by $1/\sqrt{\sigma^2+\epsilon}$, where $\sigma^2$ is the row's variance and $\epsilon$ a small constant, $2d$), and the affine map with the learned gain $\boldsymbol\gamma$ and shift $\boldsymbol\beta$ ($2d$), about $8d$ FLOPs in all, against $2ds$ bytes for the row plus $2ds$ for $\boldsymbol\gamma$ and $\boldsymbol\beta$, which are read once for all rows and so vanish per row. That gives $I_{\text{LN}} \approx 8/(2s) = 2$.

Matrix multiplication

Multiply $\mathbf{A} \in \R^{M\times K}$ by $\mathbf{B} \in \R^{K\times N}$ (generic operands here, with $K$ the inner dimension). Each of the $MN$ outputs is a dot product of length $K$, so $F = 2MNK$ (one multiply and one add per term, the book's convention since §4.7). The minimum traffic is reading both inputs and writing the output once, $Q = s(MK + KN + MN)$, so

For square matrices, $M = N = K = n$, this is $I = 2n^3/(3sn^2) = 2n/(3s)$, or $n/3$ in bfloat16. The intensity grows with the size, because each loaded number takes part in $n$ multiply-adds. A square bfloat16 matmul is compute-bound on an H100 once $n/3 > 295$, that is $n \gtrsim 890$. At $n = 4096$ the intensity is 1,365 and the matmul is firmly compute-bound.

In plain words, matmul is the one operation whose work grows faster than its data, by a factor of the matrix size, which is why it is the only operation that can use the tensor cores fully.

The case that matters most in practice is a linear layer, a batch of $B$ token vectors times a $d\times d$ weight matrix, $M = B$ and $K = N = d$. Then

In bfloat16 the intensity of a linear layer is about $B$, the number of tokens processed together. The weights dominate the traffic, and each weight that is read is used once per token in the batch. So batch size is intensity, which is the single most useful fact in this section.

Attention

Take one head with sequence length $T$ and head width $d_k$, as in §4.3. A naive implementation runs three kernels. The scores $\mathbf{S} = \mathbf{Q}\mathbf{K}^\top$ read $\mathbf{Q}$ and $\mathbf{K}$ ($2Td_k$ elements) and write $\mathbf{S}$ ($T^2$). The softmax reads $\mathbf{S}$ and writes $\mathbf{A}$ ($2T^2$). The output $\mathbf{A}\mathbf{V}$ reads $\mathbf{A}$ and $\mathbf{V}$ ($T^2 + Td_k$) and writes $\mathbf{O}$ ($Td_k$). The FLOPs are $2T^2d_k$ for each product plus about $5T^2$ for the softmax, so

With $d_k = 128$ in bfloat16 this is about 65 FLOPs per byte (63 at $T = 4096$, where the $4Td_k$ term still counts a little, as the cell below shows), well below the H100's ridge of 295. Naive attention is memory-bound, and the reason is the $4T^2$ elements of score and weight traffic, which do not shrink as the head gets wider. A fused kernel that never writes $\mathbf{S}$ or $\mathbf{A}$ to HBM moves only $\mathbf{Q}$, $\mathbf{K}$, $\mathbf{V}$, and $\mathbf{O}$, about $4Td_k$ elements if each is moved once, and its intensity becomes about $T/s$, thousands of FLOPs per byte. That is the core idea of FlashAttention. Fitting it into on-chip memory forces $\mathbf{K}$ and $\mathbf{V}$ to be streamed several times, and §12.3 counts that traffic exactly.

The cell computes all of these on an H100, for a $4096\times4096$ activation in bfloat16, linear layers of width 4,096 at several batch sizes, and one attention head at $T = 4096$.

Look at the times. The add, the softmax, and the LayerNorm each take 20 to 30 microseconds on 16 million elements, while a linear layer on 512 tokens of width 4,096, which does over 17 billion FLOPs, takes about 17. The linear layer at batch 1 and at batch 128 takes about the same time, around 10 microseconds, because in both cases the time is spent reading the same 32 MB of weights. The same table, summarized, is the roofline view of a Transformer layer on an H100.

def op_cost(op, shape, s=2):
    raise NotImplementedError

def roofline(F, Q, peak, bw):
    return F / peak, "compute"          # placeholder: ignores memory entirely

Batch size, prefill, and decode

The linear-layer formula explains the economics of serving language models, which §6.5 introduced. Prefill processes the prompt's $p$ tokens together, so every weight matrix sees $B = p$ tokens and the work is compute-bound for prompts of a few hundred tokens or more. Decode produces one token per sequence per step, so with a single sequence $B = 1$, the intensity is about 1, and the GPU runs at about $\beta \times 1 = 3.35$ TFLOP/s, a third of a percent of peak.

Put numbers on Llama 3 8B in bfloat16, about 16 GB of weights. One decode step must read all of them, at least $16\,\text{GB}/3.35\,\text{TB/s} \approx 4.8$ ms, so one sequence gets at most about 210 tokens per second however clever the kernels are. Batching $B$ sequences reuses each weight $B$ times. Until the batch approaches the ridge point, the step takes about the same 4.8 ms, so throughput grows almost linearly with $B$. This is why inference servers batch requests aggressively, and why the cost per token of serving a model falls so steeply as load grows.

There is a catch that the weights-only picture hides. Each sequence also reads its own KV cache at every step, and that traffic is not shared across the batch. Llama 3 8B's cache is $2 \times 32 \text{ layers} \times 8 \text{ KV heads} \times 128 \times 2$ bytes, 128 KiB per token (the formula is from §6.5, and the 8 shared key-value heads are the grouped-query attention of §12.5), so a sequence with 4,096 tokens of context reads another 0.5 GB per step. The cell models one decode step as $\max(F/\pi, Q/\beta)$, with $Q$ the weights plus every sequence's cache.

With weights alone, throughput rises linearly until about $B \approx 300$ and then flattens at the compute roof. With a 4,096-token context, it bends over much earlier, because the cache traffic grows with the batch. As $B \to \infty$ the intensity tends to the FLOPs per token divided by the cache bytes per token, about 34, far below the ridge point. Decode with long contexts is memory-bound at any batch size. That single fact motivates grouped-query and latent attention (§12.5), KV-cache quantization, and the capacity planning of serving systems. (Memory capacity bites too. The 64 GB left after the weights hold about 120 such sequences.)

def min_batch(d, peak, bw, s=2):
    return round(peak / bw)              # placeholder: intensity about B, ignores s and the d^2 term

Model FLOPs utilization, seen from the hardware

§6.7 defined model FLOPs utilization (MFU) for a training run, the FLOPs the model mathematically requires per second divided by the peak of all its GPUs,

where $n_{\text{gpu}}$ is the number of GPUs, $\pi$ is the peak of one GPU, and training costs about $6N + 12LdT$ FLOPs per token for a model with $N$ parameters (no longer a matrix dimension), $L$ layers of width $d$, and context length $T$. In roofline terms, MFU is how close a whole run sits to the flat roof, averaged over every kernel, every gap between kernels, and every stall. Three choices decide what number comes out, and each is a question about the hardware.

Which FLOPs. MFU counts only what the model needs, so it cannot be inflated by doing extra work. Hardware FLOPs utilization (HFU) counts what the chip actually executed, recomputation included. The forward pass is a third of a training step's FLOPs, $2N + 4LdT$ per token, so full activation recomputation (§10.1), which repeats it, makes HFU about $8/6 = 4/3$ of MFU. Selective recomputation (Korthikanti et al., 2022) recomputes only the attention scores, softmax, and weighted sum of values, which take a lot of memory but few FLOPs, about $4LdT$ per token. HFU says how busy the tensor cores were, and MFU how much of that work was useful. A configuration that recomputes more can show a higher HFU and still train more slowly, which is why MFU is the number to compare across runs.

Which peak. $\pi$ must be the dense tensor-core peak in the format the matmuls run in (the table in §11.1). Dividing by the datasheet's sparse figure halves the reported MFU, and dividing by the 67 TFLOP/s of the FP32 lanes would inflate it fifteenfold. For runs whose matmuls are in FP8, conventions differ. Against the BF16 peak, DeepSeek-V3's pretraining works out to about 35 percent, and against the FP8 peak it actually used, about 17 percent (§8.9), so always say which peak you used.

Which time. Tokens per second is measured over whole steps on the wall clock, so idle gaps between kernels, communication that is not overlapped, and data loading all count against MFU. Every kernel can sit on its own roof while the run's MFU is poor. Even the kernels set a ceiling below 100 percent. Suppose the matmuls, which carry nearly all the FLOPs, run at 75 percent of the peak, while memory-bound operations, which carry almost none, take a quarter of the step. Then MFU is at most $0.75 \times 0.75 \approx 56$ percent before a single byte is communicated.

A worked example. An 8-billion-parameter model with $L = 32$, $d = 4096$, and $T = 4096$ needs $6 \times 8\times10^{9} + 12 \times 32 \times 4096 \times 4096 \approx 5.44\times10^{10}$ FLOPs per token. Trained on 64 H100s at 600,000 tokens per second, its MFU is $5.44\times10^{10} \times 6\times10^{5} / (64 \times 989\times10^{12}) \approx 52$ percent. With full recomputation the chip executes $2N + 4LdT \approx 1.8\times10^{10}$ more FLOPs per token, an HFU of about 69 percent, and the same run divided by the sparse peak would report 26 percent. For a published pair, PaLM 540B reported an MFU of 46.2 percent and an HFU of 57.8 percent (Chowdhery et al., 2022), a ratio of 1.25. Published large dense runs typically report MFUs between about 35 and 55 percent. §10.7 reproduces published MFUs and shows how parallelism choices move them.

def utilization(n_params, n_layers, d_model, seq_len, tokens_per_s, n_gpus, peak, recompute="none", sparse_peak=False):
    u = 6 * n_params * tokens_per_s / (n_gpus * peak)      # placeholder: no attention, no recomputation, raw peak
    return u, u

More roofs, and what the model leaves out

The roofline we drew has one memory roof, HBM. Real chips have one for each level of the hierarchy, and a kernel whose data fits in L2 or shared memory is limited by those faster roofs instead. Profilers such as Nsight Compute draw this hierarchical roofline, with a point per level, and the level whose point sits closest to its roof is the bottleneck. A tiled matmul, for example, can be limited by shared memory bandwidth if its register tiles are too small (§11.5).

There are also lower ceilings below the flat roof. A kernel that cannot use tensor cores (most non-matmul code) is capped by the 67 TFLOP/s of the FP32 lanes, with a ridge point of only 20. And there is a regime the roofline does not draw at all. A kernel that moves 100 KB and does a million FLOPs should take about 30 nanoseconds by the roofline, but launching it costs a few microseconds. Such work is overhead-bound, and no amount of kernel tuning helps. Only doing more work per launch, by batching, fusing, or capturing CUDA graphs, does (§11.7). Horace He's essay names the three regimes, compute, memory bandwidth, and overhead, and gives the right first question for any slow model, which of the three am I in?

In PyTorch

You can draw a roofline for your own CPU. The block times matrix products of growing size and an elementwise add, and reports intensity and achieved throughput. Small matmuls sit on the slanted part (and lower still, because of call overhead), large ones approach the flat part.

Read more

• Williams, Waterman & Patterson (2009). Roofline: An Insightful Visual Performance Model for Multicore Architectures. The original paper, with the model, the ceilings, and worked examples on real machines.
• He (2022). Making Deep Learning Go Brrrr From First Principles. Compute, memory bandwidth, and overhead as the three regimes of deep learning performance, with the arithmetic that tells them apart.
• NVIDIA. GPU Performance Background User's Guide. Arithmetic intensity of common layers and how NVIDIA recommends reasoning about them.
• NVIDIA. NVIDIA H100 Tensor Core GPU. The datasheet behind our peak numbers (remember that starred entries assume sparsity).
• Austin et al. (2025). How to Scale Your Model. A modern, careful treatment of rooflines for training and inference, written for TPUs but equally true for GPUs.
• Korthikanti et al. (2022). Reducing Activation Recomputation in Large Transformer Models. Selective recomputation, and the model and hardware FLOPs accounting behind MFU and HFU.
• Pope et al. (2022). Efficiently Scaling Transformer Inference. Prefill, decode, batch size, and the memory-bound regime of large-model inference, worked out with these tools.

Watch next

▶

▶