What a generated token really costs, and every lever that makes it cheaper. Fewer bytes per number, fewer numbers, more tokens per weight read, and more users per GPU, from first principles to the systems that serve frontier models.

Inference is running a trained model to answer requests. The chapters before this one built such a model. They pretrained it at scale (§8.9) and post-trained it to follow instructions (Chapter 14), to reason (Chapter 18), and to behave safely (Chapter 19). Training happens once, but serving goes on for as long as the model is used. Training a model of $N$ parameters costs about $6N$ floating-point operations, or FLOPs, per token, and serving it costs about $2N$. So once a model has served three times as many tokens as it was trained on, serving has cost as much compute as training (§9.4). Reasoning models, which write thousands of tokens per answer (§18.7), get there quickly. This chapter is about making that cost small.

A language model writes its answer one token at a time, and every second of GPU time spent writing has a price. This section builds a small model of that time from two hardware numbers. The model says how long one generated token takes and which part of the GPU sets that time. It says how many users fit on one GPU, what spreading a model over several GPUs buys, and what a million tokens cost. Every later section of this chapter shrinks one term of this model, so the model is the map for the whole chapter.

Two facts show why we need it. The first is about price. In February 2025 DeepSeek charged $2.19 per million output tokens for its R1 model and $0.55 per million input tokens, so a written token cost four times as much as a read one (DeepSeek-AI 2025). The $0.55 is for prompt tokens the service had to compute. Prompt beginnings it had already stored from an earlier request cost only $0.14.

Yet a Transformer does about the same arithmetic for a token it reads as for a token it writes. Each weight on a token's path does one multiplication and one addition for that token. So a dense model, one that uses all $N$ of its weights on every token, spends about $2N$ FLOPs per token (§5.6). Why should writing a token cost four times as much as reading one?

The second fact is about speed. On one H100 GPU, Llama 3.1 8B can write at most about 200 tokens per second for a single user, faster than anyone reads. DeepSeek's service gave each user only 20 to 22 tokens per second. Yet each of its eight-GPU machines, while generating, produced about 14,800 tokens per second for hundreds of users at once. Serving many people together makes each of them slower and makes every token far cheaper. By how much, and where does the trade stop paying?

Both answers come from the roofline of §11.2, which we recap as we need it. It says a GPU program is limited either by its arithmetic or by its memory traffic, whichever takes longer. Two hardware numbers therefore appear everywhere. $\pi$ is a GPU's peak arithmetic rate in FLOPs per second. $\beta$ is its memory bandwidth, the bytes per second it can move between its main memory and its compute units. That main memory is called HBM (high-bandwidth memory).

For an H100 multiplying matrices in bfloat16, the 2-byte floating-point format models are usually served in, $\pi = 989$ TFLOP/s and $\beta = 3.35$ TB/s. Datasheets also print twice that FLOP rate, but it holds only for matrices in which two of every four weights are zero, so we never use it. Other chapters use $\pi$ and $\beta$ for other things. In this chapter they always mean these two hardware rates.

One request, from arrival to last token

A request arrives with a prompt of $T_{\text{in}}$ tokens and will generate $T_{\text{out}}$ tokens. To write a token, every attention layer needs the keys and values of all earlier tokens. Recomputing them at each step would repeat almost all the work, so the server stores them. This store is the KV cache, one key vector and one value vector per token per layer (§6.5). With it, a request lives through three phases.

1. Queueing. The request waits until the server has room for it.
2. Prefill. One forward pass processes all $T_{\text{in}}$ prompt tokens at once. It writes their keys and values into the cache. It also produces the logits, one score per vocabulary entry, from which the first output token is sampled.
3. Decode. Each later step runs the model on one new token per sequence. It reads that sequence's whole cache, appends one key and one value per layer, and samples the next token. This repeats $T_{\text{out}} - 1$ times, until an end-of-sequence token or a length limit.

The server decodes many sequences together, and the set of sequences in one decode step is the batch, of size $B$. The two computing phases behave very differently. Prefill handles many tokens at once, so each weight it reads from memory is used for all $T_{\text{in}}$ of them. Its time is set by arithmetic, which is called compute-bound.

Decode handles one token per sequence. Each weight it reads is used once per sequence in the batch, and each sequence also reads its own cache, which no other sequence uses. Its time is set by memory traffic, which is called memory-bound. §6.5 measured this, and the whole section follows from it.

How a server is measured

Before modeling a server we need to say exactly what "fast" means. A server can be fast for one user and slow for the fleet. It can also be fast on average and terrible for one request in a hundred. The standard metrics below keep these cases apart, and interviewers expect you to know them exactly.

In plain words, TTFT is how long you stare at a blank screen, and TPOT is how fast the words arrive after that. E2E is the whole wait. Throughput is how many tokens the server turns out, and goodput is how much traffic it can take before too many users are unhappy.

By the definitions, $\text{E2E}_i = \text{TTFT}_i + (T_{\text{out},i} - 1)\,\text{TPOT}_i$. Some sources write $\text{TTFT} + T_{\text{out}}\cdot\text{TPOT}$ instead (Databricks 2023), which counts one gap too many. The difference is small for long outputs and large for short ones, so say which convention you use.

For scale, a TPOT of 100 ms is 10 tokens per second. The same source puts that at about 450 words per minute, faster than people read. Below reading speed, users wait for the text. Above it, extra speed matters mostly for agents and for reasoning models, which write long chains of intermediate steps that nobody reads.

Why measure ITL when TPOT already averages the gaps? Because an average hides stalls.

Why goodput rather than throughput? Throughput can always be raised by batching more, at the price of slower tokens for everyone. A server tuned only for throughput therefore drifts past the point where its users are served well. Goodput counts only traffic that the server can carry while meeting the SLO. That is what an operator can sell, since each GPU's cost must be recovered from the requests it serves well. Zhong et al. (2024) defined it to compare serving systems, and the scheduling designs of §20.6 aim at it.

One more number is common in practice. Model bandwidth utilization (MBU) asks how close decode comes to the GPU's peak bandwidth (Databricks 2023). Divide the bytes a decode step must read (weights plus caches) by the TPOT, then divide by the peak bandwidth. A 7B model in 16-bit with a TPOT of 14 ms moves 14 GB in 14 ms. That is 1 TB/s, an MBU of 50% on a GPU with 2 TB/s. MBU plays the part for memory-bound decode that model FLOPs utilization (MFU), the fraction of peak FLOP/s that compute-bound work achieves (§6.7), plays for training and prefill.

An interviewer will expect you to define TTFT and TPOT exactly, to say that TTFT includes the queue, and to explain why SLOs are written on p99 rather than on the mean.

import numpy as np

def latency_metrics(log):
    raise NotImplementedError

How many requests are in the system

The model of a decode step below takes the batch size $B$ as given. In a real server, traffic sets it, through one of the few exact laws about queues. The number of people in a shop, on average, equals the rate at which they walk in times how long each one stays. Fifty customers an hour who each stay twelve minutes keep ten people in the shop on average, however they bunch up at the door. This is Little's law, after the operations researcher who first proved it. §11.1 used it for the bytes in flight between a GPU and its memory. Here it counts requests in a server.

The law applies to any part of the system too, such as only the requests that are decoding. That is what makes it useful for sizing memory.

A small simulation checks the law and shows its limit. The cell draws random arrivals at 50 per second for five minutes. It gives each request a random TTFT with mean 1 s and a random decode time with mean 11 s. Then it counts the requests inside at many instants.

The time averages land within about 1% of $\lambda_{\text{req}}\overline{\text{E2E}}$, as the law says. The busiest instant, though, holds about 10% more decoding sequences than the mean, and the cache memory must hold that peak. Little's law gives the mean only, so size memory with headroom for bursts, or let the scheduler queue requests when memory runs out.

A model of one decode step

Now we turn the bus picture into numbers, starting from the roofline of §11.2. A kernel is one program the GPU runs. A kernel that does $F$ FLOPs and moves $Q$ bytes takes at least $\max(F/\pi, Q/\beta)$ seconds, because it can go no faster than its arithmetic and no faster than its memory traffic. Its arithmetic intensity $F/Q$ is how many FLOPs it does per byte it moves. The ridge point $I^\star = \pi/\beta$ separates memory-bound kernels (intensity below it) from compute-bound ones. For an H100 in bfloat16, $I^\star = 989/3.35 \approx 295$ FLOPs per byte.

Here is the key fact about decode. A linear layer applied to $B$ tokens has an intensity of about $B$ in bfloat16. Each 2-byte weight is read once and used once per token, a multiply and an add, so it does $2B$ FLOPs for 2 bytes. In decode, the batch size is the intensity. §11.2 used this to plot the decode throughput of Llama 3 8B against the batch size. Here we build a model of a whole decode step that we can reason with, solve, and price.

First count the FLOPs of one decode step for $B$ sequences. Every weight on a token's path is used once for that token, so the weight matrices cost $2N_{\text{act}}$ FLOPs per sequence. Here $N_{\text{act}}$ is the number of parameters a token is actually multiplied by. For a dense model it is all $N$. A mixture of experts (MoE) splits each feed-forward layer into many small networks, called experts, and sends each token through only a few of them (§13.1). So an MoE's $N_{\text{act}}$ is far smaller than its $N$.

Attention adds its own FLOPs. The model has $L$ layers, each with $h$ query heads of width $d_k$, and the sequence has $T$ cached tokens. In each layer, each head scores its query against all $T$ cached keys, $2d_k$ FLOPs per key. It then takes a weighted sum of the $T$ cached values, $2d_k$ FLOPs per value. That is $4hd_kT$ FLOPs per layer. Write $d_{\text{attn}} = hd_k$ for the total width of the attention heads. A sequence then costs $2N_{\text{act}} + 4Ld_{\text{attn}}T$ FLOPs. For a batch whose contexts have mean length $\bar T$, the total is exactly $B(2N_{\text{act}} + 4Ld_{\text{attn}}\bar T)$, since the $B$ context lengths add up to $B\bar T$.

Then count the bytes. The weights are read once per step, whatever the batch. A dense model stored with $s_W$ bytes per weight reads $M_W = s_WN$ bytes. (An MoE reads only the experts that some token in the batch uses, as we will see.) Each sequence's cache is also read once, $m_{\text{KV}}$ bytes per cached token.

How big $m_{\text{KV}}$ is depends on how attention stores its keys and values (§12.5). Grouped-query attention lets several query heads share one key head and one value head, so each layer caches keys and values for only $n_{kv}$ heads. Latent attention caches one compressed vector of $d_c$ numbers per token per layer, plus a small key of $d_r$ numbers that carries the position. It rebuilds every head's keys and values from them. With $s_{\text{KV}}$ bytes per cached number, the cache per token is

In plain words, a grouped-query layer stores a key and a value for each of its $n_{kv}$ shared heads, and a latent layer stores one short vector plus a small position key. The whole cache of the batch is $M_{\text{KV}} = B\bar Tm_{\text{KV}}$.

In plain words, a decode step lasts as long as the slower of two jobs. One job is every sequence's arithmetic. The other is reading the weights once plus every sequence's cache once. On top of that come the time the GPUs spend talking to each other and the time they wait for the CPU. Every lever in this chapter shrinks one of these terms.

The prefill's attention term grows like $T_{\text{in}}^2$, but it matters only for very long prompts. It overtakes the weight term $2N_{\text{act}}T_{\text{in}}$ only when $T_{\text{in}} > N_{\text{act}}/(Ld_{\text{attn}})$. That is about 61,000 tokens for Llama 3.1 8B and 108,000 for the 70B. So for ordinary prompts, prefill costs about $2N_{\text{act}}$ FLOPs per token. The first output token is sampled from the prefill's last logits, so $\text{TTFT} = \text{queueing} + t_{\text{prefill}}$, with no decode step in it.

The cell writes the box as code, one line per term. It holds the GPUs of §11.2 and the three models we use all chapter. In each model, kv is the count of cached numbers per token, so $m_{\text{KV}}$ is kv * s_kv.

Read the three lines in order.

• One sequence with 4,096 tokens of context takes 4.95 ms per step, 202 tokens per second. Almost all of that time goes to reading 16 GB of weights.
• Sixty-four such sequences take 15.05 ms, three times longer, but produce 4,252 tokens per second, 21 times more. Now the caches (34 GB) outweigh the weights.
• With no context, 512 sequences reach the compute roof. At that batch, contexts beyond about 180 tokens would make the step memory-bound again.

A 2,000-token prompt needs about 67 ms of prefill, the time of 14 decode steps at batch 1.

def decode_step(cfg, hw, B, T, w_bytes=2, kv_bytes=2, n_gpu=1, overhead=0.0):
    t = w_bytes * cfg["n_params"] / hw["bw"]          # placeholder: weights only, one GPU
    return t, "weights"

The latency-throughput frontier

Every serving decision picks a point on one curve. Each batch size $B$ gives a step time. That step time gives a speed for each user, $1/t_{\text{step}}$, and a throughput for the operator, $B/t_{\text{step}}$. Plot one against the other as $B$ grows and you trace the latency-throughput frontier. Serving engines and benchmarks publish it as interactivity (tokens per second per user) against throughput (tokens per second per GPU). Its shape follows from the step-time model.

Fix a model, a GPU, and a context length. Then the box has four constants, which we name

so that $t_{\text{step}}(B) = \max(t_W + Bt_K,\; Bt_F) + t_0$. In plain words, $t_W$ is the shared cost of reading the weights, $t_K$ the cost of reading one sequence's cache, $t_F$ the cost of one sequence's arithmetic, and $t_0$ what every step pays regardless.

First, weights only

Start with short contexts. Then the caches and the attention FLOPs are negligible, so $t_K \approx 0$ and $t_F \approx 2N_{\text{act}}/(n_{\text{gpu}}\pi)$. The step costs $t_W$ until the batch's arithmetic catches up with the weight reading, and from then on it grows with every sequence.

In plain words, each weight byte read must be reused by enough sequences to pay for itself in arithmetic. One bfloat16 weight (2 bytes) does 2 FLOPs per sequence, which is 1 FLOP per byte per sequence. So it takes $I^\star$ sequences to reach the ridge. Below the knee, adding a user is free. Above it, every user slows everyone down and throughput stops growing.

Then, with the cache

Now give each sequence its cache. Each extra sequence adds $t_K$ to the memory side of the step and $t_F$ to the compute side, and whichever grows faster decides everything. Suppose a sequence's cache takes longer to read than its arithmetic takes to do, $t_K \ge t_F$. Then adding sequences can never make the step compute-bound, because the memory side stays ahead at every batch size.

In plain words, past the critical context, throughput creeps toward a ceiling set by bandwidth and context length alone, and a bigger batch only makes everyone slower. §11.2 found that long-context decode is memory-bound at any batch size. The theorem turns that finding into a throughput ceiling with a formula.

The cell sweeps the batch for Llama 3.1 8B on one H100 at three contexts and draws the frontier. The line is dotted where the batch no longer fits in 80 GB.

Read the curves from the top left. Each starts at batch 1, at 165 to 208 tokens per second per user, and slides right and down as the batch grows.

• At 512 tokens of context the curve runs almost to 40,000 tokens per second per GPU before memory runs out at 952 sequences.
• At 4,096 tokens it bends early. At the 119 sequences that fit it delivers 4,986 tokens per second, 80% of its ceiling of 6,240.
• At 32,768 tokens only 14 sequences fit. The GPU delivers 616 tokens per second, only 3.7 times what one user gets alone, while each of the 14 users now sees 44 tokens per second.

Long contexts are expensive twice. Each sequence's cache is slow to read, and few of them fit.

The widget draws the same frontier for one context at a time. The dot marks the batch you pick. The switch stores the cache in BF16, short for bfloat16, or in FP8, an 8-bit floating-point format with half the bytes.

The frontier is the first thing to draw in a serving design review. An interviewer will ask where the knee is for a given GPU and why long contexts flatten the curve. They will also ask which lever moves it, such as fewer cache bytes, more memory, or a smaller model.

How many users fit

Two limits cap the batch, and an operator must respect both. One is memory, and the other is the speed each user was promised.

Memory capacity. The weights and the caches must fit in the replica's memory. Take $n_{\text{gpu}}$ GPUs of HBM capacity $C_{\text{HBM}}$ each, and reserve $M_{\text{act}}$ bytes for activations and workspace. Then the largest batch that fits is

or 0 if the numerator is negative. Here $\bar T$ must be the context a sequence actually occupies. A server that reserves the maximum context for every sequence must use that maximum instead. Paged memory, which hands out cache space in small blocks as a sequence grows, removes that waste (§20.6). Engines also hold some memory back. vLLM, a widely used open-source serving engine, uses 90% of the GPU's memory by default.

Latency. Each user must get tokens at least as fast as a target, $t_{\text{step}}(B) \le \text{TPOT}_{\text{target}}$. Since the step time is a maximum of two lines in $B$, the target bounds $B$ through each line separately.

The cell confirms these numbers. Exercise 20.1c then asks you to handle every case exactly, including the ones where nothing fits.

def max_batch_under_slo(cfg, hw, T, tpot_target, hbm_bytes, w_bytes=2, kv_bytes=2, overhead=0.0):
    free = hbm_bytes - w_bytes * cfg["n_params"]
    return int(free // (T * kv_numbers_per_token(cfg) * kv_bytes)), "memory"   # placeholder: ignores the TPOT target

Traffic, queues, and the tail

So far we chose the batch. In a real server the batch is whatever traffic puts there, and requests arrive at random. The widget simulates one H100 serving Llama 3.1 8B, with every step's time from the model above. Requests arrive at random times and wait in a queue. Then they are prefilled. In this simulation the server pauses decoding to prefill each new arrival, as early engines did. Then the requests decode together.

Raise the arrival rate, press Play, and watch the queue form and TTFT grow. The bottom strip counts the requests in the system over time. The area under it over a window in the middle of the run, divided by the window's length, is the mean number in the system. While the server keeps up, that mean matches the measured arrival rate times the mean E2E, which is Little's law. When the server falls behind, the queue grows for as long as requests keep coming. The long-run averages the law needs then no longer exist, and the two numbers drift apart.

As the arrival rate nears what the GPU can serve, the queue stops draining between bursts. TTFT at p99 grows without bound. SLO attainment collapses, so fewer requests are served within the SLO, even though throughput is at its highest. Queueing theory makes this precise, and §20.7 uses it to decide how much spare capacity a fleet needs. Pausing decode for every prefill also produces the stalls that ITL catches, and §20.6 replaces that policy with one that mixes prefill and decode in each step.

Spreading one model over several GPUs

Llama 3.1 70B in bfloat16 is 141 GB of weights, more than one 80 GB GPU holds. So it is served with tensor parallelism, which splits each weight matrix over $n_{\text{tp}}$ GPUs (§10.4). Each GPU then holds and reads $1/n_{\text{tp}}$ of the weights, and $n_{\text{tp}}$ is called the tensor-parallel degree. For decode this is ideal on the memory side, since the step's memory time falls by $n_{\text{tp}}$.

The price is communication. With the split of §10.4, each GPU ends the attention block, and again the feed-forward block, holding only a partial sum of the output. The partial sums must be added across GPUs, which takes two all-reduces per layer in the forward pass. An all-reduce sums a vector held by every GPU and leaves the sum on all of them.

Each all-reduce sums a $B\times d$ activation, $Bds$ bytes with $s = 2$ in bfloat16, where $d$ is the model width. The standard way to do it is a ring all-reduce, which passes chunks around the GPUs in a loop. §10.2 priced a ring all-reduce of $M$ bytes over $p$ GPUs. It pays $2(p-1)$ message latencies, and each GPU sends $2(p-1)M/p$ bytes. Write $\alpha_{\text{link}}$ for the latency of one message and $\mathcal{B}_{\text{link}}$ for the link bandwidth in each direction, since $\beta$ is taken. The decode step on $n_{\text{tp}}$ GPUs then takes

where $F$ is the step's FLOPs. The cache term divides by $\min(n_{\text{tp}}, n_{kv})$ because one key-value head cannot be split across GPUs. Beyond $n_{kv}$ GPUs, each head is stored on more than one GPU (§12.5 worked this out for Llama 3 70B on 16 GPUs). A latent-attention cache is a single shared vector per token, so tensor parallelism would store all of it on every GPU. That is why SGLang, an open-source serving engine, runs DeepSeek's attention with data parallelism instead, each GPU serving its own sequences with its own copy of the attention weights.

For one token the activation is tiny, $8192\times2 = 16$ KB for the 70B model. The bandwidth term is then a fraction of a microsecond, and the latency term is everything. Write $n = n_{\text{tp}}$ and $a = M_W/\beta$, the time one GPU would take to read all the weights. At batch 1 the step is about $a/n + 4L\alpha_{\text{link}}(n-1)$. Its derivative in $n$, $-a/n^2 + 4L\alpha_{\text{link}}$, vanishes at

and past $n^\star$ more GPUs make each token slower. In plain words, splitting the weights cuts the reading time like $1/n$, but every layer pays two all-reduces whose latency grows like $n$, so per-token latency has a floor.

Take the 70B model on H100s. The GPUs of one machine talk over NVLink, NVIDIA's direct links between them, at 450 GB/s per direction. Take $\alpha_{\text{link}} = 2$ µs per ring step, an illustrative value, so measure your own. Then $n^\star = \sqrt{141.2\times10^9/(4\times80\times2\times10^{-6}\times3.35\times10^{12})} \approx 8.1$. The cell tabulates the latency for three link latencies.

The speed of light of a deployment is the fastest it could possibly go, the time to read the weights once at full bandwidth. For the 70B model on eight H100s that is $141.2/(8\times3.35) = 5.27$ ms per token, 190 tokens per second. But 160 all-reduces of 14 ring steps each at 2 µs add 4.5 ms, so the model predicts 9.76 ms. With 5 µs links, eight GPUs are already slower than four.

This is why serving engines such as vLLM and NVIDIA's TensorRT-LLM ship their own all-reduce kernels for small messages. It is also why Pope et al. (2022) spent much of their paper on how to split a model over chips. They reached 29 ms per token for PaLM 540B with 8-bit weights on 64 TPU v4 chips. An interviewer who asks for batch-1 latency on eight GPUs wants to hear that the all-reduce latency, not the bandwidth, sets the floor.

def tp_decode_latency(cfg, hw, n_tp_list, alpha_link, link_bw, B=1, T=0, w_bytes=2, kv_bytes=2, act_bytes=2):
    lat = [w_bytes * cfg["n_params"] / (n * hw["bw"]) for n in n_tp_list]   # placeholder: no communication
    return lat, n_tp_list[lat.index(min(lat))]

Mixtures of experts spread differently

An MoE is usually served with expert parallelism, which places different experts on different GPUs (§13.4). Its weight bytes per step then depend on the batch, because a step reads only the experts some token chose. Suppose each token picks $k$ of the $E$ experts in a layer. If routing is uniform and independent across tokens, a decode step of $B$ tokens touches on average $E(1 - (1 - k/E)^B)$ of them (§13.6). Skewed routing touches fewer. Either way, $M_W$ itself grows with the batch.

• At batch 1 an MoE reads only its active weights, so it is fast.
• At moderate batches it reads almost all of its weights but does the arithmetic of only the active ones, the worst of both.
• Only at very large batches does each expert see enough tokens to pay for its bytes.

If every expert is read every step, the knee of the frontier moves from $s_WI^\star/2$ to $s_WI^\star N/(2N_{\text{act}})$. The bytes are those of all $N$ weights, but the arithmetic is that of $N_{\text{act}}$. For DeepSeek-V3 ($N = 671$ billion, $N_{\text{act}} = 37$ billion) on H100s in bfloat16 that is about 5,400 sequences. DeepSeek's answer is to spread decode over units of 18 nodes, 144 GPUs that each hold two routed experts and one shared expert. Each expert's matrices then serve a large batch gathered from many GPUs.

From seconds to dollars

Now we can answer the opening question about price. A replica of $n_{\text{gpu}}$ GPUs rented at $\kappa$ dollars per GPU-hour costs $\kappa n_{\text{gpu}}/3600$ dollars per second, busy or not. Dividing that by the tokens it produces per second gives the cost of a token.

In plain words, a token costs the price of the GPUs for the time it takes to make it. An output token costs more than an input token by exactly the factor by which prefill outruns decode.

Take the 70B node above at $2 per GPU-hour, so $16 an hour for the node.

• The roofline throughput of 7,774 tokens per second gives $0.57 per million output tokens, and the 6,133 with all-reduces gives $0.72.
• Prefill of 2,000-token prompts at 50% MFU runs at about 27,500 tokens per second per node, which is $0.16 per million input tokens.

Output costs about 3.5 times as much as input with the roofline throughput, and 4.5 times with the all-reduces. The tensor-parallel cell prints both. That ratio is no accident, and we can derive it.

In plain words, an output token is dearer for two reasons. The weights it reads are shared by only $B$ tokens, while in prefill they are shared by a whole prompt. And it must read its own cache, which a prefill token never does.

Two examples show the sizes. Llama 3.1 8B on an H100 at batch 64, with 4,096 tokens of context and 50% prefill MFU, gives $r = 0.5\times(295/64 + 9.87) = 7.2$. The 70B node above, at batch 185 and 8,192 tokens, gives $r = 0.5\times(1.60 + 5.61) = 3.6$. The tensor-parallel cell found 3.5 with the prefill's attention included.

The second term is the critical context in disguise. For a grouped-query model $I^\star m_{\text{KV}}/(2N) \approx 1/\bar T_c$, so $r \approx \text{MFU}\,(B^\star/B + \bar T/\bar T_c)$. The critical context is a few hundred tokens. Once contexts are much longer than that, output tokens cost several times input tokens however well the server batches. So the factor of 4 on price lists is not a pricing whim. It follows from the time spent reading caches.

def cost_per_million(kappa, n_gpu, prefill_tps, decode_tps, cache_hit=0.0, hit_cost=0.0):
    return 1e6 * kappa / prefill_tps, 1e6 * kappa / decode_tps      # placeholder: per hour, one GPU

An interviewer who asks why output tokens are dearer wants this derivation, and a candidate who can check it against DeepSeek's day of numbers stands out.

def prefill_decode_price_ratio(cfg, hw, T_in, mfu_prefill, B_decode, T_ctx, w_bytes=2, kv_bytes=2):
    return 4.0          # placeholder: the number on the price list

Where the model breaks

The step-time model is a lower bound with a few honest adjustments. Each of its assumptions fails somewhere in practice, and knowing where is what makes an estimate trustworthy.

Overhead. For small models at small batch, the GPU can finish a step faster than the CPU can prepare the next one. A decode step is hundreds of kernels, and §11.7 showed that 300 launches at 6 µs each take 1.8 ms of CPU time. That hides behind an 8B model's 4.8 ms of kernels only while the CPU stays ahead. It shows at every step boundary, where the CPU waits for the sampled token, and whenever kernels are shorter than a launch. Engines therefore record the whole decode step once as a CUDA graph, which the GPU then replays with a single launch. They keep one graph per batch-size bucket. They also prepare the next step's batch on the CPU while the GPU runs the current one. Sampling, turning token ids back into text, and streaming the token over the network add to $t_{\text{ovh}}$ too.

Imperfect kernels. Real decode kernels reach perhaps 70 to 90% of peak bandwidth (an MBU below 1), and prefill reaches an MFU of roughly 40 to 60%. Divide the model's times by the efficiency you measure, MBU for decode and MFU for prefill. Always label a model number as a bound.

The spread of contexts. The total cache bytes depend only on $\bar T$, so the model's memory term is exact. Real kernels, though, can be badly unbalanced. In a batch that mixes one 100,000-token sequence with fifty short ones, one group of GPU threads gets far more work than the rest. The fix is a kernel that splits each long cache into pieces, processes them in parallel, and merges the partial results. That kernel is Flash-Decoding (§12.4). And memory must hold the actual lengths at their peak, not their average over a day.

The tail. The model describes a steady batch. Real arrivals are bursty, prefills interrupt decodes, and queues form. A server can meet its mean TPOT and miss its p99 badly. That is why SLOs are written on percentiles, and why goodput, not throughput, is the number to optimize.

Mixtures of experts. The weight bytes depend on the batch through the experts touched, as shown above. Imbalance among experts also makes the busiest GPU set the step time.

Long outputs. A sequence that starts with $T_{\text{in}}$ tokens and writes $T_{\text{out}}$ holds about $T_{\text{in}} + T_{\text{out}}/2$ tokens on average during its decode. Reasoning models write thousands of tokens per answer (§18.7). That moves a provider's cost toward decode and toward the cache term, which batching cannot remove. Methods that cap $\bar T$ attack exactly that term. One is sliding-window attention, which keeps only recent tokens plus a few early ones (§12.6). Another is recurrent models with a fixed-size state, whose memory does not grow with the context at all.

Attention FLOPs at very long context. Prefill's attention term passes the weight term at about 61,000 tokens for an 8B model. Beyond that, a prompt's cost grows quadratically with its length. And latent attention's high intensity can make even decode attention compute-bound.

The table lists the levers on the cost of a token, the term of the step-time model each one attacks, and the section that teaches it.

The same few equations, applied to whole fleets, are the core of back-of-envelope estimates for LLM systems. The latency budgets of voice assistants also start from TTFT and TPOT.

In PyTorch

You can watch the knee on your own CPU. The block times one linear layer of width 4,096 for growing batches. Its 64 MB of float32 weights do not fit in most CPU caches, so they stream from main memory on every call.

• At small batch the time is set by streaming the weights and grows slowly, so rows per second rise almost in proportion to the batch.
• Past some batch the time grows in proportion to the batch, and rows per second level off.

Those are the two branches of the step-time model. Where the knee falls depends on your machine's ratio of FLOP/s to memory bandwidth. The smallest batches can be erratic, because math libraries switch kernels at small sizes. On the laptop we used, the time grew only from about 6 to 8 ms between batches 16 and 128. After that it roughly doubled with each doubling of the batch.

On a server you measure the real thing from the client side, which is how users experience it. The block below (shown, not run) streams a completion from a server and records when each chunk of text arrives. It uses the OpenAI-style web interface that vLLM and SGLang both offer. One caveat makes this a measurement of chunks rather than tokens. A server may pack several tokens into one streamed chunk. So count tokens with the tokenizer, or ask the server for its usage statistics, before dividing.

Read more

• Pope et al. (2022). Efficiently Scaling Transformer Inference. The analytical model of inference cost, partitioning for low latency and high throughput, and the latency-MFU frontier for PaLM 540B.
• Austin et al. (2025). All About Transformer Inference. Chapter 7 of How to Scale Your Model, with the critical batch size, the step-time bounds, and worked problems that this section's model formalizes.
• Zhong et al. (2024). DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving. Defines per-GPU goodput under TTFT and TPOT targets and shows why it, not throughput, should be optimized.
• DeepSeek-AI (2025). Day 6: One More Thing, DeepSeek-V3/R1 Inference System Overview. A day of real production statistics, node counts, throughputs, cache hit rates, and cost, the rarest kind of data in this field.
• SGLang Team (2025). Deploying DeepSeek with PD Disaggregation and Large-Scale Expert Parallelism on 96 H100 GPUs. An open reproduction of DeepSeek's serving system with per-node throughputs and cost.
• Databricks (2023). LLM Inference Performance Engineering: Best Practices. TTFT, TPOT, and model bandwidth utilization defined and measured on real models across tensor-parallel degrees.
• Yuan et al. (2024). LLM Inference Unveiled: Survey and Roofline Model Insights. A survey of inference techniques organized by the roofline term each one attacks.
• Little (1961). A Proof for the Queuing Formula: L = λW. The original proof of Little's law for stationary systems, in Operations Research 9(3). Read it with Stidham (1974), A Last Word on L = λW, the sample-path proof, which needs only that the arrival rate and the mean time inside exist and are finite, in Operations Research 22(2).

Watch next

▶

▶