Serving a model means running it for many users at once, on GPUs you rent by the hour. This section turns the ideas of the whole chapter into a working service and a bill. It shows how one program keeps a GPU busy with hundreds of requests, how many GPUs a product needs, how to split the model over them, and what each million tokens costs. A team must answer these questions before it buys hardware, and they come up in every ML systems interview.

Here is the concrete question the section answers. You must serve a 70B chat model to a product that peaks at 200 requests per second. Prompts average 2,000 tokens and answers 400. The product team wants the first token within a second and every later token within 50 ms. How many GPUs do you need, of which kind, split how, and what will a million tokens cost? And what should the fleet do at night, when traffic falls to a fifth of the peak?

We build the answer in three layers. First comes the engine, the program that runs one copy of the model and serves many requests with it. One copy of the model, on however many GPUs it needs, is called a replica. Then comes the fleet, many replicas behind a router that sends each request somewhere. Last comes the plan, a number of GPUs and a price. Along the way we meet the parts of serving that are neither arithmetic nor memory traffic. These are the CPU work around each step, output that must follow a format, many fine-tuned versions of one model, queues, and machines that take minutes to start.

What the chapter has built so far

Every number in this section comes from one model of time, built in §20.1. Here it is again in brief. A request goes through two phases. Prefill runs the whole prompt through the model in one pass and produces the first output token. Decode then produces one token per step. Each decode step reads the KV cache, the keys and values that every earlier token stored so that attention never recomputes them (§6.5). A server decodes many sequences together, and $B$ is the number of sequences in one decode step.

Two latencies matter to a user. Time to first token (TTFT) is the time from a request's arrival to its first output token, queueing and prefill included. Time per output token (TPOT) is the mean gap between the later tokens. The opening question asks for a TTFT of at most 1 s and a TPOT of at most 50 ms. Targets like these are called service-level objectives (SLOs).

A decode step does two jobs at once. It does arithmetic, and it reads memory. It lasts as long as the slower of the two. Take Llama 3.1 8B on one H100 GPU, stored in bfloat16, the 2-byte number format in which models are usually served. Reading its 16 GB of weights takes 4.8 ms. The arithmetic for one token, 16 billion floating-point operations (FLOPs), takes 16 µs. So at small batches the step is set by reading, and each extra sequence in the step is almost free. For one replica on $n_{\text{gpu}}$ GPUs the general model is

Here $\pi$ is a GPU's peak arithmetic rate in FLOP/s and $\beta$ the bandwidth of its main memory, called HBM (high-bandwidth memory), in bytes per second. Other chapters use $\pi$ for a policy and $\beta$ for a KL coefficient, but here both are hardware rates. Their ratio $I^\star = \pi/\beta$ is the ridge point of the roofline, the model that bounds a kernel's time by its arithmetic or by its memory traffic (§11.2). For an H100 in bfloat16, $\pi = 989$ TFLOP/s, $\beta = 3.35$ TB/s, and $I^\star \approx 295$ FLOPs per byte.

The model being served has $N$ parameters, of which $N_{\text{act}}$ multiply each token ($N_{\text{act}} = N$ for a dense model). It has $L$ layers and attention width $d_{\text{attn}}$. Its weights take $M_W = s_WN$ bytes at $s_W$ bytes per weight, and its cache takes $m_{\text{KV}}$ bytes per token. The sequences have mean context $\bar T$. The last two terms are the time the GPUs spend talking to each other and the fixed overhead of a step.

In plain words, a step lasts as long as the arithmetic of every sequence, or as long as reading the weights once plus every cache once, whichever is longer. Throughput is $\Theta = B/t_{\text{step}}$ tokens per second, and every user gets one token per step.

The middle sections attacked the terms of that formula. §20.2 and §20.3 shrank the bytes per number. §20.4 shrank the number of weights. §20.5 made each weight read yield several tokens. §20.6 packed more sequences into each step, with continuous batching, a paged cache, prefix caching, chunked prefill, and separate prefill and decode GPUs. Each of these is recalled in a sentence where we use it.

An engine is a loop

From the outside an engine is a web endpoint. It accepts a chat request and streams tokens back. Inside it is one loop that runs as fast as the GPU allows. Each pass of the loop is a step. A step decides which requests get how many tokens of work, and runs one forward pass over all of those tokens at once. It then picks one new token for every request that needs one, and cleans up. Everything an engine does well, it does by keeping that loop full and never letting the GPU wait.

In plain words, each step picks the work, does it in one forward pass, picks the tokens, and tidies up. The engine is the scheduler's decisions plus the machinery that carries them out fast.

One number per request is enough

The scheduler's decision is only a list of numbers $n_i$, one per request. That list can express every mechanism of the last two sections.

• A request still reading its prompt gets a chunk of it, $n_i > 1$. This is chunked prefill, which splits a long prompt over several steps so that it never stalls the others (§20.6).
• A request whose first $T_{\text{hit}}$ prompt tokens are already in the prefix cache, the store of caches left by earlier requests with the same opening, starts with those tokens done. Its chunks cover only the rest.
• A decoding request gets $n_i = 1$.
• With speculative decoding, a small drafter guesses $\gamma$ tokens and the model checks them all in one pass (§20.5). The request then gets $n_i = \gamma + 1$.

vLLM's 2025 rewrite of its engine, called V1, made exactly this its scheduling interface. Each step's plan is a dictionary from request to number of tokens, with no separate prefill steps and decode steps (vLLM Team 2025). The usual policy is the stall-free schedule of §20.6 (Agrawal et al. 2024). It gives every running decode its token first and fills the rest of the budget with prompt chunks in arrival order.

The cell writes this step as code. A step is a list of (new_tokens, cached_tokens) pairs, one per scheduled request. Its time is the step-time model with the work and bytes of every pair added up. Each new token of a chunk attends to the cache plus, on average, half the chunk before it.

The next two cells turn the step into a whole engine. serve admits waiting requests in arrival order while there is a free sequence slot and room in the cache for the whole prompt and answer. It schedules decodes first and prompt chunks after them. It charges each step its GPU time plus a CPU time, and records when every token is streamed. The CPU time stands for scheduling, building inputs, sampling, and turning tokens into text. We model it as 1 ms plus 20 µs per scheduled request, a plausible figure that you should measure for your own engine. The trace is 120 requests arriving at random, 40 per second on average.

The CPU is part of the step

The two runs differ in one line. If the CPU does its work after the GPU finishes, a step takes $t_{\text{gpu}} + t_{\text{cpu}}$. If the scheduler prepares step $k+1$ while the GPU runs step $k$, a step takes $\max(t_{\text{gpu}}, t_{\text{cpu}})$. Overlapping cuts the median TPOT from 10.3 to 7.9 ms. It also halves the median TTFT, from 86 to 44 ms, because the same work fits into fewer seconds and the queue drains faster. When a step lasts about 5 ms, a millisecond or two of CPU work is a large fraction of it.

This is a real problem in production. The vLLM team wrote that the CPU work of "running the API server, scheduling work, preparing inputs, de-tokenizing outputs, and streaming responses" becomes "increasingly pronounced" for "smaller models like Llama-8B running on NVIDIA H100 GPUs", where a step takes about 5 ms on the GPU. Its V1 engine, rebuilt largely to cut that overhead, reported up to 1.7 times the throughput of the old one (vLLM Team 2025). The SGLang team wrote that an unoptimized engine "can spend as much as half of its time on CPU overhead" (SGLang Team 2024). Engines use five fixes.

• Separate processes. The HTTP server, the tokenizer, and the detokenizer run in processes apart from the engine core, which only schedules and executes. Text work for one request then never delays the step of another.
• Overlap scheduling. The scheduler runs one step ahead. Step $k+1$ needs the tokens that step $k$ will sample, so SGLang plans it with placeholder "future tokens" that the GPU fills in once they exist. Its profiles showed no idle time on the GPU between consecutive decode steps (SGLang Team 2024).
• Persistent batches. Rather than rebuild every input tensor each step, vLLM V1 keeps them and applies only the changes (requests added, finished, or advanced).
• CUDA graphs. A decode step launches hundreds of small kernels, the programs the GPU runs, and launching them one by one can take longer than running them (§11.7). A CUDA graph records the whole sequence of launches once and replays it with one call. Engines record a graph for each batch size in a set of buckets and pad each step up to the next bucket. Attention's inputs change shape with every batch, so vLLM V1 first recorded only the pieces of the step between attention calls ("piecewise" graphs) and ran attention outside them. Its default now also records full graphs, attention included, for batches of pure decodes (vLLM docs).
• Incremental detokenization. Each new token becomes text as soon as it can. A token that ends in the middle of a UTF-8 character is held back until the rest arrives (§2.3). Stop strings are checked on the text, not on the ids.

In an interview, "where does the time go in a decode step of a small model?" has those two answers. The CPU is the one candidates forget.

Three engines and the layer above them

Most open serving runs on one of three engines. They share the loop above and differ in emphasis.

vLLM began as the PagedAttention paper, which stored each request's cache in fixed-size blocks like pages of virtual memory (Kwon et al. 2023). Its V1 engine is built around the token-count scheduler. It keeps a pool of 16-token cache blocks in a free queue. It finds shared prefixes by hashing blocks, which it measured as cheap enough to turn on by default, with "less than 1% decrease in throughput even when the cache hit rate is 0%". When memory runs out it evicts a request and later recomputes its cache (vLLM Team 2025; Gordić 2025). It supports the widest range of models and hardware.

SGLang began as a system for programs that call a model many times (Zheng et al. 2024). It keeps every request's cache in a radix tree, a tree of token sequences in which shared openings are stored once, and calls this RadixAttention. It evicts the least recently used (LRU) leaves first and orders requests to maximize reuse (§20.6). It introduced compressed automata that speed up structured output (described below) and the overlap scheduler. Its data-parallel attention was built to serve DeepSeek's latent-attention models at scale.

TensorRT-LLM is NVIDIA's engine. It has in-flight batching, its name for continuous batching, where requests join and leave the batch at every step. It also has a paged cache with block reuse, speculative decoding, and kernels tuned for FP8 and FP4 on NVIDIA's own GPUs. It now runs on a PyTorch-based architecture. The original backend, which compiled each model into a TensorRT engine, has been removed (NVIDIA).

The table is a snapshot of late 2026. All three move fast and borrow each other's best ideas within months. Choose by measuring your own traffic, which is what the rest of this section teaches you to model.

Above the engines sits an orchestration layer that treats many replicas as one service. NVIDIA Dynamo adds a planner that adjusts GPU resources to keep meeting the latency targets, and a router that knows which replica holds which cache blocks. It also adds the NIXL library, which moves caches between GPUs, and a manager that spills caches to cheaper memory. It runs SGLang, TensorRT-LLM, or vLLM underneath. llm-d is a similar open stack built on Kubernetes, the standard system for running containers on a cluster, and it is now at version 1.0. It routes by prefix cache, separates prefill from decode, spreads experts over many GPUs, and autoscales, running vLLM or SGLang.

Choosing a parallel layout for serving

A model that fits on one GPU is served by copying it, one replica per GPU, behind a router. This is data parallelism for inference, and it needs no communication at all. A model that does not fit must be split. Tensor parallelism splits every weight matrix over $n_{\text{tp}}$ GPUs, so each GPU holds and reads $1/n_{\text{tp}}$ of the weights (§10.4). The price is two all-reduces per layer. An all-reduce is a collective operation that sums a vector across all the GPUs and gives every GPU the result.

The reading time shrinks like $1/n_{\text{tp}}$, while the latency of each all-reduce grows with $n_{\text{tp}}$. So adding GPUs lowers the latency of one token only up to about

GPUs, where $\alpha_{\text{link}}$ is the latency of one message between GPUs (§20.1). For Llama 3.1 70B in bfloat16 on H100s, with 2 µs per message, $n^\star \approx 8$. That answers "how fast can one user go". The operator asks a different question. Which layout gives the most tokens per GPU while every user still gets a token every 50 ms?

A decode replica serving real traffic usually runs out of memory before its steps get too slow for the TPOT target. Each GPU then holds its share of the weights and fills the rest of its memory with caches. A short calculation shows what that buys.

In plain words, a GPU full of cache takes as long to decode one step as it takes to read its own memory once. For an H100 at 90% of 80 GB that is about 21.5 ms, whatever the model. Its throughput is the most that cache reading allows, $\beta/(\bar Tm_{\text{KV}})$, times the share of memory that holds caches rather than weights. Spreading the weights over more GPUs raises that share, and the all-reduces take some of it back.

The cell adds the all-reduces of §20.1, with 2 µs per ring step and 450 GB/s links (illustrative values), and the 50 ms target. It finds the best layout on H100s and on B200s, NVIDIA's next generation, with 180 GB of HBM and 8 TB/s each.

On H100s the cheapest layout is TP4, at about 2,890 tokens per second per GPU. TP8 gives the lowest latency at batch 1 (7.2 ms per token) but 9% less throughput per GPU than TP4, because its all-reduces take back more than its extra cache room gives. On B200s TP2 is best. The 50 ms target never binds, because a full GPU takes only 20 to 33 ms per step. For this workload memory, not latency, sizes every replica.

An interviewer who asks "TP8 or two TP4 replicas?" wants exactly this trade.

Two other layouts appear in serving. Pipeline parallelism puts consecutive layers on different GPUs (§10.5). It splits the memory but not the latency of a token, since the token still passes through every stage in turn. Engines use it mainly to span machines when a model does not fit in one. Llama 3.1 405B in bfloat16, for example, needs 810 GB against the 640 GB of an eight-H100 node. A mixture-of-experts model replaces each feed-forward layer by many expert networks, of which each token uses a few. Expert parallelism spreads those experts over many GPUs (§13.4). Running prefill and decode on separate GPUs lets each phase choose its own layout.

Structured output inside the engine

Many applications parse the model's answer, as JSON or a function call, and they need a guarantee that it parses. The mechanism is a mask. At each step the engine works out which tokens could still lead to valid output. It sets the logit of every other token to $-\infty$, where the logit is the score the model gives a token before the softmax. A token with logit $-\infty$ has probability exactly zero, so it is never sampled.

The format is written as an automaton, a machine that reads the output one character at a time and keeps a state $\sigma$ summarizing what it has read. A token is allowed in state $\sigma$ if its characters lead from $\sigma$ to a state from which the format can still be completed. The allowed set $\mathcal{A}(\sigma)$ holds those tokens. It also holds the end-of-sequence token when $\sigma$ is an accepting state, one in which the output may stop.

The cell checks the four sets by running every token through the automaton. It also packs each set into one 32-bit integer, with bit $j$ set when token $j$ is allowed, which is how engines send masks to the GPU.

A real format is larger, and its allowed sets must be ready at every step. When a format is described by a regular expression, its automaton has finitely many states and is called a DFA (deterministic finite automaton). Then $\mathcal{A}(\sigma)$ can be computed for every state once, before serving starts. That precomputed table is the index of the Outlines library (Willard and Louf 2023).

JSON can nest to any depth, so a JSON grammar needs a pushdown automaton, a state machine with a stack that remembers the open brackets. No finite table covers every possible stack. XGrammar's answer is to split the vocabulary in two (Dong et al. 2024). Most tokens can be judged from the current position in the grammar alone, and their verdicts are precomputed. The rest depend on the whole stack and are checked at serving time. For Llama 3.1's 128,256 tokens and a JSON grammar, only 1,134 tokens fall in the second group. A further step cuts them to 120 by rejecting, ahead of time, the tokens that no rule enclosing the current one could ever accept next.

§24.2 builds the index in full, proves that the masked decoder never gets stuck and only emits valid strings, and covers schemas and tool calls. It writes $q$ and $\operatorname{allow}(q)$ for what we write $\sigma$ and $\mathcal{A}(\sigma)$. Here we look at what the engine does with the mask.

For Llama 3.1's 128,256 tokens a row is 4,008 words, about 16 KB. Even a batch of 256 constrained requests sends only 4 MB to the GPU per step. The arithmetic is not the problem. The problem is producing the masks in time. vLLM (Gordić 2025) and XGrammar arrange the work in four steps.

1. A new request's grammar is compiled in the background, and the request waits until it is ready. Compiled grammars are cached by schema, because compiling a complex schema can take seconds and a product reuses a few schemas millions of times.
2. After scheduling, the CPU fills the bitmask rows of the constrained requests from their automata's current states. It does this while the GPU runs the forward pass. That works because the mask for step $k$ depends only on tokens sampled before step $k$, so neither job needs the other's output until sampling (Dong et al. 2024).
3. After the forward pass, the GPU expands the bitmask and masks the logits. Only then does it apply temperature, top-p, and the rest, so that top-p measures probability among allowed tokens (§6.4).
4. After sampling, each automaton advances by its request's new token.

The CPU must finish all of a step's masks within the GPU's step. XGrammar reports under 40 µs per token for JSON schemas. So 64 constrained requests need about 2.6 ms of CPU per step on one core, against a decode step of 5 to 30 ms, which is why engines fill masks on several threads. Speculative decoding complicates step 2. Each drafted position needs its own mask, computed by advancing a copy of the automaton along the draft and discarding the copy on rejection.

Two interview questions follow from this pipeline. Why must top-p come after the mask? Because top-p applied first can keep only tokens the mask then removes, leaving nothing to sample. Why is the mask cheap on the GPU but not on the CPU? Because applying it is one pass over the logits, while computing it walks every constrained request's automaton.

import numpy as np

def pack_bitmask(allowed, V):
    raise NotImplementedError

def apply_bitmask(logits, bitmask):
    raise NotImplementedError

Jump-forward decoding

Formats are full of text the model has no choice about. In the JSON object {"name": "..."}, every character before the name is fixed. A masked decoder still spends one forward pass per token of such forced text, and each pass produces a distribution with only one allowed continuation.

SGLang compresses those runs (Zheng et al. 2024). Call a state of the character automaton forced if it is not accepting and has exactly one outgoing character. Start from any state and follow forced states, collecting their characters. The result is the forced string of that state, the text that every valid continuation must start with. The engine appends the forced string's tokens to the sequence in one go and runs them through the model as a small prefill chunk. The chunk's last position gives the logits for the next real choice.

A run of $k$ forced tokens then costs one forward pass instead of $k$. While decode is memory-bound, a pass over $k + 1$ tokens costs about the same as a pass over one (§20.1). The SGLang paper measured 1.6 times the throughput on a JSON decoding benchmark from this alone. It also found that rebuilding the compressed automaton for every request, instead of once for a batch of requests that share it, made throughput 2.4 times lower.

There is one catch, and it comes from the tokenizer (§2.6). The forced text must be turned into tokens, and cutting it at the boundary with the text already generated can produce a split the model never saw in training. Suppose the model has written " and the forced text is }. The usual tokenization of the whole may be the single token "}. So SGLang retokenizes the previous text together with the forced string. The tokens that change at the boundary are replaced, which means their cache entries are dropped and recomputed in the same chunk.

Jump-forward therefore emits exactly the characters that masked decoding would have been forced to emit, in fewer passes, but not always the same tokens. Masked decoding lets the model spell forced text with any allowed tokens. Retokenization spells it the usual way, which is usually closer to what the model saw in training. Neither one samples the model's own distribution over valid strings, as the next part shows.

def forced_string(dfa, q):
    return "", q            # placeholder: never jumps

def jump_forward(tokens, forced, vocab):
    return len(tokens), tokenize(forced, vocab)    # placeholder: tokenizes the forced text alone

Masking keeps the format, not the model's distribution

A mask guarantees that the output is valid. It does not guarantee that valid outputs come with the probabilities the model gives them. Masking each step and renormalizing samples every token from the model's next-token distribution restricted to $\mathcal{A}(\sigma)$. The product of those per-step choices is not, in general, the model's distribution conditioned on producing a valid string. The mask at each step cannot see whether the model was about to wander out of the format a few tokens later.

The cell computes both distributions exactly and checks the masked one by sampling.

§24.2 proves this distortion in general and implements grammar-aligned sampling (Park et al. 2024), which removes it at the cost of extra samples. For a serving team the distortion has two practical consequences. First, a format can lower answer quality. One study found that format restrictions lowered accuracy on reasoning tasks (Tam et al. 2024). A rerun with consistent prompts found no loss (Kurt 2024), so measure it on your own task and prompt. Second, the order of fields in a schema matters. A model forced to write its answer field before its reasoning field has no room to reason first.

Serving many fine-tuned models at once

Suppose a thousand customers each fine-tuned the same base model with LoRA. LoRA leaves the base weights frozen and learns a small adapter for each weight matrix, a product of two thin matrices (§14.5). Adding the adapter into the base weights, called merging, gives a model with no added latency. But a thousand merged copies of a 70B model in bfloat16 is 141 TB of weights. The alternative is to keep one base in memory with the small adapters beside it, and to batch requests for different adapters together through the shared base.

Here is the adapter in symbols. This chapter writes a weight as $\mathbf{W} \in \R^{d_{\text{in}}\times d_{\text{out}}}$, so a row of inputs multiplies it from the left. An adapter of rank $r$ is the update $\Delta\mathbf{W} = \gamma\,\mathbf{A}\mathbf{B}$, with $\mathbf{A} \in \R^{d_{\text{in}}\times r}$ and $\mathbf{B} \in \R^{r\times d_{\text{out}}}$. The scale is $\gamma = \alpha/r$ for a hyperparameter $\alpha$. §14.5 writes weights the other way round, as $d_{\text{out}}\times d_{\text{in}}$ matrices that multiply a column from the right. Its $\mathbf{A}$ and $\mathbf{B}$ are the transposes of ours, so its update $\gamma\mathbf{B}\mathbf{A}$ is the transpose of our $\gamma\mathbf{A}\mathbf{B}$. In this part $\gamma$ is the LoRA scale, not the draft length of speculative decoding, and the bold $\mathbf{B}$ is a matrix, not the batch size $B$.

The two products are kept in the order $(\mathbf{x}_i\mathbf{A})\mathbf{B}$, so the $d_{\text{in}}\times d_{\text{out}}$ update is never formed. Punica introduced the segmented kernel. It reported 12 times the throughput of the serving systems of its day on multi-LoRA traffic, "while only adding 2ms latency per token" (Chen et al. 2023).

S-LoRA noticed that adapters behave like caches (Sheng et al. 2024). Both vary in size and come and go with requests. So it stores both in one pool of pages, each page one hidden vector wide. A rank-$r$ adapter then occupies $r$ pages, and a sequence of $T$ tokens occupies $T$ pages. S-LoRA keeps all adapters in CPU memory and loads each one when its requests are scheduled, and it added kernels for adapters of different ranks in one batch. It served 2,000 adapters at once with little overhead. Its throughput was up to 4 times that of vLLM with naive LoRA support and up to 30 times that of the Hugging Face PEFT library.

What does an adapter cost the step? Count it the way §20.1 counted everything else, in FLOPs and in bytes read.

The cell puts real numbers in. It takes a rank-16 adapter on all seven linear layers of every block of Llama 3.1 8B (the query, key, value, and output projections, and the three feed-forward matrices), and a decode step of 64 requests at 2,048 tokens of context.

The adapter is 84 MB and adds 0.6% to the arithmetic, so the FLOPs never matter. The bytes can. A batch of 64 requests that all use one adapter decodes in 9.95 ms instead of 9.92. If they use 64 different adapters, the step reads 5.4 GB more and takes 11.52 ms. That is 16% longer for everyone in the step, including requests that use no adapter at all.

This is why engines cap the number of distinct adapters per step (vLLM's max_loras), and why routers send an adapter's traffic to the replicas that already hold it. It is also why one tenant with a rare adapter slows its neighbors. The interview question "is LoRA serving free?" has the answer "free in FLOPs, not in bytes".

import numpy as np

def multi_lora_forward(X, W, A, B, scale, adapter_idx):
    return X @ W            # placeholder: base model only

Queues and headroom

A replica that is 95% busy sounds efficient. Its users disagree. Requests arrive at random, so sometimes several arrive close together. A server that is almost always busy has almost no spare capacity to absorb such a burst, so each burst leaves a queue that drains only slowly. Queueing theory turns this into formulas, and the simplest one already explains why fleets are run with headroom.

Start with numbers. A replica can serve 10 requests per second, so each takes 0.1 s on average, and 8 arrive per second. It is busy 80% of the time. The formulas below say that a request then waits 0.4 s on average before service starts, four times its own service time. At 9.5 arrivals per second the wait is 1.9 s, nineteen service times. The last few percent of load cost more delay than all the rest.

The model behind those numbers is the M/M/1 queue. The name lists the arrival process, the service times, and the number of servers, and M stands for memoryless.

• Requests arrive as a Poisson process of rate $\lambda_{\text{req}}$. The gaps between arrivals are independent and exponential with mean $1/\lambda_{\text{req}}$.
• One server handles requests one at a time in arrival order, called first-come first-served. Service times are independent and exponential with rate $\mu$, so a request takes $1/\mu$ seconds on average.
• The utilization $\rho = \lambda_{\text{req}}/\mu$ is the fraction of time the server is busy.

Write $W_q$ for the mean time a request waits before service starts and $W_{\text{sys}}$ for its mean time in the system, waiting plus service. In a serving system, the wait is the queueing part of TTFT.

In plain words, the mean wait is the service time multiplied by $\rho/(1-\rho)$, a factor that explodes as the load nears 1.

A prefill's time is set by its prompt length, so prefill service times vary only as much as prompt lengths do, often far less than an exponential. The M/G/1 queue allows any distribution of service times (G for general), and one formula covers it.

In plain words, the wait grows with the second moment of the service time, so variable service makes queues longer even at the same mean. Making every service the same length halves the wait.

The cell checks both formulas by simulation. A first-come first-served queue obeys Lindley's recursion. The wait of request $k$ is $W_k = \max(0, W_{k-1} + S_{k-1} - A_k)$, where $A_k$ is the gap between arrivals $k-1$ and $k$. In words, a request waits for the work the previous request left behind, minus the time since that request arrived, and never less than zero. Unrolling the recursion gives $W_k = \Sigma_k - \min_{0\le j\le k}\Sigma_j$ with $\Sigma_k = \sum_{i\le k}(S_{i-1} - A_i)$ and $\Sigma_0 = 0$, a running minimum that NumPy computes without a loop. The last line of the cell compares eight replicas with a queue each against eight replicas sharing one queue.

Simulation and theory agree to within a few percent, the error of a finite run near saturation. The last line makes the case for routing well. Eight replicas, each with its own queue at 80% load, make a request wait four service times on average. One shared queue feeding the same eight replicas cuts the wait to 0.29 service times, 14 times less. A request then never waits behind a busy replica while another sits idle. A router that sends each request to the least loaded replica comes close to the shared queue. A router that assigns requests at random gives up the whole gain. Poisson arrivals split at random give each replica Poisson arrivals of its own, which is exactly the case of one queue per replica.

LLM servers are not M/M/1 queues. A decode replica serves a whole batch at once and slows every user a little as the batch grows (§20.1), and requests queue only when its memory is full. The lesson still carries over. Delay is set by the slack $1 - \rho$, not by the load, and the safe operating point is well below full. Practitioners often run fleets at 60 to 80% of the throughput at which the latency targets still hold. The formulas above say how to pick the number from a TTFT target, and an interviewer who asks "why not run at 95%?" wants the factor $\rho/(1-\rho)$.

import numpy as np

def mm1(lam, mu):
    rho = lam / mu
    return {"rho": rho, "L": rho, "W": 1 / mu, "Wq": 0.0, "W_p99": 1 / mu}    # placeholder: no queueing

def mg1_wq(lam, ES, ES2):
    return lam * ES2 / 2                                                      # placeholder

Routing and autoscaling

A fleet is many replicas behind a router, which picks a replica for each request, and an autoscaler, which decides how many replicas there should be. Each makes a decision that the step-time model cannot make for it.

Where to send a request

A router balances three goals that pull apart.

• Load. Send each request where it will wait least. vLLM's data-parallel coordinator scores each engine as four times its waiting requests plus its running ones and picks the lowest (Gordić 2025). This least-loaded rule approximates the shared queue of the previous part.
• Cache affinity. Send each request where its prefix is already cached, so its prefill shrinks (§20.6). SGLang's router keeps an approximate radix tree of what each worker holds, updated from the requests it has routed. On a workload of long shared prefixes it reported 1.9 times the throughput of the previous version's routing, with a cache hit rate of 75% against 20% (SGLang Team 2024). Dynamo's router and llm-d route on the same signal.
• Adapter affinity. Send an adapter's requests to the replicas that have it loaded, so that each step mixes few distinct adapters.

Affinity concentrates load, and a popular system prompt can overload the one replica that holds it. So routers follow affinity only while the target's load stays within some margin of the least loaded replica, and they copy hot prefixes to more replicas. When imbalance appears anyway, a request can be moved. Llumnix migrates running requests together with their caches between replicas, like an operating system moving a process between cores. It reported tail latencies an order of magnitude lower, and up to 36% cost savings at similar tail latency (Sun et al. 2024).

How many replicas to run

Traffic follows the clock. DeepSeek's service peaked at 278 nodes and averaged 226.75 over its published day, a peak-to-mean ratio of 1.23. It moved nodes from inference to research and training at night (DeepSeek-AI 2025). An autoscaler adds replicas when load rises and removes them when it falls. Two choices decide whether it works.

The signal. The GPU utilization that nvidia-smi reports is the "percent of time over the past sample period during which one or more kernels was executing on the GPU" (NVIDIA). A decode replica serving a single user runs kernels almost all the time. It reads close to 100% at batch 1 and at batch 300 alike, so an autoscaler that targets 70% of it scales on noise. Other signals track how close a replica is to its latency targets. They are the number of waiting requests, the fraction of cache blocks in use, the percentiles of TTFT and of the gaps between tokens, and the tokens in flight against the capacity the plan computes.

The delay. A new replica must get a machine, pull the engine image, load the weights, start the engine, and record its CUDA graphs. This is its cold start. Loading 70.6 GB of FP8 weights from network storage at an assumed 2 GB/s takes 35 s alone, and provisioning adds minutes when no warm machine is free. Small fixes add up. Shipping precompiled Python bytecode in vLLM's container image saved about 18 s per startup in one H100 benchmark (Matteson 2026). So a scale-up decided now helps only one cold start later, while a scale-down takes effect at once.

That asymmetry has an exact consequence. Count time in steps, and say a replica requested at one step is ready $c$ steps later. Then a fleet that reacts to the current load is always $c$ steps behind on a rising curve.

In plain words, to be ready for the morning you must start the machines one cold start before the morning. That means scaling on a forecast, at least $c$ steps ahead, rather than on the current load. Daily traffic is regular enough to forecast. The price is idle capacity on the way down and whenever the forecast is wrong.

The widget simulates one day of traffic in 5-minute steps. The gray line is the demand, and the staircase is the capacity of the replicas that are ready. Pink shading marks the times when demand exceeds that capacity. Demand peaks at 200 requests per second, falls to a fifth of that at night, and carries 5% noise. The controls set the policy, the cold start, and the target utilization, and Play runs the day from midnight.

import numpy as np

def autoscale(load, per_replica_rps, cold_start, target_util=1.0, policy="reactive", min_replicas=1):
    load = np.asarray(load, dtype=float)
    ready = np.maximum(min_replicas, np.ceil(load / per_replica_rps)).astype(int)   # placeholder: instant scaling
    return ready, np.flatnonzero(load > ready * per_replica_rps)

From traffic to GPUs and dollars

We can now answer the opening question. A capacity plan sizes the prefill work and the decode work separately, adds headroom to each, and prices the result. Prefill is compute-bound, so its pool is sized by FLOPs. Decode is memory-bound, so its pool is sized by how many sequences a replica can hold. Here is the plan in general.

In plain words, prompts cost FLOPs and are sized by arithmetic. Answers cost memory and bandwidth and are sized by the batch a replica can hold. Both get slack so that queues stay short. The largest batch in step 2 is the smaller of two bounds from §20.1. One is the number of sequences whose caches fit in the free memory, and the other is the largest batch whose step still meets the TPOT target. Little's law gives the same decode pool from the other side. The pool holds $\lambda_{\text{req}}(T_{\text{out}}-1)\,t_{\text{step}}$ sequences at once, and that must fit in the replicas' memory at the target utilization.

The cell plans the opening question on H100s. Its assumptions are FP8 weights and arithmetic, a bfloat16 cache, and TP4 instances for both pools (the best decode layout found above). Prefill runs at 50% MFU, and both pools target $\rho_{\max} = 0.8$. The cell checks TTFT by simulating each prefill instance's queue with wait_times, with requests split at random among the instances. It prices the fleet at an assumed $2 per GPU-hour. It then follows the load through a day that peaks at 200 requests per second at 2 p.m. and bottoms out at 40 around 2 a.m., and repeats the plan on B200s.

Prefill dominates this plan, with 76 of its 112 GPUs. There are five times as many prompt tokens as output tokens, while a prompt token is only about 2.4 times cheaper to process than an output token here (about 6,900 against 2,900 tokens per second per GPU). A prefix cache that served the first half of each prompt would cut prefill FLOPs nearly in half and free 36 GPUs (9 of the 19 instances), which is why system prompts and conversation histories are worth caching. Every number here is a roofline bound with stated efficiencies. Real kernels, real distributions of prompt lengths, and engine overheads must be measured before the order is placed.

Two refinements are worth knowing. First, separating the pools, as this plan does, lets each be sized and laid out on its own, but it costs a cache transfer, 13 ms here. It also wastes the idle arithmetic of memory-bound decode steps. A shared pool with chunked prefill fills that idle arithmetic with prompt work for free, up to the limit of §20.6. Second, headroom for failures matters more than it looks. Losing one of the 9 decode replicas at peak moves the other eight from 77% to 86% of their throughput.

This is the system design question of inference interviews, and the interviewer wants the structure more than the digits. Size prefill by FLOPs and decode by memory, put headroom on both, then check TTFT and state a price.

The planner below runs the same arithmetic at any request rate, for Llama 3.1 8B on single H100s or the 70B model on groups of four. The other assumptions are the example's, from the request shape and the TPOT target to $2 per GPU-hour. It compares the fleet's cost per thousand requests with an API that charges one price per million tokens, input and output alike. The default API prices, $0.10 and $0.60 per million, are inputs to play with, not market data. Lower the request rate and watch the fleet's cost per request climb once even the smallest fleet sits partly idle.

import math

def capacity_plan(traffic, model, hw, slo, kappa):
    raise NotImplementedError

A published fleet, checked

A plan is only as good as its check against reality. DeepSeek published a day of its production service (§20.1), and the same arithmetic can be run backward on it. DeepSeek served on H800 GPUs, the export version of the H100, in nodes of eight.

Self-hosting against an API

The plan gives a cost per token when the fleet is fully used. You pay for the fleet whether or not it is used, so what matters is how much of it you use.

Proof. Over any period the fleet costs the same as at full use while serving the fraction $u$ of the tokens, so the cost per token is multiplied by $1/u$. Comparing $c/u$ with $p$ gives the threshold. ∎

The threshold makes the usual advice exact. A fleet sized for the peak and never scaled down runs at its mean-to-peak ratio times its headroom, which for our day is $120/200 \times 0.8 = 0.48$ of its capacity. A small product at a few requests per second may keep a minimum fleet of 8 GPUs mostly idle.

The engineers who run the fleet, the spare capacity, and the evaluation of every model update come on top. That is why many teams self-host only their highest-volume models. In an interview, "self-host or API?" is answered by this utilization and by requirements the price ignores, such as where the data may live.

Where it breaks

Nondeterminism. The same request at temperature 0 can give different answers on different runs. GPU kernels choose their order of summation by batch size, and floating-point addition is not associative, so a request's logits depend on what else shares its step (§6.4). Batch-invariant kernels fix the order of summation whatever the batch, at some cost in speed (He 2025). Reinforcement learning now drives that work, because the engine that generates rollouts and the trainer must agree on every log-probability. A 2026 recipe in vLLM made the two bitwise identical for DeepSeek-V4-Flash at 91% of the optimized rollout throughput, and carried the method to a second model (Bai and Shen 2026).

CPU work that grows with the output. Detokenizing, checking stop strings, and parsing reasoning and tool calls run per token on the CPU. In vLLM's frontend with no GPU, streaming a parsed reasoning output replays the token history through a fresh parser for each chunk. On outputs of several thousand tokens that measured about 9 times the CPU of serving in one process (Hickey 2026).

Grammars that are expensive to compile or to mask. The first request with a new schema waits for compilation. Grammars whose tokens mostly depend on the stack defeat the precomputed masks.

Tenants that slow each other. Distinct LoRA adapters in a step, a prefix that every router sends to one replica, and one customer's 100,000-token prompts all slow the steps that other requests share.

Lengths with heavy tails. Reasoning models write outputs whose lengths vary by orders of magnitude (§18.7). Plans built on mean lengths fill memory sooner than expected and need preemption, which costs a recomputed prefill (§20.6). Size memory for the percentiles, not the means.

Scaling on the wrong signal, or too late. Scaling on nvidia-smi utilization, scaling reactively against a cold start of minutes, and scaling down so fast that every dip is followed by a cold start all show up as spikes in TTFT.

The table collects the main serving decisions, what each buys, and what it costs.

In PyTorch

The block implements the multi-LoRA layer twice, in the gather form used for decode and the segmented form used for prefill, and checks both against a loop. It then applies a packed grammar bitmask to a batch of logits with bit operations, the way engines do it on the GPU.

In production you start an engine and talk to it over HTTP. The commands below are shown, not run. They start vLLM with Llama 3.1 70B in FP8 on four GPUs with adapters enabled. Then they send a request that names an adapter and demands JSON that matches a schema. Flags change between versions, so check them against the documentation of the version you run.

vllm serve meta-llama/Llama-3.1-70B-Instruct \
    --quantization fp8 --tensor-parallel-size 4 \
    --max-num-batched-tokens 8192 --max-num-seqs 256 --gpu-memory-utilization 0.90 \
    --enable-lora --max-loras 8 --max-lora-rank 16 \
    --lora-modules support=/adapters/support-v3

Read more

• vLLM Team (2025). vLLM V1: A Major Upgrade to vLLM's Core Architecture. The unified token-count scheduler, zero-overhead prefix caching, persistent batches, and piecewise CUDA graphs, with the CPU-overhead argument that motivated them.
• Gordić (2025). Inside vLLM: Anatomy of a High-Throughput LLM Inference System. A walk through vLLM's engine core, scheduler, KV manager, guided decoding, speculative decoding, disaggregation, and multi-node serving, at the level of the code.
• Zheng et al. (2024). SGLang: Efficient Execution of Structured Language Model Programs. RadixAttention, cache-aware scheduling, and compressed finite-state machines with jump-forward decoding.
• SGLang Team (2024). SGLang v0.4: Zero-Overhead Batch Scheduler, Cache-Aware Load Balancer, Faster Structured Outputs. Overlap scheduling with future tokens, a prefix-aware router, and XGrammar integration, each with measurements.
• Dong et al. (2024). XGrammar: Flexible and Efficient Structured Generation Engine for Large Language Models. Context-independent and context-dependent tokens, the persistent stack, and overlapping mask generation with the GPU.
• Sheng et al. (2024). S-LoRA: Serving Thousands of Concurrent LoRA Adapters. Unified paging of adapters and KV caches, heterogeneous-rank kernels, and tensor parallelism for adapters. Read it with Chen et al. (2023), Punica: Multi-Tenant LoRA Serving, for the segmented gather kernel (SGMV) and a scheduler that consolidates multi-tenant LoRA traffic onto few GPUs.
• Sun et al. (2024). Llumnix: Dynamic Scheduling for Large Language Model Serving. Live migration of requests and their caches across replicas for load balancing, isolation, and priorities.
• Hickey (2026). Taking vLLM Apart: A Practical Guide to Disaggregated Serving. When to separate prefill from decode and the CPU frontend from the GPUs, with measured costs of both splits.

Watch next

▶

▶

▶

▶