Chapter 6 · Section 6.4 · 55 min

Sampling and decoding

A trained language model doesn't produce text. It produces a probability distribution over the next token. Turning a stream of those distributions into text takes a separate algorithm, the decoder, and the choice of decoder changes what you get as much as the choice of model does. Ask GPT-2 to continue a story with the obvious strategy, always taking the most likely token, and within a paragraph it is repeating itself in a loop. Sample faithfully from its full distribution and it wanders into incoherence. Holtzman et al. (2020) called this neural text degeneration and made it famous with examples exactly like these.

Here's the puzzle at the heart of this section. If the model is good, shouldn't its most probable text be its best text? We'll see that the answer is no, measure why, and then build every decoding method in common use (greedy, temperature, top-k, top-p, min-p, repetition and frequency penalties, beam search, and stop sequences), each with its exact semantics, because interviewers ask about the edge cases.

A model to decode from

Every decoder in this section needs only one thing from a model, a function next_logprobs(context) that maps the tokens so far to log-probabilities for the next one. In this section's widgets that function is the tiny GPT itself, the §6.2 architecture trained with the §6.3 recipe on all 154 sonnets, running in your browser.

The Python cells can't call it, and it's worth saying why. They also run in your browser, but in a Python interpreter that cannot reach the GPT's 112,000 trained weights (shipping them inside the page as code would add about 150 KB of numbers to it), and training a GPT from scratch there, in NumPy with a hand-written backward pass and no GPU, would take minutes instead of the second a cell gets. So the cells use a stand-in with the same interface and very similar predictions, a character 5-gram built by counting the same seven training sonnets. It predicts the next character from the previous four and backs off smoothly to shorter contexts when a context is rare (the n-gram models of §3.1). On held-out sonnet 116 it scores 2.55 nats per character, against 2.53 for the GPT of §6.3 trained on the same data, so its next-character distributions are about as peaked or as flat as that GPT's, and the shape of the distribution is all a decoder ever sees. Every function below works unchanged with a GPT. The PyTorch block at the end of the section shows it, by training the §6.3 GPT for a few seconds and decoding from it with the same sampler.

Python · cell 1

Python · cell 2

The decoding loop

Every decoder is the same loop with a different pick step. Start from a prompt, ask the model for the next-token distribution, choose a token, append it, and repeat until a stop condition. For a GPT with a fixed context $T$ , the model only ever sees the last $T$ tokens, so the loop crops (§6.5 shows how to avoid recomputing everything each step).

Python · cell 3

Greedy decoding found a loop within a few words, "the world's day arising" forever. The mechanism is simple. Once a phrase has occurred, the context now contains it, which makes the phrase's continuation more likely, which makes it occur again. Large models do the same thing with whole sentences. Holtzman et al. showed GPT-2 falling into these loops under greedy and beam search, and found that the probability a model assigns to a repeated phrase increases with each repetition.

The most likely text is not typical text

Now the measurement that resolves the puzzle. Let's score four continuations of a prompt from the held-out sonnet by their average log-probability per character under the model. The four are greedy, beam search (which we build below), plain sampling, and what Shakespeare actually wrote.

Python · cell 4

The search methods find text the model likes far more than anything Shakespeare wrote, with an average log-probability between $-0.5$ and $-0.65$ against $-3.5$ . Plain sampling lands in between, at about $-1.1$ , roughly the model's own entropy, which is what a typical draw from a distribution should score. Human text is full of surprises, low-probability characters and words that carry the information, so it lives well below the mode of the distribution. Text that maximizes probability is the blandest, most repetitive text the model can produce. In plain words, we don't want the most likely sequence. We want a typical sample from the distribution, minus its unreliable long tail. Sampling is the right starting point, and the rest of this section is about trimming the tail.

Temperature

Temperature $\tau \gt 0$ rescales the logits before the softmax,

p_i(\tau) = \frac{\exp(z_i/\tau)}{\sum_{j=1}^{V}\exp(z_j/\tau)} .

In plain words, dividing by $\tau \lt 1$ stretches the gaps between logits, so the distribution gets sharper. Dividing by $\tau \gt 1$ shrinks the gaps, so it gets flatter. At $\tau = 1$ you sample from the model as trained. As $\tau \to 0$ all the mass moves to the largest logit, which is greedy decoding, and as $\tau \to \infty$ it spreads uniformly over every token with a finite logit. Temperature is the same knob as in the softmax of §4.1, and it reappears in distillation and in contrastive losses (§7.2).

Raising the temperature never lowers the entropy. That's worth proving, because it is the reason temperature is a clean "creativity" dial.

∂ Entropy increases with temperature

Write $p_i = e^{z_i/\tau}/Z$ with $Z = \sum_j e^{z_j/\tau}$ . Then $\log p_i = z_i/\tau - \log Z$ , so the entropy is

H(\tau) = -\sum_i p_i \log p_i = \log Z - \frac{1}{\tau}\,\mathbb{E}_p[z].

Differentiate the pieces. First, $\frac{d}{d\tau}\log Z = \frac{1}{Z}\sum_j e^{z_j/\tau}\big(-z_j/\tau^2\big) = -\mathbb{E}_p[z]/\tau^2$ . Second, $\frac{dp_i}{d\tau} = p_i\big(-z_i/\tau^2 + \mathbb{E}_p[z]/\tau^2\big)$ , so

\frac{d}{d\tau}\mathbb{E}_p[z] = \sum_i z_i \frac{dp_i}{d\tau} = -\frac{1}{\tau^2}\Big(\mathbb{E}_p[z^2] - \mathbb{E}_p[z]^2\Big) = -\frac{\operatorname{Var}_p(z)}{\tau^2}.

Putting it together,

\frac{dH}{d\tau} = -\frac{\mathbb{E}_p[z]}{\tau^2} - \left(\frac{1}{\tau}\cdot\frac{-\operatorname{Var}_p(z)}{\tau^2} - \frac{\mathbb{E}_p[z]}{\tau^2}\right) = \frac{\operatorname{Var}_p(z)}{\tau^3} \;\ge\; 0 .

The entropy is strictly increasing unless all finite logits are equal, in which case it is already at its maximum.

Python · cell 5

Low temperature reads like the training data and soon repeats. High temperature invents words. Two edge cases matter in code. $\tau = 0$ can't be computed by division, so every library special-cases it as argmax, and negative temperatures are an error, since they would invert the preferences. And with a tiny positive $\tau$ the scaled logits are enormous, so the max must be subtracted before exp, exactly as in §6.3.

Warm-up

Temperature, including the edge cases

Write temperature_probs(logits, tau) returning the probability vector $\operatorname{softmax}(\mathbf{z}/\tau)$ for a 1-D array of logits, which may contain -inf (forbidden tokens). Raise ValueError for tau < 0. For tau == 0 return the greedy one-hot distribution on np.argmax(logits) (the first maximum if there are ties). It must not overflow or produce NaN for tau = 1e-8 or for tau = 1e8, and forbidden tokens must always get exactly 0.

import numpy as np

def temperature_probs(logits, tau):
    raise NotImplementedError

Hidden tests, hints, and AI help, from $1.99 a month.

Truncation, top-k, top-p, and min-p

Temperature reshapes the whole distribution, but it can't fix the real problem with sampling, the tail. A character model's tail is 40 or 50 characters that each have a tiny probability. A BPE model's tail is tens of thousands of tokens. Each tail token is individually unlikely, but together they hold real mass, so a long generation is almost sure to draw a few, and each one can derail the text. Truncation methods throw the tail away and renormalize what's left. They differ in how they decide where the tail starts.

Top-k (Fan et al., 2018) keeps the $k$ tokens with the largest logits. Simple, but the right $k$ depends on the context. After "Shall I compare thee to a summer's d", only a few vowels are plausible, and $k = 40$ lets dozens of bad characters in. At the start of a new line, dozens of characters are fine, and $k = 5$ cuts good ones. Implementations keep every token whose logit is at least the $k$ -th largest, so ties at the boundary all survive and you can get more than $k$ tokens.

Top-p, or nucleus sampling (Holtzman et al., 2020), adapts to the context. Sort the tokens by probability, and keep the smallest prefix whose total probability reaches $p_{\text{top}}$ . Precisely, with the tokens in descending order $p_{(1)} \ge p_{(2)} \ge \cdots$ , token $(r)$ survives exactly when the mass strictly before it is below the threshold,

\text{keep } (r) \iff \sum_{j \lt r} p_{(j)} \lt p_{\text{top}}, \qquad \tilde{p}_{(r)} = \frac{p_{(r)}}{\sum_{\text{kept}} p_{(j)}} .

In plain words, walk down the sorted list adding up probability, and stop as soon as you've collected $p_{\text{top}}$ . The token that pushes the total over the line is kept. The top token is always kept, since the mass before it is 0.

Let's do one by hand. Take $\mathbf{p} = (0.5, 0.2, 0.15, 0.1, 0.05)$ and $p_{\text{top}} = 0.8$ . The masses before each token are $0, 0.5, 0.7, 0.85, 0.95$ . The first three are below 0.8, so three tokens survive, holding 0.85 of the mass, and renormalizing gives $(0.588, 0.235, 0.176)$ . With $p_{\text{top}} = 0.7$ the third token's mass-before is exactly 0.7, which is not below 0.7, so only two survive, holding exactly 0.7. That's the "smallest set reaching $p_{\text{top}}$ " definition at work, and it's where implementations that use $\le$ instead of $\lt$ disagree. (Hugging Face's version removes tokens whose cumulative mass counted from the bottom is at most $1 - p_{\text{top}}$ , which is the same rule up to floating-point rounding.)

Min-p (Nguyen et al., 2024) sets the cutoff relative to the top token,

\text{keep } i \iff p_i \ge p_{\min}\cdot \max_j p_j .

In plain words, a token survives if it is at least $p_{\min}$ times as likely as the favorite. When the model is confident, the favorite is near 1 and the cutoff is high, so almost nothing else survives. When it is unsure, the favorite is small and many tokens survive. The authors report that this keeps text coherent at higher temperatures than top-p does. Implementations differ on whether temperature is applied before or after truncation, and the order changes the result, so check it when you compare settings across libraries.

The widget below runs the tiny GPT, trained with §6.3's code on all 154 sonnets, in your browser. Pick a prompt and watch which next characters survive each filter, and with what renormalized probability.

Python · cell 6

After "summer's d", top-p and min-p both keep the three vowels the model finds plausible, while top-k 5 also admits 'e' and 'i' only because it must keep five. That adaptivity is why top-p (often around 0.9 to 0.95) and min-p (around 0.05 to 0.1) are the common defaults for open-ended text, frequently combined with a moderate temperature.

Interview

Top-k filtering, with ties

Write top_k_filter(logits, k) returning a new array in which every logit strictly smaller than the k-th largest logit is replaced by -inf. Ties with the k-th largest are kept, so more than k tokens may survive. If k <= 0 or k >= len(logits), return an unchanged copy. Do not modify the input. Entries that are already -inf stay -inf.

import numpy as np

def top_k_filter(logits, k):
    raise NotImplementedError

Hidden tests, hints, and AI help, from $1.99 a month.

Interview

Top-p (nucleus) filtering, with exact threshold semantics

Write top_p_filter(probs, top_p) for a probability vector probs (non-negative, summing to 1). Order tokens by probability, descending, breaking ties by lower index first. Keep a token exactly when the total probability of the tokens before it in that order is strictly less than top_p. Return the kept probabilities renormalized to sum to 1, with zeros elsewhere, as a new array. The top token is always kept.

import numpy as np

def top_p_filter(probs, top_p):
    raise NotImplementedError

Hidden tests, hints, and AI help, from $1.99 a month.

Repetition, frequency, and presence penalties

Truncation doesn't stop loops, because the loop's tokens are the likely ones. Penalties attack repetition directly by lowering the logits of tokens that already appeared. Let $c_i$ be the number of times token $i$ occurs in the generated text so far. Two families are in common use.

The repetition penalty of CTRL (Keskar et al., 2019) uses a factor $\rho \gt 1$ (they suggest about 1.2) on every token that has appeared. The paper writes it as dividing each such logit by $\rho$ inside the softmax. Libraries implement it with two cases,

z_i \leftarrow \begin{cases} z_i / \rho & c_i \gt 0 \text{ and } z_i \gt 0, \\ z_i \cdot \rho & c_i \gt 0 \text{ and } z_i \le 0, \\ z_i & c_i = 0. \end{cases}

The two cases matter. Dividing a negative logit by $\rho$ would move it toward zero, making the token more likely, the opposite of a penalty. Multiplying moves it further down. This rule is what Hugging Face's repetition_penalty implements. Libraries also differ on what counts as "already appeared". Hugging Face's version looks at every token in the input, the prompt included, so a long prompt penalizes the very words a faithful answer should reuse.

The frequency and presence penalties of OpenAI's API are additive,

z_i \leftarrow z_i - \alpha_{\text{freq}}\, c_i - \alpha_{\text{pres}}\, \mathbb{1}[c_i \gt 0] .

In plain words, the presence penalty charges a flat fee for any token already used, which nudges the model toward new topics, and the frequency penalty charges per use, which specifically damps words repeated over and over. Both are blunt instruments. Code, lists, names, and rhyme schemes repeat legitimately, and a strong penalty breaks them, so penalties are usually kept small or left at zero for factual tasks. A third, stricter tool is no-repeat n-gram blocking, which forbids any token that would complete an n-gram already present.

Core

Repetition, frequency, and presence penalties

Write apply_penalties(logits, generated, repetition_penalty=1.0, frequency_penalty=0.0, presence_penalty=0.0). generated is a list of token ids produced so far. First apply the CTRL repetition penalty to every token that occurs in generated (divide positive logits, multiply non-positive ones), then subtract frequency_penalty * count + presence_penalty * (count > 0). Return a new array and leave the input unchanged. Tokens never generated must be untouched, and -inf logits must stay -inf.

import numpy as np

def apply_penalties(logits, generated, repetition_penalty=1.0, frequency_penalty=0.0, presence_penalty=0.0):
    raise NotImplementedError

Hidden tests, hints, and AI help, from $1.99 a month.

The whole pipeline, batched

A real sampler applies several of these steps to every row of a batch at once, and the order matters, because each filter reads the distribution the previous one left behind. The common library order, which Hugging Face uses and the PyTorch block below follows, is penalties, then temperature, then top-k, then top-p on the distribution renormalized over the top-k survivors, then min-p on what is left. Put temperature after top-p and the nucleus is measured on the wrong distribution. Measure top-p on the full softmax instead of on the top-k survivors and you keep more tokens than the library does. Either way your settings no longer mean what they mean everywhere else. Serving systems also never loop over rows in Python, so the whole pipeline is written with array operations over a $(B, V)$ block of logits.

Interview

A batched logits processor

Write filter_probs(logits, temperature=1.0, top_k=0, top_p=1.0, min_p=0.0) for a (B, V) array of logits, which may contain -inf, returning the (B, V) probabilities to sample from, with each row summing to 1. If temperature == 0, return the one-hot rows of the greedy choice (first maximum on ties). Otherwise divide by the temperature, apply top-k (keep logits at least the k-th largest, ties kept, disabled when top_k <= 0 or top_k >= V), take the softmax, apply top-p to that distribution (keep a token when the mass strictly before it, in descending order with ties by lower index, is below top_p; disabled at 1.0) and renormalize, then apply min-p (keep tokens with probability at least min_p times the row's largest; disabled at 0) and renormalize. Use no Python loop over the rows.

import numpy as np

def filter_probs(logits, temperature=1.0, top_k=0, top_p=1.0, min_p=0.0):
    """(B, V) logits -> (B, V) probabilities after temperature, top-k, top-p, min-p"""
    raise NotImplementedError

Hidden tests, hints, and AI help, from $1.99 a month.

Beam search for language models

Beam search keeps the $b$ highest-scoring partial sequences at every step instead of one, extends each by every possible token, and keeps the best $b$ of the results. You met it for translation in §3.5. The score of a sequence $y_1, \dots, y_n$ is its total log-probability, and because every extra token adds a negative term, raw scores favor short outputs. So finished hypotheses are compared with a length penalty,

\operatorname{score}(y) = \frac{\sum_{t=1}^{n} \log p(y_t \mid y_{\lt t}, \text{prompt})}{n^{\alpha}},

with $\alpha \in [0, 1]$ , where $\alpha = 0$ is no normalization and $\alpha = 1$ is the average log-probability per token. (GNMT used the smoother $((5 + n)/6)^\alpha$ , Wu et al., 2016.)

For open-ended text, beam search inherits greedy's problem with more force. It is a better optimizer of likelihood, and maximum likelihood is the degenerate, repetitive mode, as the scores above showed. Wider beams can even make the output worse (Koehn & Knowles, 2017), an effect later dubbed the beam search curse. Where there is one right answer, and the model's distribution concentrates on it, such as translation, speech recognition, or short structured outputs, beam search is still the best tool. Its cost is $b$ forward passes per step, and $b$ KV caches (§6.5).

Interview

Beam search with EOS and length normalization

Implement beam_search(step_logprobs, eos, beam_width, max_len, alpha) exactly as follows. step_logprobs(seq) takes the list of tokens generated so far and returns a NumPy array of next-token log-probabilities. Start with one empty hypothesis of score 0. At each of up to max_len steps, extend every alive hypothesis by every token, giving candidates scored by total log-probability. Sort all candidates by that score, descending, and keep the top beam_width. Of those, candidates ending in eos move to a finished list and the rest stay alive. Stop early if nothing is alive. Afterwards, add the still-alive hypotheses to the finished list, and return the finished sequence (a list, including eos if it has one) with the highest total / len(seq) ** alpha.

import numpy as np

def beam_search(step_logprobs, eos, beam_width, max_len, alpha):
    raise NotImplementedError

Hidden tests, hints, and AI help, from $1.99 a month.

Stop sequences and streaming

Generation ends at the EOT token, at a length limit, or at a stop sequence, a string such as "\n\n" or "User:" that marks the end of the answer. Stop strings are matched in decoded text, not in token ids, because the same string can be tokenized many ways and can span several tokens. Streaming makes this subtle. If the stop string is "END" and the model has produced "...E", you can't show the E yet, since the next token might complete the stop string, which must never reach the user. You hold back exactly the longest suffix of the text that could still be the start of a stop string, and release it as soon as it can't.

Interview

Streaming with stop sequences

Write stream_with_stops(pieces, stops). pieces is the list of decoded text pieces, one per generated token, and stops is a list of non-empty stop strings. Process pieces one at a time, maintaining the accumulated text. After each piece, if any stop string occurs in the text, emit the not-yet-emitted text up to the earliest occurrence of any stop string, and end. Otherwise emit everything except the longest suffix of the text that is a proper prefix of some stop string. Return the list of chunks emitted, one per processed piece, plus, if the pieces run out without a stop, one final chunk with the held-back text.

For example, with stops ["END"] and pieces ["Hello", " E", "N", "D!"], the chunks are ["Hello", " ", "", ""].

def stream_with_stops(pieces, stops):
    raise NotImplementedError

Hidden tests, hints, and AI help, from $1.99 a month.

Determinism and seeds

Sampling is random, so reproducibility means controlling the random numbers. Draw them from a seeded generator, one per request, and the same model, prompt, and settings give the same text. Tie the generator to the request, not to the process. A global seed makes one request's output depend on how many random numbers the other requests in the batch consumed.

Even then, the same seed on the same model is not always the same answer in production, and neither is greedy decoding at temperature 0. The logits themselves can differ in their last bits from run to run, because GPU kernels choose different reduction orders for different batch sizes, and floating-point addition is not associative. When two candidates are nearly tied, a difference in the seventh decimal flips the argmax, and from then on the two generations diverge completely. The Thinking Machines team traced most nondeterminism in LLM inference servers to this lack of batch invariance and showed that batch-invariant kernels make temperature-0 outputs bit-identical, at some cost in speed (He, 2025).

In PyTorch

A production sampler works on a whole batch of logits at once, $(B, V)$ , with no Python loop over the vocabulary. Here is one, in the order of Exercise 6.4g, with per-request generators. To decode from the model this chapter builds, the block first trains the GPT of §6.3 on the seven training sonnets (the same architecture and recipe, compacted, a few seconds on a CPU), then generates from three prompts at once.

🔥 In PyTorchfor a machine with PyTorch

The GPT, after two hundred steps on 4,300 characters, writes with the same texture as the 5-gram's samples above, plausible letter sequences and line breaks with few real words, which is what their nearly equal held-out losses predict. The 154-sonnet model in the widgets, with twenty times the data and fifteen times the steps, does much better.

A real server avoids the per-row loop by drawing each row's random numbers from that request's own stream, for example as Gumbel noise, and then sampling the whole batch in one kernel. With Hugging Face models the same knobs are arguments to generate.

method	what it does	good for	failure mode
greedy	argmax every step	short, factual, deterministic answers	loops and blandness on long text
beam search	keeps the $b$ best prefixes	translation, ASR, constrained outputs	even blander, wider is not always better
temperature	sharpens or flattens	the main creativity dial	high values reach the tail
top-k	keeps $k$ tokens	a simple cap on the tail	the right $k$ depends on context
top-p	keeps mass $p_{\text{top}}$	adaptive truncation, a common default	a flat distribution still admits many tokens
min-p	keeps tokens above $p_{\min} p_{\max}$	coherence at higher temperatures	ordering with temperature varies by library
penalties	lower logits of used tokens	breaking loops	harms code, lists, and names

🎯 Interview questions on this section

Why does greedy decoding produce repetitive text?

Generated text becomes context, and a phrase in the context makes its own continuation more likely, so the argmax keeps choosing it. Holtzman et al. found the probability of a repeated phrase rises with each repetition. More fundamentally, maximum-likelihood text is atypical. Human text has much lower per-token probability than the model's mode, so searching for the most likely sequence finds bland, repetitive outputs.

Define top-p precisely. What happens at the boundary?

Sort tokens by probability, descending, and keep the smallest prefix whose total probability is at least $p_{\text{top}}$ , then renormalize. Equivalently, keep token $(r)$ when the mass strictly before it is below $p_{\text{top}}$ . The token that crosses the threshold is included, and the top token is always kept. If the mass before a token is exactly $p_{\text{top}}$ , that token is excluded, because the set already reaches the threshold. Implementations that compare with $\le$ keep one extra token in that case.

Top-k versus top-p versus min-p, when would you use each?

Top-k is a fixed cap, cheap and predictable, but blind to how peaked the distribution is. Top-p adapts the number of survivors to the distribution, many when the model is unsure and few when it is confident, which is why it became the default for open-ended text. Min-p sets the cutoff relative to the top probability, so it is even more aggressive when the model is confident and tends to stay coherent at higher temperatures. They compose, and Hugging Face applies temperature, then top-k, then top-p, then min-p, but other libraries order them differently, so check before comparing settings.

What does temperature do to entropy, and what are its edge cases?

$dH/d\tau = \operatorname{Var}_p(z)/\tau^3 \ge 0$ , so entropy increases monotonically with temperature, from 0 (greedy, as $\tau \to 0$ ) to $\log$ of the number of allowed tokens (uniform, as $\tau \to \infty$ ). $\tau = 0$ must be special-cased as argmax, negative temperature inverts preferences and should be rejected, and tiny $\tau$ needs the max-subtraction trick to avoid overflow. With exact ties, $\tau \to 0^+$ splits mass evenly among the tied maxima while argmax picks one.

How does the CTRL repetition penalty handle negative logits, and why?

The paper's formula divides every seen token's logit by $\rho$ , but the standard implementation (Hugging Face's) multiplies negative logits by $\rho \gt 1$ instead. Dividing a negative logit by $\rho$ would move it toward zero and make the token more likely, the opposite of a penalty. So positive logits of seen tokens are divided by $\rho$ and non-positive ones multiplied, both pushing the token down.

When is beam search the right choice for an LLM, and what does it cost?

When the output is short and has a narrow set of correct answers, such as translation, transcription, or structured fields, where the mode of the distribution is the answer. For open-ended generation it produces bland, repetitive text and wider beams can be worse. It costs $b$ times the compute and memory per step, including $b$ KV caches, and needs length normalization to avoid favoring short outputs.

You set temperature to 0 and seed everything, yet the API sometimes returns different answers. Why?

The logits aren't bit-identical across calls. Inference servers batch requests together, and GPU kernels pick different reduction orders for different batch sizes, so the same prompt gets slightly different floating-point results depending on what else is in the batch. A near-tie in the argmax then flips, and the generations diverge. Batch-invariant kernels fix it at some cost in throughput.

🧭 What to remember

Decoding is a separate algorithm from the model. The most likely text is degenerate, so good decoders sample a typical continuation and cut the unreliable tail.
Temperature divides the logits. Entropy rises monotonically with it, $\tau = 0$ means argmax, and negative values are invalid.
Top-k keeps a fixed number (ties at the boundary included), top-p keeps the smallest prefix reaching $p_{\text{top}}$ (the crossing token included), min-p keeps tokens above $p_{\min}$ times the top probability.
Repetition penalties divide positive and multiply negative logits of seen tokens. Frequency and presence penalties subtract per use and per presence.
Beam search with a length penalty is for tasks with one right answer. Stop strings are matched in text, and streaming must hold back possible prefixes.
Same seed is not same output when batch composition changes the floating-point arithmetic.
Next, §6.5 makes the decoding loop fast, by noticing that each step recomputes almost everything the previous step already computed.

Holtzman et al. (2020). The Curious Case of Neural Text Degeneration. Why likelihood-maximizing decoding degenerates, and nucleus sampling.
Fan, Lewis & Dauphin (2018). Hierarchical Neural Story Generation. Top-k sampling for open-ended generation.
Nguyen et al. (2024). Turning Up the Heat: Min-p Sampling for Creative and Coherent LLM Outputs. The min-p paper, with its behavior at high temperature.
Keskar et al. (2019). CTRL: A Conditional Transformer Language Model for Controllable Generation. The source of the repetition penalty.
Wu et al. (2016). Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation. Beam search with length normalization at production scale.
Koehn & Knowles (2017). Six Challenges for Neural Machine Translation. Including the observation that wider beams can lower quality.
He (2025). Defeating Nondeterminism in LLM Inference. Batch invariance and why temperature 0 is not deterministic.
von Platen (2020). How to generate text. A practical tour of the same methods in the Hugging Face library.

Zero to Frontier is made for larger screens

Sampling and decoding

A model to decode from

The decoding loop

The most likely text is not typical text

Temperature

Temperature, including the edge cases

Truncation, top-k, top-p, and min-p

Top-k filtering, with ties

Top-p (nucleus) filtering, with exact threshold semantics

Repetition, frequency, and presence penalties

Repetition, frequency, and presence penalties

The whole pipeline, batched

A batched logits processor

Beam search for language models

Beam search with EOS and length normalization

Stop sequences and streaming

Streaming with stop sequences

Determinism and seeds

In PyTorch

Read more

Watch next