Chapter 6 · Section 6.6 · 45 min

Evaluating a language model

Our GPT from §6.3 ends training with a validation loss of 2.53 nats. Is that good? The number alone says nothing. It needs a reference point, a unit that survives a change of tokenizer, and a connection to what we actually want the model to do. GPT-2's paper reports 0.93 bits per byte on Wikipedia text and 63 percent accuracy on LAMBADA, a test of predicting the last word of a passage. Can you compare either of those numbers with ours? This section gives you the tools to answer, and the habits that keep evaluation honest.

There are three layers of evaluation, from cheapest to most meaningful. Intrinsic metrics measure how well the model predicts held-out text (loss, perplexity, bits per byte). Downstream metrics measure whether it can do tasks, by prompting it with a few examples or by comparing how likely it finds each answer of a multiple-choice question. And sample quality asks whether what it writes is any good, which no single number captures. A good evaluation uses all three, because each can be fooled in ways the others catch.

Baselines first

We'll need the corpus once more, and a family of reference models. The best way to make sense of 2.53 nats is to line it up against simple models that make clear assumptions, from a uniform guess to counting models with more and more context.

Python · cell 1

Python · cell 2

The GPT beats the uniform guess by a mile and the unigram clearly, but the smoothed trigram, which predicts from just two previous characters, does better on held-out sonnet 116 than our GPT. Longer n-grams get worse again. Both facts are the data talking. With 4,300 training characters there is enough evidence to estimate three-character statistics well, but not five-character ones, and not the much richer function a Transformer can represent. The 154-sonnet model that runs in the widgets, with the same architecture and twenty times the data, reaches 1.62 nats per character on its own held-out sonnets. Always run the cheap baselines. They tell you whether your model has learned more than counting, and they catch evaluation bugs, since nothing should beat a correct model by an absurd margin.

Held-out loss and perplexity

The basic intrinsic metric is the mean negative log-likelihood of held-out text, $\bar{\ell} = -\frac{1}{n}\sum_{t=1}^{n} \log p_\theta(w_t \mid w_{\lt t})$ in nats per token over its $n$ tokens, and its exponential, the perplexity $\operatorname{PPL} = e^{\bar{\ell}}$ . §0.5 derives both and shows how to pool them over a corpus. The one-line reading is that perplexity is an effective branching factor, the number of equally likely choices that would leave you as unsure as the model is. Our GPT's 2.53 nats is a perplexity of 12.5, as unsure as choosing among about 12 characters, against 56 for the uniform guess.

Two practical details decide whether the number is right. First, what counts as held out. The validation set must be split by document and deduplicated against training (§6.1), or perplexity measures memorization. Second, how much context each prediction gets. A model with a context of $T$ tokens can't read a long document in one pass. Cutting the document into non-overlapping windows, as our val_loss in §6.3 did, makes the first tokens of each window predict from almost nothing, which inflates the loss. The fix is a sliding window, advancing by a stride $s \lt T$ and scoring only the tokens that are new in each window, so every token gets at least $T - s$ tokens of context. A stride of 1 gives every token the maximum context at the price of one forward pass per token.

Interview

Perplexity with a sliding window

Write sliding_window_nll(ids, logprob_fn, T, stride). The model can read at most T tokens. logprob_fn(window) takes a list of up to T ids and returns an array of length len(window) - 1 whose entry j is $\log p(\text{window}[j+1] \mid \text{window}[0..j])$ . Windows start at 0, stride, 2*stride, ..., and each is ids[b : b + T]. Every target position i from 1 to len(ids) - 1 must be scored exactly once, by the first window that contains it at a position other than 0. Stop after the first window that reaches the end of ids. Return the mean negative log-likelihood over all targets. Assume 1 <= stride <= T - 1.

import numpy as np

def sliding_window_nll(ids, logprob_fn, T, stride):
    raise NotImplementedError

Hidden tests, hints, and AI help, from $1.99 a month.

Bits per byte, the unit that survives a tokenizer change

Loss per token depends on what a token is. GPT-2's tokenizer packs about four characters of English into a token, so its loss per token is roughly four characters' worth of surprise, and a character model and a BPE model can't be compared on that scale. §0.5 gave the fix, bits per byte, the total information converted to bits and divided by the text's length in UTF-8 bytes, $n_{\text{bytes}}$ ,

\operatorname{BPB} = \frac{\sum_{t=1}^{n} -\log p_\theta(w_t \mid w_{\lt t})}{\ln 2 \cdot n_{\text{bytes}}} = \frac{\bar{\ell}}{\ln 2}\cdot\frac{n}{n_{\text{bytes}}} .

Any model that assigns probabilities to whole strings has one, whatever its tokenizer, and it is the model's compression rate, since an arithmetic coder driven by the model needs within a couple of bits of $\sum_t -\log_2 p(w_t \mid w_{\lt t})$ in total. That makes it the fair currency for comparing language models (the Pile and many later benchmarks report it, Gao et al., 2020), and it is the number GPT-2's largest model reported on Wikipedia's enwik8, 0.93 bits per byte without ever training on it (Radford et al., 2019).

In a training repository two details trip people up. Count bytes, not characters, since "é" is one character and two bytes. And exclude tokens that represent no text, such as EOT or padding, which have zero bytes and whose loss shouldn't be charged to the text. nanochat's evaluation does exactly this, giving each token id a byte count and zero for special tokens, which is what evaluate_bpb below does too. Here are our models in bits per byte, against a general-purpose compressor.

Python · cell 3

zlib at its strongest setting needs 3.91 bits per byte for the sonnets, a little more than our tiny models, which is the compression view in action. The last two lines are a hypothetical worth understanding. A BPE model with a loss of 3.30 nats per token sounds much worse than a character model at 1.62, but if each of its tokens covers 4.2 bytes it is spending about 1.13 bits per byte, far better than the character model's 2.34. Per-token losses are only comparable between models with the same tokenizer.

Try it on the real tiny GPT. Type anything into the widget below, and every character is colored by how surprised the model was to see it. A sonnet it trained on costs about 1.9 bits per character, a held-out sonnet about 2.3, modern prose about 3, and scrambled letters more than 6.

Downstream evaluation, few-shot prompts and multiple choice

Low perplexity is necessary but not sufficient. What we usually want is for the model to do things. GPT-2 and GPT-3 showed how to measure that without any fine-tuning, by framing a task as text the model continues (Radford et al., 2019; Brown et al., 2020). A zero-shot prompt states the task and asks. A few-shot prompt first shows $k$ solved examples, then the question, and the model's continuation is the answer. Few-shot accuracy rises with model size far faster than zero-shot accuracy, which was GPT-3's headline result. It is also sensitive to the prompt's wording, the choice and order of the examples, and the answer format, so a careful evaluation fixes all of them and reports how much they matter. §6.8 looks at why prompting works at all, and at how far those choices move the score.

Generating free text and checking it is one way to score. For multiple-choice tasks there is a cleaner way that needs no generation at all. Compute how likely the model finds each option as a continuation of the context, and pick the most likely. For an option $c$ made of tokens $c_1, \dots, c_m$ ,

\log p(c \mid \text{ctx}) = \sum_{i=1}^{m} \log p_\theta(c_i \mid \text{ctx}, c_{\lt i}) .

The catch is length. Every extra token adds a negative term, so the raw sum favors short options. Three normalizations are common, and they really do give different answers.

scoring rule	formula	bias it fixes	bias it adds
sum	$\log p(c \mid \text{ctx})$	none	favors short options
per token or per character	$\log p(c \mid \text{ctx}) / \lvert c \rvert$	length	favors long, fluent, generic options
mutual information	$\log p(c \mid \text{ctx}) - \log p(c \mid \text{neutral prompt})$	options that are likely in any context	noisier, needs a second pass

EleutherAI's lm-evaluation-harness reports the sum as acc and the per-character version as acc_norm (it divides by the option's length in characters), and the GPT-2 reproduction scores HellaSwag (Zellers et al., 2019) with the average loss per token of each ending. Notice that every option starts with its own leading space. With a BPE tokenizer, " doom" and "doom" are different tokens, and web text almost always has the space attached to the following word, so the context must end without its trailing space and each option must begin with one, or the scores measure a tokenization the model rarely saw. The mutual-information rule is from Holtzman et al. (2021), who called the problem "surface form competition", meaning that many strings with the same meaning split the probability among themselves. Let's watch the rules disagree on held-out lines, using the trigram, our best model for this sonnet.

Python · cell 4

The sum picks the shortest option both times, " doom." and " was.". Per-character normalization picks the longest option both times. The mutual-information rule ties the real line with a generic phrase in the first question and misses in the second. Three reasonable rules, three different behaviors, and none of them rescues a model that knows spelling but not meaning. With real models and real benchmarks the effect is the same in kind. On HellaSwag, EleutherAI's harness scores OpenAI's GPT-2 small at 28.9 percent with the sum and 31.1 percent per character, and GPT-2 XL at 40.0 against 50.9, as recorded in the HellaSwag script of Karpathy's build-nanogpt. An eleven-point swing from the scoring rule alone is why reports must say which rule they used.

Core

Likelihood-based multiple-choice scoring

Write pick_option(option_token_logprobs, option_texts, method). option_token_logprobs[k] is the array of per-token log-probabilities of option k given the context, and option_texts[k] is that option's string. Return the index of the chosen option under method, which is "sum" (total log-probability), "token" (mean per token), or "char" (total divided by the number of characters in the text, as in acc_norm). Ties go to the lowest index.

import numpy as np

def pick_option(option_token_logprobs, option_texts, method):
    raise NotImplementedError

Hidden tests, hints, and AI help, from $1.99 a month.

Judging samples

Likelihood metrics never look at what the model writes, and §6.4 showed that the most likely text is the worst text. Judging samples needs its own tools, and each catches a different failure.

Diversity statistics such as distinct- $n$ (the fraction of $n$ -grams in the output that are unique) and repetition rates catch the loops of greedy decoding at a glance.
Likelihood under a stronger model catches gibberish that diversity statistics reward.
Distribution comparisons such as MAUVE (Pillutla et al., 2021) embed many generations and many human texts and measure how far apart the two clouds are, penalizing both blandness and nonsense.
Human judgment, or a strong model used as a judge with a careful rubric, is the final word on quality, coherence, and helpfulness. Model judges have known biases, such as favoring the answer shown first, longer answers, and their own outputs, which is why LLM-as-judge evaluation calibrates them against human ratings. And the most useful evaluation set is usually one you build yourself from real failure cases, the habit behind eval-driven development.

Python · cell 5

Greedy decoding is caught instantly, with 6 percent unique words. But full sampling at temperature 1 beats Shakespeare on distinct- $n$ while writing words that don't exist. No single number is enough. That's the rule for this whole section, and for evaluation at large.

⚠️ Ways an evaluation lies

Test data that leaked into training, through duplicates or benchmark questions posted on the web, inflates everything (§8.3 covers contamination checks). Comparing per-token losses across tokenizers compares units, not models. Few-shot scores swing with prompt format, so changing the template between two models invalidates the comparison. Evaluating a chat model without its chat template, or a base model with one, measures the wrong thing. Leaving dropout on (model.train()) during evaluation adds noise and bias. And a single benchmark number hides variance. With 1,000 questions, the standard error of an accuracy near 50 percent is about 1.6 points, so differences of one or two points are within noise (§0.11 builds the intervals and paired tests that settle such comparisons).

In PyTorch

Here is evaluation code in the style of a real training repository, run on a tiny bigram model trained in a few hundred steps so the numbers mean something. evaluate_bpb accumulates total nats and total bytes over batches and masks special tokens, as nanochat does. score_options scores all multiple-choice options in one padded batch, which is how harnesses evaluate thousands of questions quickly.

🔥 In PyTorchfor a machine with PyTorch

The mask in score_options is where these functions usually go wrong. Only the option's own tokens may count, never the context and never the padding, and the log-probability of the token at position $i$ comes from the logits at position $i - 1$ .

🎯 Interview questions on this section

What is perplexity, and how do you interpret a perplexity of 20?

The exponential of the average negative log-likelihood per token on held-out text, $\exp(-\frac{1}{n}\sum_t \log p(w_t \mid w_{\lt t}))$ . It is the effective branching factor, so a perplexity of 20 means the model is, on average, as uncertain as if it were choosing uniformly among 20 tokens. It only compares models that share a tokenizer and an evaluation set.

Why use bits per byte, and how do you compute it?

Loss per token depends on the tokenizer, so models with different vocabularies can't be compared per token. Bits per byte divides the total information, $\sum_t -\log p / \ln 2$ , by the UTF-8 byte length of the text, which any string model shares. Exclude special tokens that represent no text. It equals the compression rate an arithmetic coder driven by the model would achieve, so a better language model is literally a better compressor.

How do you evaluate a model whose context is shorter than the documents in your test set?

With a sliding window. Advance a window of $T$ tokens by a stride $s \lt T$ and score only the tokens that are new in each window, so each gets at least $T - s$ tokens of context. Non-overlapping windows ( $s = T$ ) are cheapest but give the first tokens of each window almost no context, which inflates perplexity. A stride of 1 is exact but needs one forward pass per token.

How does likelihood-based multiple-choice scoring work, and what normalization would you use?

Compute $\log p(\text{option} \mid \text{context})$ by summing the log-probabilities of the option's tokens, and pick the largest. The raw sum favors short options, so harnesses also report a length-normalized score (per token, or per character as in acc_norm), which can instead favor long generic options. A mutual-information score subtracts the option's likelihood under a neutral prompt to cancel options that are likely in any context. I'd report the normalization explicitly and check that conclusions don't flip between rules.

What are few-shot and zero-shot evaluation, and what makes them fragile?

Zero-shot frames the task in the prompt and reads the continuation. Few-shot first shows $k$ solved examples. Neither updates the weights. Results depend on the instruction wording, which examples are shown and in what order, the answer format, and whitespace and tokenization at the prompt boundary, so you fix all of them, report them, and ideally average over several prompts.

Your new model has lower validation loss but its samples look worse. How can that be?

Loss measures average prediction of held-out text, not the quality of decoded samples. Possible causes are a decoding change (temperature, truncation), validation data closer to the training distribution than the prompts you are sampling from, a model that improved on frequent easy tokens while getting worse on the rare ones that matter for coherence, or contamination making the loss look better than it is. Check samples with diversity statistics and human or model-based judgment, and measure loss on a validation set that matches the use case.

🧭 What to remember

Always compare against baselines. On our tiny corpus a smoothed trigram even beats the GPT, which tells you the model is data-limited.
Perplexity is $e^{\bar{\ell}}$ , an effective branching factor. Evaluate long texts with a sliding window so every token gets real context.
Bits per byte divides total bits by UTF-8 bytes, excluding special tokens. It compares models across tokenizers and equals the model's compression rate.
Few-shot prompts and likelihood-based multiple choice measure what a base model can do without fine-tuning. Report the normalization, since sum, per-character, and mutual-information scores can pick different answers.
Sample quality needs its own measures, from distinct- $n$ for loops to likelihood under a stronger model and human judgment for nonsense.
Next, §6.7 scales the toy up, from the GPT-2 124M recipe to the changes that make a LLaMA-style model.

Radford et al. (2019). Language Models are Unsupervised Multitask Learners. Zero-shot evaluation and bits per byte on enwik8, in Table 3.
Brown et al. (2020). Language Models are Few-Shot Learners. Zero-, one-, and few-shot evaluation, and how it scales with model size.
Gao et al. (2020). The Pile: An 800GB Dataset of Diverse Text for Language Modeling. A benchmark that reports bits per byte across 22 domains, and why.
Holtzman et al. (2021). Surface Form Competition: Why the Highest Probability Answer Isn't Always Right. Why the most probable option is not the best answer, and mutual-information scoring.
Zellers et al. (2019). HellaSwag: Can a Machine Really Finish Your Sentence?. The sentence-completion benchmark used to track GPT-2 reproductions.
Biderman et al. (2024). Lessons from the Trenches on Reproducible Evaluation of Language Models. How evaluation choices change results, from the lm-evaluation-harness team.
Pillutla et al. (2021). MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontiers. Comparing the distribution of generated text with human text.
Hugging Face. Perplexity of fixed-length models. The sliding-window evaluation, with code.

Zero to Frontier is made for larger screens