Your fine-tuned model looks wonderful in the demo. It answers in a warm, confident voice, formats its answers with headings and bullet points, and the team's favorite prompts all come out better than before. Then it ships. A customer asks for "a summary in under 50 words" and gets 180. A dashboard shows MMLU down four points from the base model. And an automatic judge that the team trusted rates it far above the previous version, mostly, it later turns out, because its answers got longer.

Each of those failures is a measurement you did not take, or took badly. This section builds the evaluation that should sit around every fine-tuning run. Verifiable instruction-following checks, regression tests on base capabilities with error bars that account for sampled answers and grouped questions, LLM judges with their biases and a statistically sound way to correct them with a few human labels, human review that measures what you care about, and a small, sharp eval set of your own. It closes the chapter, and its tools will reappear in every chapter of Part IV, because a post-training method is only as good as the evaluation that says it worked.

What to measure

A fine-tuned model can be good or bad in several independent ways, and no single number sees all of them.

The first and the third rows are cheap, objective, and catch most regressions, so they should run on every checkpoint (§14.4). The second is where chat models are actually compared, and it is the easiest to fool. The last is what decides whether the model is useful to you.

Verifiable instruction following

Asking a judge whether a response "follows the instructions" is slow and noisy. Zhou et al. (2023) noticed that many instructions are verifiable by code, such as "write more than 400 words", "mention the keyword AI at least 3 times", "do not use any commas", "wrap your entire answer in JSON", or "end your response with this exact phrase". Their benchmark, IFEval, uses 25 types of such instructions across 541 prompts, each prompt carrying one or more of them, and scores responses with no model in the loop.

It reports four numbers. Prompt-level accuracy counts a prompt as passed only if every instruction in it is followed. Instruction-level accuracy counts instructions individually. Each comes in a strict version, the raw check, and a loose version that also accepts the response after any combination of three transformations, removing markdown emphasis markers (* and **), dropping the first line (an intro such as "Sure, here it is"), and dropping the last line (an outro such as "Hope this helps"). With the identity, that makes 8 variants, and an instruction passes loosely if any of them passes. Loose scoring reduces false negatives from harmless decoration at the cost of some false positives.

The emphasis markers make the strict "ends with" check fail on a response that clearly followed the instruction, and the loose version accepts it. The word limit and the comma rule both fail strictly because of the chatty first line (its words, and its comma in "Sure, here it is"), and both pass loosely once the first and last lines are dropped. That is reasonable for the intro, and it is also the false-positive risk of loose scoring, since the dropped last line here was part of the answer the user asked for. IFEval became a standard part of post-training suites (Tulu 3 used it as an evaluation and as a source of verifiable rewards for RL, §18.3), and the idea generalizes. Any constraint your product depends on, a JSON schema, a length limit, a required disclaimer, a language, should have a checker that runs on every checkpoint.

import re
import itertools

def check_inst(resp, inst):
    raise NotImplementedError

def ifeval_scores(responses, instructions):
    raise NotImplementedError

Regression testing on base capabilities

Fine-tuning can quietly cost knowledge and skills (§14.4), so every fine-tuned model should be compared with its base on a fixed suite of capability benchmarks, chosen to cover what you must not lose (knowledge, math, code, reasoning, the languages you serve). Two details decide whether the comparison means anything.

Score the models the same way. A base model is usually evaluated few-shot with a plain prompt, and an assistant zero-shot inside its chat template. Comparing those two numbers measures the formats as much as the models. Run both models with the same prompts, and for the fine-tuned model also report its score in its chat template, since that is how it will be used. Likelihood-based multiple choice (§6.6) is less sensitive to format than generation followed by answer extraction, which can fail on a model that phrases its answers differently.

Put error bars on the difference. Section 0.11 built the statistics, and here is the short version. An accuracy on $n$ independent questions has standard error $\sqrt{\hat p(1-\hat p)/n}$, about 1.6 points at $n = 1{,}000$ near 50%. Two models answer the same questions, so compare them through the per-question differences $d_i = a_i - b_i$ of their scores $a_i$ and $b_i$, whose mean has standard error $\operatorname{SE}(\bar d) = s_d/\sqrt n$, with $s_d$ the standard deviation of the $d_i$ and $s_d^2 = \operatorname{Var}(a) + \operatorname{Var}(b) - 2\operatorname{Cov}(a, b)$. Questions differ in difficulty, so the covariance is large and pairing removes much of the noise. The paired bootstrap and McNemar's test (Exercise 0.11e) turn this into an interval and a p-value. And to detect a true gain $\Delta$ with 80% power you need about $n = \big((1.96 + 0.84)\,s_d/\Delta\big)^2$ questions (Exercise 0.11f), twice as many as it takes to measure the difference to within $\pm\Delta$. The widget shows both views on simulated runs of an eval.

So a fine-tuned model that scores 1.5 points above its base on a 1,000-question benchmark has shown nothing until the paired interval excludes zero. That much is true of any classifier. Evaluating a generative model that you just fine-tuned adds three sources of noise that a textbook interval leaves out, and each one makes the honest error bar wider than the naive one.

Sampled answers, clustered questions, and training seeds

Sampled answers. Run a model at a temperature above zero and each question's grade is itself random. Say you draw $K$ answers per question and average their grades into a per-question score $x_i$. If question $i$ has true pass rate $p_i$, then given $p_i$ the score $x_i$ is a mean of $K$ Bernoulli draws, with $\mathbb{E}[x_i \mid p_i] = p_i$ and $\operatorname{Var}(x_i \mid p_i) = p_i(1-p_i)/K$ (§0.4). Averaging over questions, $\operatorname{Var}(x_i) = \mathbb{E}[x_i^2] - \mathbb{E}[x_i]^2 = \mathbb{E}\big[\operatorname{Var}(x_i \mid p_i) + p_i^2\big] - \mathbb{E}[p_i]^2$, which is the law of total variance, and it splits the spread of $x_i$ across questions into two parts,

In plain words, more samples per question shrink only the sampling term, and more questions shrink both. You can estimate $\sigma_w^2$ as the average variance of the $K$ grades within each question, and $\sigma_b^2$ as the variance of the $x_i$ minus $\sigma_w^2/K$, which tells you whether to buy questions or samples (Miller 2024). The common mistake is to treat all $nK$ grades as independent, which divides by $\sqrt{nK}$ and makes the error bar too small, because samples of the same question share its difficulty. Greedy decoding removes the sampling term but measures a different quantity, the model's single most likely answer, and even greedy outputs are not perfectly reproducible on GPUs, where batched kernels change the order of floating-point sums.

Clustered questions. Many benchmarks group questions, several about one reading passage, one coding problem with many tests, or one generated template with different numbers. Questions in a group succeed or fail together, so they carry less information than the same number of independent questions. The clustered standard error sums the residuals within each cluster $c$ before squaring,

which equals the plug-in standard error $\sqrt{\sum_i (x_i - \bar x)^2}/n$ when every cluster has one question, and grows when questions in a cluster move together. For two models, apply it to the differences $d_i$.

Training seeds. Every interval above measures the noise of the questions. The fine-tuning run is random too. Dodge et al. (2020) fine-tuned BERT hundreds of times on each of four small GLUE tasks, changing only the random seeds that set the new classifier's initialization and the order of the training data. The scores moved so much that the best of many reruns of plain BERT rivaled results reported for newer models, and the two seeds contributed comparably to the spread. When a decision rests on a gain of a point or two, train two or three seeds and report their spread next to the question-level interval.

Here are the first two effects on a simulated reading benchmark, 100 passages with 5 questions each, graded on 4 sampled answers per question.

Treating the 2,000 sampled answers as independent claims an error bar of 1.1 points. The honest per-question value is 1.5, and once the passages are accounted for it is 2.7, two and a half times the naive number. The sampling term is worth reducing up to a point. Going from 1 sample per question to 4 shrinks the per-question error from 2.2 to 1.5 points, while 64 samples reach only 1.3, because the between-question term does not budge. Past a handful of samples, buy questions.

import numpy as np

def eval_error_bars(S, clusters):
    raise NotImplementedError

def se_for_k(S, k_new):
    raise NotImplementedError

LLM judges and their biases

Open-ended answers have no checker, so the field grades them with a strong model. Zheng et al. (2023) introduced MT-Bench, 80 multi-turn questions in 8 categories (writing, roleplay, extraction, reasoning, math, coding, and two kinds of knowledge), graded by GPT-4, and showed that GPT-4's judgments agreed with human experts' over 80% of the time, the same level at which humans agreed with each other. Judges come in two forms. Single-answer grading asks for a score, say 1 to 10, and is cheap but drifts in scale. Pairwise comparison shows two responses and asks which is better, which is more reliable and is what AlpacaEval and Chatbot Arena-style evaluations report as win rates.

The same paper catalogued the biases, and every one of them matters for SFT.

• Position bias. Judges tend to prefer the response shown first (or, for some judges, second). The fix is to judge every pair in both orders and count a win only when the verdicts agree, or average the two.
• Verbosity bias. Judges prefer longer answers, even when the extra length adds nothing. SFT data and tricks such as NEFTune (§14.4) often make answers longer, so a judge can reward the length rather than the content. AlpacaEval's length-controlled win rate fits a regression of the judge's preference on the length difference and reports the win rate with that term set to zero, which raised its Spearman correlation with Chatbot Arena's human rankings from 0.94 to 0.98 (Dubois et al. 2024).
• Self-enhancement bias. A judge may favor responses from its own model family, so do not let a model judge its own outputs or its distillation students without checking against humans.
• Limited reasoning. A judge can be fooled on math and code it cannot solve itself. Reference-guided grading, giving the judge a correct answer, helps.

Here is a simulated judge with a position bias and a length bias, comparing a fine-tuned model A that is genuinely better against a model B that writes 30% longer.

Shown first, A wins 61% of comparisons, and shown second only 41.5%, a 19-point swing from position alone. Averaging both orders removes the position bias but leaves 51%, nearly a tie, because B's extra length cancels A's better content. Controlling for length recovers a clear preference for A, 55%, close to the 56% that quality alone would give. Judges are indispensable and treacherous in equal measure, and the full treatment of building and validating them is a subject of its own, using language models as evaluators. Correcting a judge with a few hundred human labels is covered later in this section.

def pairwise_summary(ab, ba):
    raise NotImplementedError

Correcting a judge with a few human labels

Swapping orders and controlling for length remove the biases you can name. A judge also makes errors you cannot name, and those are not symmetric. If it wrongly sides with model A more often than with model B, its win rate for A is biased however many prompts you judge, and more prompts only make the wrong number more precise. Humans are unbiased by definition here (their preference is what we want to estimate), but expensive. The way out is to use both.

Judge all $N$ prompts, and have humans label a random subset of $n$ of them as well. On that subset you can see the judge's errors, and their average is exactly the judge's bias. So estimate the win rate as the judge's average over everything, corrected by its average error on the labeled subset,

with $j_i$ the judge's verdict on prompt $i$ (1 if it prefers A) and $h_i$ the human's. In plain words, the judge supplies the volume and the humans supply the correction. The first term has mean equal to the judge's win rate, the second has mean equal to the human rate minus the judge's, so the sum has mean equal to the human rate, whatever the judge's bias. This is prediction-powered inference (Angelopoulos et al. 2023), and Boyeau et al. (2024) applied it to model evaluation. The better the judge agrees with humans, the smaller $\operatorname{Var}(h - j)$, and the more each human label is worth. A useless judge leaves you exactly where humans alone would.

The judge alone is precise and wrong, 62% for a true 55%, and no amount of judging fixes it. The 150 human labels alone are right on average but uncertain by 4.2 points. Together they are right on average with a spread of 3.2 points, as good as about 260 human labels, and the interval covers the truth at its promised rate. This judge agrees with humans only 84% of the time. A judge that agrees 95% of the time would make each human label worth several, which is why teams invest in improving the judge and keep a human-labeled slice of every evaluation.

import numpy as np

def ppi_mean(judge_all, judge_lab, human_lab, z=1.96):
    raise NotImplementedError

Human review

Humans remain the reference for what "good" means, and they are expensive and noisy in their own ways. Gudibande et al. (2023) showed the classic failure. Crowd raters judged imitation models nearly as good as ChatGPT because the style was right, and missed that the facts were not (§14.2). Good human evaluation is designed to avoid exactly that.

• Blind and randomized. Raters see two anonymized responses in random order, never model names.
• A rubric, not a vibe. Define what to judge (correctness, completeness, instruction following, tone, safety) and in what priority, with examples of each grade. Ask for correctness checks on factual and technical prompts, by raters who can check them.
• Calibration and agreement. Start with a calibration round on shared items, measure inter-annotator agreement, and revise the guidelines until it is acceptable. Cohen's $\kappa$ corrects raw agreement for the agreement two raters would reach by chance,

where $p_o$ is the observed fraction of items on which they agree and $p_e = \sum_c p_{1c}\,p_{2c}$ is the agreement expected if each rater labeled independently with their own label frequencies $p_{1c}$ and $p_{2c}$. In plain words, $\kappa = 0$ is chance and 1 is perfect agreement. Two raters who both say "A is better" 90% of the time agree 82% of the time by chance alone, so 85% raw agreement is barely above chance.

• Enough items. A win rate from 100 comparisons has a standard error of about 5 points, so a difference of a few points needs several hundred.

def cohen_kappa(r1, r2):
    raise NotImplementedError

Building a small eval set of your own

Public benchmarks tell you how your model compares with others. A small, private eval set tells you whether it does your job. Every serious team has one, and it is worth more per item than anything public. Here is how to build it.

1. Start from real usage. Sample prompts from production logs or user studies (with the privacy review that requires), or have the people who will use the model write the requests they actually make. Synthetic prompts are fine for coverage, but real ones carry the phrasing and the hard cases.
2. Read outputs before you write metrics. Run the current model on a few hundred prompts, read the outputs, and write down every failure you see. Group the failures into categories. The categories become your eval's structure, and their frequencies tell you what to fix first. Practitioners describe this error analysis as the highest-value step in building evals.
3. Stratify and size it. Give each category enough items to detect the differences you care about, using the sample-size formulas of §0.11. A few hundred items in total, with 30 to 100 per important category, is typical.
4. Write the grader for each item. Prefer verifiable checks (exact answers, unit tests, format validators, the IFEval style). Where judgment is needed, write a rubric and a reference answer, and validate an LLM judge against human labels on a subset before relying on it, then keep that labeled subset to correct the judge's win rate as above.
5. Include regressions and refusals. Keep items the model must keep doing well (formats, facts, behaviors that past versions got right), harmful requests it must decline, and benign ones that superficially resemble harmful requests, which it must not decline.
6. Keep it clean and frozen. Decontaminate your training data against the eval set (§14.2), never train on it, version it, and add new items as new failures appear rather than editing old ones, so that scores stay comparable across versions.

Even with this pessimistic calculation (it ignores the correlation that pairing exploits), a category near 90% accuracy needs about 280 items to see a 5-point change, and one near 60% needs about 740. That is why teams track a few categories closely and accept wider error bars elsewhere, and why the paired analysis, which can cut those numbers substantially, is worth the trouble.

The real way

The evaluation harness most open models are scored with is EleutherAI's lm-evaluation-harness. It runs the same tasks for a base model and its fine-tuned version, with or without the chat template. It is shown, not run, because it downloads models and data.

# base model, few-shot, the way base models are usually reported
lm_eval --model hf --model_args pretrained=meta-llama/Llama-3.1-8B,dtype=bfloat16 \
        --tasks mmlu,gsm8k --num_fewshot 5 --batch_size auto --output_path results/base --log_samples
# the fine-tuned model, same tasks and shots, then again in its chat template
lm_eval --model hf --model_args pretrained=./my-sft-model,dtype=bfloat16 \
        --tasks mmlu,gsm8k --num_fewshot 5 --batch_size auto --output_path results/sft --log_samples
lm_eval --model hf --model_args pretrained=./my-sft-model,dtype=bfloat16 \
        --tasks ifeval,gsm8k --apply_chat_template --fewshot_as_multiturn --batch_size auto --output_path results/sft-chat --log_samples

--log_samples saves per-item results, which is what makes paired error bars possible. Compare models item by item, with the paired bootstrap or McNemar's test of §0.11 and the clustered errors above, rather than comparing two headline numbers.

Read more

• Zhou et al. (2023). Instruction-Following Evaluation for Large Language Models. IFEval, its 25 verifiable instruction types, and the strict and loose metrics.
• Zheng et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. MT-Bench, judge agreement with humans, and the catalogue of judge biases.
• Dubois et al. (2024). Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators. Regression-based length control for win rates.
• Miller (2024). Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations. Paired differences, clustered standard errors, and power analysis for evals.
• Angelopoulos et al. (2023). Prediction-Powered Inference. Valid confidence intervals from a few gold labels and many model predictions.
• Boyeau et al. (2024). AutoEval Done Right: Using Synthetic Data for Model Evaluation. Prediction-powered inference applied to evaluating models with an automatic judge.
• Dodge et al. (2020). Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping. How much fine-tuning results move with nothing but the random seed.
• Gudibande et al. (2023). The False Promise of Imitating Proprietary LLMs. Why human raters can be fooled by style, and how targeted evaluations reveal it.
• EleutherAI. lm-evaluation-harness. The standard open harness for benchmark evaluation, with chat-template support.

Watch next

▶

▶