How a model that completes documents becomes an assistant that answers you. The map of post-training, the data it runs on, the loss it optimizes, and the full and parameter-efficient ways to fine-tune, with every mask, gradient, and byte accounted for.

Type "What is the capital of France?" into a freshly pretrained language model and you rarely get "Paris." A typical continuation looks more like this.

What is the capital of France?
What is the capital of Germany?
What is the capital of Italy?
Answers: 1. Paris 2. Berlin 3. Rome

Nothing is broken. The model was trained to continue documents (§6.3), and on the web a line ending in a question mark is very often followed by another question, because it sits in a quiz, a worksheet, or a list of interview prompts. The model knows that Paris is the capital of France. The fact is right there in the answer key it wrote. What it does not know is that you are a person asking it a question and expecting it to answer, briefly, and then stop.

You have already met the workaround. Section 6.8 showed that you can get an answer out of a base model by writing the beginning of a document whose most natural continuation is that answer, a few "Q, A" pairs followed by your question, and that with a careful system-style preamble and three example conversations a base model can come surprisingly close to a chat model (Lin et al. 2023). It also showed the limits. Every request pays for its examples in context, the model still has no notion of a turn or of when to stop, and its behavior shifts with the order and wording of the examples. Post-training makes the workaround permanent. Instead of putting demonstrations in every prompt, we put them in the weights.

Closing the gap between a document completer and an assistant is the job of post-training, everything that happens to a model after pretraining and before you talk to it. This section opens Part IV of the book, so we'll draw the whole map first, the stack of supervised fine-tuning, preference tuning, and reinforcement learning that Chapters 14 to 19 walk through. Then we'll ask the question that organizes the field. How much of what an assistant can do is created by post-training, and how much was already in the base model, waiting to be selected?

A base model is a model of documents

Recall what pretraining optimizes. For a corpus of documents $\mathbf{w} = (w_1, \dots, w_T)$, the model with parameters $\theta$ learns

the probability of the whole document, factored token by token (§3.1). Give it a prompt $x$ and ask for a continuation $y$, and you get $p_\theta(y \mid x)$, the distribution of whatever came next in documents that started like $x$. In plain words, a base model answers the question "what text usually follows this text?", which is a different question from "what would a helpful assistant say to this person?".

The two questions have the same answer surprisingly often, which is why few-shot prompting works at all (GPT-3 was used this way for years). But they diverge exactly where it matters. Documents that start with a request are rarely followed by a careful answer to that request. Documents rarely end where an answer should end. And the web contains every kind of author, so a model of the web will happily continue in the voice of a troll, a spammer, or a confident crank.

Here is a useful way to picture it. A document is written by someone, a forum regular, a textbook author, a quiz page, a customer support agent, and the model never sees who. So it has to infer the author from the text so far. Write $z$ for this hidden author, or persona, out of $K$ possibilities. Then the base model's prediction is a mixture,

In plain words, the model guesses who is writing, and then writes like them. The first factor, $p_\theta(z \mid x)$, is how likely each persona is given the prompt. The second is how that persona would continue. This is a picture, not a theorem. Nobody has found a clean persona variable inside a Transformer. But it is a useful picture with some evidence behind it, and you have seen its twin. In §6.8 the hidden variable was a document-level concept (the task), and each few-shot demonstration in the prompt was evidence about which concept was in play (Xie et al. 2021). A persona is the same kind of latent variable, and the view of language models as models of the agents who produce text (Andreas 2022) argues for it directly. The one change is where the evidence comes from. A prompt updates the posterior for one request and forgets it afterward. Fine-tuning data updates it once, in the weights, for every request that follows. And the picture makes one prediction we can test with arithmetic.

Why selecting a behavior is cheap

Suppose one of the personas is exactly the assistant we want, but the base model gives it only 1% of its prior weight. We fine-tune on $n$ demonstrations $y_1, \dots, y_n$ written by that persona. In the mixture picture, the ideal learner does not need to learn how to write like an assistant. It already knows. It only needs to update its belief about who is writing. By Bayes' rule, the posterior weight of persona $z$ after the demonstrations is

Compare the assistant persona $a$ with any rival persona $b$ by the log of their odds,

Each demonstration adds one term to the sum. Since the demonstrations really come from $a$, the expected value of each term is

the KL divergence from §0.5. This is the argument of §6.8 for few-shot prompts, now applied to fine-tuning demonstrations. So the log odds grow linearly in $n$ at a rate equal to the KL between the two personas. To push the posterior odds against $b$ from the prior up to 99 to 1, you need about

demonstrations. In plain words, the number of examples you need grows only with the logarithm of how unlikely the behavior was to begin with, and shrinks with how distinguishable it is. A persona with a prior of one in a million needs only $\log 10^6 \approx 14$ nats of evidence to draw level with its rivals, plus $\log 99 \approx 4.6$ to pull ahead, and if each response carries a few nats, that is a handful of examples.

Now compare learning a behavior that is not in the mixture. Then there is nothing to select, and the model has to estimate a whole new distribution over responses. For a categorical distribution over $V$ outcomes estimated from $n$ samples by counting (with a little smoothing, so that no outcome gets probability zero and the KL stays finite), the expected KL from the truth falls like $(V-1)/(2n)$ for large $n$, a classical asymptotic result. It comes from a second-order expansion. With $\hat p_i$ the estimate of $p_i$, $D_{\mathrm{KL}}(p \Vert \hat p) \approx \frac12\sum_{i} (\hat p_i - p_i)^2/p_i$, and since each count is binomial, $\mathbb{E}[(\hat p_i - p_i)^2] = p_i(1-p_i)/n$, so the expectation is $\frac12\sum_i (1-p_i)/n = (V-1)/(2n)$. Every new free parameter costs data, and the cost falls only like $1/n$, not exponentially.

Let's watch both happen. Six personas, each a distribution over 40 possible "responses" (a stand-in for the space of real responses), and a base model that gives the assistant persona a prior of 1%.

The hardest rival is persona 4, which gives up about 1.15 nats per demonstration and starts with nine times the prior, so the formula predicts about six demonstrations. Now simulate it, averaging over 20 random draws of the demonstrations, and compare with a learner that ignores the personas and fits the 40 response probabilities from counts.

Eight demonstrations take the reweighting learner from 1.6 nats to 0.003, with nearly all of the posterior (0.985) on the assistant. The learner that builds the distribution from scratch is still at 0.64 nats after eight examples and needs a thousand to reach 0.02, almost exactly the $(V-1)/(2n) = 39/2048 \approx 0.019$ the asymptotic formula predicts. And if the behavior you want is not one of the base model's personas at all, reweighting gets stuck.

Three regimes in one plot. Selecting an existing behavior is fast and exact. Selecting when nothing fits plateaus at about 1.07 nats, however much data you add. Learning from scratch always works, eventually, at a rate of $1/n$. Real fine-tuning is gradient descent on all the weights, not Bayesian persona selection, so it can do some of all three. But the toy explains the central empirical fact of post-training. Changing which thing a model does takes very little data. Teaching it a genuinely new thing takes a lot.

import numpy as np

def persona_posterior(log_lik, prior):
    raise NotImplementedError

def demos_needed(P, prior, a, ys, threshold=0.99):
    raise NotImplementedError

What post-training actually changes

Lab reports and ablations agree that post-training changes these things.

• the format of the conversation, the chat template with its roles and turn markers (§2.5), and, crucially, when to stop,
• the persona, a consistent, helpful, honest voice that answers the question asked rather than continuing the document,
• what to refuse and what not to, which is a policy decision as much as a technical one (§19.1),
• the style of reasoning and the length of answers, including whether to think step by step, and for reasoning models whether to think for thousands of tokens (§18.5),
• new protocols, calling tools, following a system prompt, and honoring an instruction hierarchy,
• some skills, most clearly in math and code, where reinforcement learning against checkable answers keeps improving the model long after the format is learned.

What it does not do well is add knowledge. Gekhman et al. (2024) fine-tuned models on closed-book question answering while controlling what fraction of the training facts were unknown to the base model. The unknown facts were learned much more slowly than the known ones, and as they were eventually fit, the model's tendency to hallucinate on other questions rose roughly linearly. Facts are mostly put in during pretraining. Post-training teaches the model to use them, and teaching it new ones by fine-tuning has side effects.

The post-training stack

Every frontier lab's pipeline is some arrangement of the same few stages. Here they are with the data each one consumes and the objective it optimizes, and with the chapter where we build it.

Written as objectives, the first and the third make the shape of the whole field visible. From here on we write the model being post-trained as $\pi_\theta$, a policy that maps a prompt to a distribution over responses, which is the notation Chapters 15 to 17 use. It is the same network as the $p_\theta$ above, with $\pi_\theta(y \mid x) = \prod_t \pi_\theta(y_t \mid x, y_{<t})$. SFT maximizes

over a dataset $\mathcal{D}$ of prompts $x$ and demonstrations $y$. In plain words, make the model's distribution put as much probability as possible on the responses a good demonstrator wrote. RLHF instead maximizes

with a reward $r$, a frozen reference policy $\pi_{\text{ref}}$ (usually the SFT model), and a penalty strength $\beta$. In plain words, produce responses that score well, but do not wander far from the model you started with. The difference that matters is where $y$ comes from. In SFT the responses come from the dataset, someone else's distribution, and the model only imitates. In RL they come from the model itself, so it learns from the consequences of its own outputs. That one difference is why RL can improve past the demonstrations and why it needs a reward it cannot easily fool (§16.5). Section 15.7 builds this view of a language model as a policy.

Why does SFT almost always come first? Three reasons. RL needs a starting policy that already produces sensible responses most of the time, or the rewards it collects are almost all zero and nothing is learned (an exploration problem). Preference data is collected by comparing the current model's own outputs, so you need a model that writes answers worth comparing. And SFT is cheap, stable, and easy to debug, so it is where you install the format, the persona, and the stop token before spending compute on anything subtler. DeepSeek-R1-Zero is the instructive exception. It ran RL directly on a base model and learned to reason, but its outputs mixed languages and were hard to read, and the released DeepSeek-R1 put a small SFT stage back in front (DeepSeek-AI 2025).

Here is how four well-documented recipes arrange the stages.

Notice the loops. Labs do not run the stack once. They iterate, using each improved model to generate or filter the next round's data, which is why "SFT" in a modern pipeline often means fine-tuning on the best samples of the previous model, a recipe called rejection sampling fine-tuning (§18.2).

import numpy as np

def token_ranks(base_probs, tokens):
    return np.argsort(-base_probs, axis=1)[:, 0]     # placeholder: not a rank

def shift_fractions(ranks):
    raise NotImplementedError

The superficial alignment hypothesis

In 2023 a group at Meta put the selection picture to a sharp test. Zhou et al. fine-tuned LLaMA 65B on just 1,000 prompts and responses, with no reinforcement learning and no preference data, and called the result LIMA, for "Less Is More for Alignment". The 1,000 examples were chosen with obsessive care. 750 came from community Q&A sites (Stack Exchange, wikiHow, and hand-picked Reddit WritingPrompts), filtered for quality and sampled for diversity. The other 250 were written or edited by the authors in a consistent assistant style, 200 of their own prompts with their own answers (including 13 that partly or fully decline harmful requests) and 50 examples adapted from Super-Natural Instructions.

In a human preference study on 300 test prompts, LIMA's answers were judged equivalent to or better than GPT-4's in 43% of cases, than Bard's in 58%, and than OpenAI's RLHF-trained DaVinci003 in 65%. It also beat a 65B model trained the same way on Alpaca's 52,000 examples. The authors stated the hypothesis their result suggests. A model's knowledge and capabilities are learned almost entirely during pretraining, and alignment mostly teaches it which subdistribution of formats to use when interacting with users. That is the superficial alignment hypothesis, and it is the persona-selection picture in words.

Their ablations, on a 7B model graded by ChatGPT, are the part to remember. Training on 2,000 diverse Stack Exchange examples beat 2,000 homogeneous wikiHow ones. Filtering responses for quality was worth about half a point on a six-point scale. And growing the filtered Stack Exchange set from 2,000 to 32,000 examples, a 16-fold increase, did not improve the score at all. They even found that adding 30 multi-turn dialogues raised the share of excellent responses in conversations from 45.2% to 76.1%. Thirty examples taught a new interaction format.

More evidence for. Lin et al. (2023) compared base and aligned 7B models token by token. At each position of the aligned model's answer they asked where its chosen token ranks under the base model. For Llama-2-7b versus Llama-2-7b-chat, 77.7% of positions were the base model's top choice and 92.2% were in its top three. The positions that did shift were mostly stylistic, words like "However", "Thank", "Here", "cannot", and safety disclaimers, while the knowledge-bearing tokens were ones the base model would have picked anyway. They then aligned base models with no tuning at all, using the prompt from the opening of this section, three fixed in-context examples and a system prompt (URIAL), and matched supervised-tuned models on their benchmark. Gudibande et al. (2023) showed the flip side. Models fine-tuned to imitate ChatGPT's outputs fooled crowd raters, who judged them competitive, but closed little or none of the gap on targeted benchmarks the imitation data did not cover. They had copied the style, not the capability.

Evidence against. Raghavendra et al. (2024) measured post-training with objective benchmarks instead of preference ratings and found task performance growing as a power law in the number of fine-tuning examples, for math, coding, multi-hop reasoning, and instruction following, with a handful of examples aligning the style but nowhere near saturating the scores. Tulu 3's SFT ablations agree. Their models kept improving on average as more SFT data was added, with large gains on GSM8K (Lambert et al. 2024). Llama 2's authors also found that tens of thousands of high-quality annotations, not one thousand, were what it took. And reinforcement learning with verifiable rewards keeps raising math and coding scores for thousands of steps, which is hard to call superficial (§18.5).

Both camps are right about different things, and the toy above says which. If the behavior you want is already one of the base model's personas, a little data selects it, and more data adds nothing. That is style, format, tone, and stopping. If you want the model to be measurably better at a skill, you are partly building something that was not there, and that costs data and compute at a rate closer to $1/n$. The hypothesis is a good description of alignment of style and an over-simplification of post-training as a whole, which is what Raghavendra et al. concluded.

Capability versus behavior

The superficial alignment debate is really a confusion between two things a model can be said to "have". A capability is what the model can do under the best elicitation you can find, with the right prompt, enough samples, a good format. A behavior is what the model does by default, on a plain prompt, with one sample. The base model in our opening has the capability to answer the capital question (it wrote "Paris" in its own answer key) but not the behavior.

Two consequences follow. First, evaluations measure behavior, and a low score can mean the capability is missing or that it was not elicited. A base model graded zero-shot in chat format looks far worse than the same model graded few-shot, which is why base-model benchmarks are run with few-shot prompts and assistant benchmarks are not (§14.7). Measuring capability needs many samples, and the gap between pass@1 and pass@$k$ for large $k$ is one way to see how much of what a model can do it actually does by default (§18.2).

Second, behaviors installed cheaply can be removed cheaply. Qi et al. (2023) undid GPT-3.5 Turbo's safety training by fine-tuning on only 10 adversarially designed examples through OpenAI's API, for less than 20 cents, and found that even benign fine-tuning data eroded safety behavior somewhat. If ten examples can install a persona, ten examples can swap it for another. That is why safety cannot rest on post-training alone (§19.7), and why anyone fine-tuning an aligned model has to re-check its safety behavior afterward.

And finally, post-training can cost capability. InstructGPT's authors called it the alignment tax. Their RLHF models regressed on several public NLP benchmarks, and they recovered most of the loss by mixing pretraining gradients into the RL updates (their "PPO-ptx"). Catastrophic forgetting in fine-tuning is the same phenomenon, and §14.4 measures it.

import numpy as np

def kl_div(p, q):
    return float(np.sum(p * np.log(p / q)))

Read more

• Zhou et al. (2023). LIMA: Less Is More for Alignment. The superficial alignment hypothesis, the 1,000-example dataset, and the diversity, quality, and quantity ablations.
• Lin et al. (2023). The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learning. Token distribution shifts between base and aligned models, and alignment by in-context examples alone.
• Raghavendra et al. (2024). Revisiting the Superficial Alignment Hypothesis. Power-law scaling of post-training data on objective benchmarks, the strongest case against.
• Ouyang et al. (2022). Training language models to follow instructions with human feedback. The InstructGPT pipeline that set the template for SFT, reward modeling, and PPO.
• Lambert et al. (2024). Tulu 3: Pushing Frontiers in Open Language Model Post-Training. A fully open modern recipe with SFT, DPO, and RL with verifiable rewards, and many ablations.
• Gudibande et al. (2023). The False Promise of Imitating Proprietary LLMs. Imitation copies style but not capability, and human raters can miss the difference.
• Gekhman et al. (2024). Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?. Why fine-tuning is a poor way to add facts.
• Qi et al. (2023). Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!. Ten examples and twenty cents to undo safety training.

Watch next

▶

▶

▶