Give GPT-3 a passage and ask it for the missing last word, and it gets 76.2 percent of LAMBADA right. Show it one solved example first and it does worse, 72.5 percent. Show it a few and it jumps to 86.4 percent (Brown et al., 2020). Not one weight changed between those three runs. The only difference is text in the prompt, and somehow the model treated that text as training data.

Two more puzzles come from the same few years. Zhao et al. (2021) gave GPT-3 2.7B four labeled movie reviews and asked it to label a fifth. Depending only on the order of the same four examples, accuracy on SST-2 ranged from 54 percent, barely above a coin flip, to 93 percent. And Min et al. (2022) replaced the correct labels in the examples with random ones, and accuracy fell by only 0 to 5 points. How can a model learn from examples whose labels are wrong, yet be thrown off by shuffling examples whose labels are right?

This section answers those questions as far as anyone can today. We start from the smallest in-context learner there is, a bigram model whose counts come from the prompt, and watch a two-layer Transformer discover the same trick during training. Then come the two leading theories of what few-shot prompting does, Bayesian inference over tasks and gradient descent inside the forward pass, with a careful account of what each actually proves. Then the practical half. Why prompts are fragile, how to calibrate them, what chain of thought adds, why a base model is not a chat model, and how to evaluate a prompt without fooling yourself. It is the last section of the chapter, and it turns the GPT you built into something you can use.

Zero-shot, few-shot, and what "learning" means here

§6.6 introduced the vocabulary. A zero-shot prompt describes the task and poses the question. A few-shot prompt first shows $k$ solved examples, called demonstrations, and then the question. Written as text, a 3-shot sentiment prompt looks like this.

Review: A gripping, beautiful film.
Sentiment: positive

Review: Dull and far too long.
Sentiment: negative

Review: I loved every minute.
Sentiment: positive

Review: The plot made no sense at all.
Sentiment:

Formally, write the demonstrations as pairs $(x_i, y_i)$ for $i = 1, \dots, k$, turned into text by a fixed template (here "Review", a newline, "Sentiment", and the label word), and call the whole prompt $S_k$. With the model's parameters $\theta$ frozen, the prediction for a query $x$ is

where $\mathcal{Y}$ is the set of labels and $\text{verbalize}(y)$ is the word that stands for each one (" positive", " negative"). The probability is the likelihood score of §6.6, the sum of log-probabilities of the label's tokens.

In plain words, nothing is trained. The demonstrations are part of the input, and the model's next-token distribution happens to depend on them in a way that looks like learning. That is the whole definition of in-context learning (ICL). It is an empirical phenomenon, not a mechanism, and most of this section is about candidate mechanisms.

GPT-3's results show how much the demonstrations can matter. On TriviaQA the 175B model scored 64.3 percent zero-shot, 68.0 one-shot, and 71.2 few-shot. The LAMBADA numbers from the opening have a lesson of their own. The few-shot prompt used a fill-in-the-blank format, and the authors' suggested explanation for the one-shot dip is that the model needs several examples to recognize that format. A single example mostly taught it the wrong thing. Brown et al. also found that the gain from few-shot over zero-shot grows with model size, and on LAMBADA the few-shot format hurt their smallest model by almost 20 points while helping the largest by 10.

There is a second, broader meaning of in-context learning that we will need. Olsson et al. (2022) defined it as the model getting better at predicting later tokens in a document than earlier ones, and measured it with an in-context learning score, the loss on the 500th token of a context minus the loss on the 50th, averaged over many documents. A model that uses its context has a negative score. Few-shot prompting is one special case. Remembering a name introduced three paragraphs ago is another.

The smallest in-context learner, a bigram model of the prompt

Before any neural network, here is a learner you can run in your head. Read the context so far and count, inside it, which token followed which. To predict what comes after the current token $a$, use those counts.

where $n_{\le t}(a, b)$ counts the positions $s \le t$ with $w_{s-1} = a$ and $w_s = b$, $n_{\le t}(a) = \sum_b n_{\le t}(a, b)$, $p_0$ is a fallback distribution (the model's prior), and $\alpha \gt 0$ says how much to trust the prior over the context. In plain words, if the current token appeared before, predict what followed it last time, and otherwise fall back on what you knew already. The weights of this "model" are just $p_0$ and $\alpha$. Everything it learns about a particular sequence, it learns from the context.

Work it by hand. Take $V = 10$ tokens, a uniform prior $p_0(b) = 0.1$, and $\alpha = 1$, and feed it five distinct tokens followed by the same five again. In the first copy every token is new, so every prediction is the prior, $0.1$, a loss of $\ln 10 = 2.303$ nats. In the second copy the current token was seen once, followed by the right answer, so the right answer gets $(1 + 0.1)/(1 + 1) = 0.55$, a loss of $-\ln 0.55 = 0.598$ nats. The loss drops by 1.7 nats per token the moment the text starts repeating, and the in-context learning score is strongly negative.

The loss at position 4 is still 2.303, because the token there (3) has never been followed by anything. Only from position 5 on does the context hold the answer. Now dress the same learner up as a few-shot prompt, with words as tokens.

After the second "Japan" it answers "Tokyo" with probability 0.56, a few-shot success. After "Italy" it has nothing, because Italy never appeared, and it falls back to the uniform prior. A real language model answers "Rome", because it knows capitals from pretraining and it read the format from the prompt. That's the gap between copying and in-context learning in the full sense. Copying reuses an exact earlier pattern. Few-shot learning recognizes which task the examples describe and applies knowledge stored in the weights to a new input. Hold on to both halves, because the two theories later in this section are about the second.

import numpy as np

def context_bigram_probs(tokens, V, alpha, p0):
    raise NotImplementedError

Induction heads, the bigram trick inside a Transformer

A Transformer can't keep a count table, but it can implement "find where the current token appeared before and copy what came next" with attention. You built that circuit with explicit weights in §4.7, so here is the one-paragraph recap. A previous-token head in layer 1 (the head of Exercise 4.2b in §4.2) copies the identity of token $w_{j-1}$ into the residual stream at position $j$. An induction head in layer 2 then uses the current token $w_t$ as its query and that "my predecessor was" information as its keys, so its score $\mathbf{q}_t^\top \mathbf{k}_j$ is large exactly when $w_{j-1} = w_t$. It attends to the position right after the earlier occurrence and its value copies that token, which becomes the prediction. The pattern is $[A][B] \dots [A] \to [B]$. No single head can do it, because the key at position $j$ must describe position $j - 1$, which only an earlier layer can write there. Elhage et al. (2021) call this K-composition, and §5.1 showed why heads in different layers can compose through the residual stream.

An induction head is the context bigram of the last section, implemented with attention. Where the counting model looks up $n_{\le t}(a, \cdot)$, the induction head spreads its attention equally over every earlier position whose predecessor was $a$, so the tokens it copies arrive in proportion to $n_{\le t}(a, b)$, the same counts divided by their total. When nothing matches, a head typically parks its attention on the first token, which plays the role of the prior. And it has one thing the counting model lacks. The match $\mathbf{q}_t^\top \mathbf{k}_j$ is a dot product of learned vectors, so it can be fuzzy. Olsson et al. found heads that match a token to a different but related one, which lets the same circuit continue a pattern across a translated sentence, one of their examples, and in principle copy a label from the most similar earlier example.

That last ability is few-shot classification in its simplest mechanistic form. In the 3-shot prompt above, the position after the query's Sentiment: can attend to the earlier label words, weighting each by how similar its review is to the query, and copy the labels in proportion. Here is that idea built as a single head. Each demonstration's label gets the score $s_i = \beta \cos(\mathbf{x}, \mathbf{x}_i) + \gamma \tfrac{i - 1}{k - 1}$, where $\mathbf{x}$ and $\mathbf{x}_i$ are vector representations of the query and of demonstration $i$, $\beta \ge 0$ sets how sharply the match counts, and $\gamma \ge 0$ adds a preference for later positions, the kind of positional bias attention heads routinely learn. Attention is $\operatorname{softmax}(\mathbf{s})$ over the $k$ labels, and the predicted distribution is the attention mass on each label.

The head classifies the first query correctly with 97 percent confidence, because the query's review is most similar to the two negative examples. But look at the three orders of the same four demonstrations. Alternating labels gives 91 percent accuracy. Putting both negatives last drops it to 77 percent and makes the model say "positive" only 27 percent of the time on a test set that is half positive, and putting both positives last flips the bias to 77 percent positive. We built a toy with a recency preference and it duly shows recency bias. Keep this head in mind, because the same biases appear in GPT-3, and the widget and exercises later in this section measure and repair them.

This is a model built to make a point, so be clear about its status. That trained Transformers contain label-copying heads of roughly this kind is a hypothesis with partial evidence (Olsson et al. show fuzzy induction heads, and Wang et al. (2023) traced information in GPT-2 XL and GPT-J gathering at the demonstrations' label words and flowing from there to the final prediction). That a single such head explains few-shot learning in a large model is not claimed by anyone. Large models also use the label meanings they know, which is the subject of the next sections.

Finding induction heads in a trained model

How do you know a head in a real model is an induction head? Olsson et al. used two tests, both of which you can run on any model. The first is behavioral. Feed a sequence of random tokens repeated twice, $[\text{BOS}, r_1, \dots, r_n, r_1, \dots, r_n]$, and for each head average the attention that each token of the second copy pays to the token right after its earlier occurrence. That is the prefix-matching score, near 1 for a clean induction head and about $1/t$ at position $t$ for a head that attends uniformly. Random tokens matter, because on natural text many heads attend to the same places for other reasons.

The second test checks that the head copies. A head's effect on the logits through its value and output matrices is the $V \times V$ matrix $\mathbf{E}\,\mathbf{W}_{OV}\,\mathbf{E}_U$, where $\mathbf{E} \in \R^{V \times d}$ embeds tokens, $\mathbf{W}_{OV} = \mathbf{W}_V \mathbf{W}_O \in \R^{d \times d}$ is the head's value-output product, and $\mathbf{E}_U \in \R^{d \times V}$ is the unembedding. If attending to token $b$ raises the logit of $b$ itself, the matrix has large positive eigenvalues. Elhage et al. summarize this with the copying score $\sum_i \lambda_i / \sum_i \lvert \lambda_i \rvert$ over its eigenvalues (real parts in the numerator), which is $+1$ for a pure copier, $-1$ for a head that suppresses what it attends to, and near 0 for one that does neither.

import numpy as np

def prefix_matching_scores(attn, n):
    raise NotImplementedError

def copying_scores(W_E, W_OV, W_U):
    raise NotImplementedError

How induction heads form, a phase change

Induction heads are not designed in. They appear during training, and the way they appear is one of the cleanest stories in the science of deep learning. Olsson et al. (2022) trained dozens of models, from one and two layers to large ones, and found the following, which are empirical findings across their models.

• Early in training, in a narrow window (roughly 2.5 to 5 billion tokens in their runs), induction heads form abruptly.
• At the same moment the in-context learning score jumps. Before the window the model gains less than 0.15 nats from late context, after it roughly 0.4 nats, and the gain then barely changes for the rest of training.
• The training loss shows a small bump, a short stretch where it stops falling smoothly, at exactly that point.
• One-layer models never form induction heads, since the circuit needs two heads in different layers, and they never get much in-context learning.

They offered six lines of evidence that induction heads cause most of the in-context learning in these models. For small attention-only models the evidence is causal (ablating the heads largely removes the in-context learning, and changing the architecture to make them form earlier moves the jump earlier). For large models with MLPs it is correlational, and the authors call the overall case circumstantial. That's the honest summary to give in an interview.

Why does it happen abruptly? The usual explanation is that the circuit only pays off when both halves exist. A previous-token head on its own does nothing useful for prediction, and an induction head without the previous-token information in its keys has nothing to match. Gradient descent improves each half only in proportion to how good the other already is, so for a while both grow slowly, and then they reinforce each other and snap into place. Data matters too. Chan et al. (2022) showed that in-context learning emerges when training data is bursty (items recur in clusters within a context, as names do in a document) and has many rare classes, which is how natural language looks, while uniform, independent data teaches the model to memorize in its weights instead.

You can watch the phase change on a laptop. The block below trains two attention-only Transformers, one with one layer and one with two, on random tokens whose first stretch of 5 to 14 tokens repeats once. The repeat length varies, so the model can't memorize a fixed offset, and the only way to predict the second copy is to copy from context. Every 100 steps it reports the loss on the repeated tokens, the loss elsewhere, and for the two-layer model the previous-token score of its first head and the prefix-matching (induction) score of its second.

Read the two-layer log from the top. For 300 steps the loss on repeated tokens creeps down from 3.84 to about 3.2, a little below the $\ln 32 = 3.47$ of guessing, about as fast as the one-layer model's. Between steps 300 and 500 it collapses to about 1.2, and by step 800 it is near 0.6. The induction score climbs from 0.2 to about 0.8 in the same window. The previous-token score rises too, but only to about 0.3, which shows something real. The first head does not need to be sharp. A modest weight on the previous token puts enough information about it into the residual stream, and the second head's query-key matrix amplifies it. The one-layer model, meanwhile, ends at about 3.0. It learns that tokens seen earlier in the context are more likely, which is worth a little, but it cannot learn which earlier token comes next.

One more detail is worth noticing. After the phase change the loss elsewhere rises from 3.50 to about 3.6. The model now bets some probability on copying everywhere, and where nothing repeats that bet is lost. Induction heads are a prior that sequences repeat, priced into every prediction, which is a good trade on natural text and a bad one on random tokens.

Few-shot learning as inference over tasks

Induction heads explain copying. They don't explain how GPT-3 answers "Rome" for a country it never saw in the prompt, or why random labels barely hurt. For that, the most useful lens is Bayesian. Xie et al. (2021) start from what pretraining text looks like. A document is not a bag of unrelated sentences. It has a topic, a style, a format, and a purpose, which stay fixed across the whole document. Call that hidden, document-level property a concept $\phi$. To predict the next token of a document well, a model must infer $\phi$ from what it has read so far, because the concept decides what comes next. A model that has learned to do this has learned, as a side effect, to infer a task from examples.

Here's the math. Suppose the model's predictive distribution is the Bayes-optimal one for text generated by first drawing a concept from a prior $p(\phi)$ and then drawing text given that concept. For a prompt $S_k$ of demonstrations and a query $x$, the prediction averages over concepts,

and Bayes' rule (§0.4), with the demonstrations independent given the concept, gives the posterior (the query input contributes one more factor, $p(x \mid \phi)$, which we leave out to keep the formula short)

In plain words, each demonstration is evidence about which task you are being asked to do, and the model answers the query the way that task would. Its knowledge of how to do each task, $p(y \mid x, \phi)$, lives in the weights. Only the choice of task comes from the prompt.

A worked example by hand. Two concepts are equally likely a priori, "country to capital" (A) and "country to language" (B). The demonstration "France, Paris" has input probability $p(\text{France} \mid A) = 0.3$ and $p(\text{France} \mid B) = 0.1$ (capital quizzes favor big countries, say), and label probability $p(\text{Paris} \mid \text{France}, A) = 0.9$ against $p(\text{Paris} \mid \text{France}, B) = 0.05$. The likelihoods are $0.3 \times 0.9 = 0.27$ and $0.1 \times 0.05 = 0.005$, a ratio of 54, so after one example the posterior on A is $54/55 = 0.982$. Given "Italy" next, the model answers "Rome" with probability close to 0.98 times what concept A gives Rome, without Italy ever appearing in the prompt.

Why does the posterior concentrate as $k$ grows? Take logs and compare the true concept $\phi^*$ with any other $\phi$,

When the demonstrations really come from $\phi^*$, each term of the sum has expectation $\mathbb{E}_{\phi^*}\big[\log \tfrac{p(x, y \mid \phi^*)}{p(x, y \mid \phi)}\big] = \operatorname{KL}\big(p(\cdot \mid \phi^*) \,\Vert\, p(\cdot \mid \phi)\big) \ge 0$, the KL divergence of §0.5. So the log-odds in favor of the right task grow linearly in $k$, at a rate equal to how distinguishable the two tasks are, and the posterior mass on every wrong task shrinks exponentially. Xie et al. prove a version of this for a pretraining distribution that is a mixture of hidden Markov models, with one real complication handled. A prompt is not a natural document. The jumps between unrelated examples, and the delimiters between them, are unlikely under any single concept, and they show that in-context learning still works if each example carries enough evidence to outweigh that mismatch. They also built a small synthetic dataset (GINC) on which both Transformers and LSTMs learn in context, and reproduced several large-model behaviors there, including better in-context learning from bigger models at the same pretraining loss, sensitivity to example order and cases where zero-shot beats few-shot.

Now Min et al.'s puzzle has an answer. The likelihood of a demonstration factors into evidence from its input ($p(x_i \mid \phi)$, what kind of text the task is about), from its label space (which words appear as answers), from the format, and only then from the mapping (whether $y_i$ is the right answer for $x_i$). Random labels destroy only the last. If the first three already identify the task, the posterior still lands on it and the model answers from its weights, correctly. If tasks differ only in their mapping, random labels are fatal. The cell below makes that concrete with three concepts in two worlds, one where tasks look different and one where they look alike.

When tasks look different, one demonstration identifies the task from its label space alone, and random labels cost nothing at all. When the tasks share inputs and labels, gold demonstrations still work (94 percent after one), but random labels leave accuracy at 39 to 47 percent, worse with more examples than with none, since every wrong pair is evidence for a wrong task. Real benchmarks sit closer to the first world, since a sentiment task's reviews and label words look nothing like a topic task's, which is why Min et al. saw drops of only 0 to 5 points across more than a dozen models.

The Bayesian view has limits, and the research has found them. Wei et al. (2023) flipped the labels in demonstrations (positive reviews labeled "negative"). Small models ignored the flip and kept answering from their priors, exactly as the Bayesian picture with a strong prior predicts. But large models followed the flipped labels, overriding what they knew, and could even learn mappings to arbitrary label words ("foo" and "bar") that carry no meaning at all. Instruction tuning strengthened both abilities, the use of priors more. So larger models read the mapping in the demonstrations more than the simple picture suggests. A practical rule follows. Gold labels are cheap insurance, and the larger the model, the more they matter.

import numpy as np

def task_posterior(log_prior, log_lik_demos, log_lik_query):
    raise NotImplementedError

In-context learning as gradient descent, on linear regression

A second theory says something more mechanical. Inside its forward pass, a Transformer might be running a learning algorithm on the demonstrations, the way you would run a few steps of gradient descent on a training set. This sounds far-fetched until you see that it is exactly true in one setting, and that setting is worth knowing by heart.

Garg et al. (2022) asked a clean question. If you train a Transformer from scratch on prompts $(\mathbf{x}_1, y_1, \dots, \mathbf{x}_k, y_k, \mathbf{x}_q)$, where every prompt has its own random linear function $y = \mathbf{w}^\top \mathbf{x}$ with $\mathbf{x}, \mathbf{w} \in \R^d$, can it learn to predict $y_q$ for functions it has never seen? It can, and its error matches that of ordinary least squares fit to the $k$ examples, the optimal estimator for this problem. The same recipe learned sparse linear functions, two-layer networks, and decision trees about as well as algorithms designed for each. That is a behavioral fact. Von Oswald et al. (2023) and Akyürek et al. (2023) asked what algorithm sits inside, and the starting point of both is a short construction.

One step of gradient descent is one layer of linear attention

Fit $\mathbf{w}$ to the demonstrations by minimizing $\mathcal{L}(\mathbf{w}) = \frac{1}{2k}\sum_{i=1}^{k} (\mathbf{w}^\top \mathbf{x}_i - y_i)^2$, starting from $\mathbf{w}_0 = \mathbf{0}$, with one step of gradient descent at learning rate $\eta$ (§0.6). The gradient is $\nabla \mathcal{L}(\mathbf{w}) = \frac{1}{k}\sum_i (\mathbf{w}^\top \mathbf{x}_i - y_i)\,\mathbf{x}_i$, so at $\mathbf{w}_0 = \mathbf{0}$ it is $-\frac{1}{k}\sum_i y_i \mathbf{x}_i$ and

Now put each demonstration in a token, $\mathbf{r}_i = (\mathbf{x}_i, y_i) \in \R^{d+1}$, and the query in a token whose label slot is empty, $\mathbf{r}_q = (\mathbf{x}_q, 0)$. Consider linear attention, attention without the softmax, where the query token's output is $\sum_i (\mathbf{r}_q \mathbf{W}_Q)(\mathbf{r}_i \mathbf{W}_K)^\top\, \mathbf{r}_i \mathbf{W}_V$ with row vectors as in Chapter 4. Choose $\mathbf{W}_Q = \mathbf{W}_K \in \R^{(d+1)\times d}$ to read the $\mathbf{x}$ part of a token, so the score is $\mathbf{x}_q^\top \mathbf{x}_i$, and choose $\mathbf{W}_V \in \R^{(d+1)\times(d+1)}$ to read $y_i$, scaled by $\eta/k$, into the label slot. The label slot of the output is then

In plain words, one gradient step from zero predicts with a similarity-weighted vote of the training labels, and that is precisely what attention computes, with queries and keys measuring similarity and values carrying labels. The query token's own value is zero, because its label slot is empty, so it can attend to itself harmlessly.

Von Oswald et al. went beyond the construction. They trained linear self-attention layers on random regression tasks and found that the learned weights are this construction, up to symmetries, and that stacking layers gives several steps of gradient descent with a learned correction for curvature that beats plain gradient descent. Akyürek et al. proved that Transformers can also implement closed-form ridge regression, found that trained models' predictions track gradient descent, ridge, or exact least squares depending on depth and noise, and reported preliminary probing evidence that the models' late layers encode the weight vector being fit.

What step size would training choose?

One step of gradient descent can't match least squares, which is exact for noiseless data once $k \ge d$. So what does the best one-step learner look like, and how far behind is it? We can answer exactly. Let every $\mathbf{x}_i$, $\mathbf{x}_q$, and $\mathbf{w}$ be standard normal in $\R^d$ and $y_i = \mathbf{w}^\top \mathbf{x}_i$. With $\hat{\boldsymbol{\Sigma}} = \frac{1}{k}\sum_i \mathbf{x}_i \mathbf{x}_i^\top$ the sample covariance, the one-step weights are $\mathbf{w}_1 = \eta \hat{\boldsymbol{\Sigma}} \mathbf{w}$, and the expected squared error on a fresh query is

which is minimized at

In plain words, the best single step is shorter than 1 and lengthens as examples accumulate, because with few examples the sample covariance is a noisy estimate of the identity and a full step overshoots. And the error of the best one-step learner falls only like $1/k$, while least squares is exact from $k = d$ on. That gap is why trained Transformers, which do match least squares, must be doing more than one step, as the multi-layer results show.

Let's check the formula by simulation. For each $k$ we fit the best step size directly from 3,000 random tasks (the step that minimizes squared error is a one-variable least-squares fit) and measure the error at $\eta^*$.

The fitted step sizes land on $k/(k+d+1)$ and the errors on $d(d+1)/(k+d+1)$, within the noise of 3,000 tasks. If you train a single linear attention layer on this task, the step size it learns is this $\eta^*$, which is the precise sense in which "the model learned to do gradient descent".

import numpy as np

def gd_weights(d, eta, k):
    raise NotImplementedError

What these results show, and what they don't

This line of work is often summarized as "Transformers do gradient descent in context". That sentence is true in a narrow setting and unproven in the one people care about, so here is the careful version.

The last row matters most. Every positive result above trains the model on in-context regression itself, with tokens that are clean numeric vectors. Shen et al. (2024) tested the hypothesis on LLaMA-7B and found that in-context learning and gradient descent on the same demonstrations change the model's outputs differently, respond differently to the order of the demonstrations (gradient descent from zero is order-free, in-context learning is not), and that the hand-built weights in these constructions look nothing like real trained weights. What the regression results do give you is a proof that in-context learning can be real learning, an algorithm run on the data in the prompt, and a clean test bed for studying when it emerges. They don't tell you what GPT-4 does with your prompt.

So we have three pictures. Induction heads copy patterns from the context. The Bayesian view says demonstrations select a task the model already knows. The learning-algorithm view says the model fits a small predictor to the demonstrations. They are not rivals so much as different regimes. Copying explains why repetition is cheap, inference explains why random labels barely matter, and in-context algorithms explain how large models learn genuinely new mappings like "foo" and "bar".

Sensitivity to order, labels, and format

Now the practical consequences. Few-shot prompting is fragile, and the fragility has been measured carefully.

Order. Zhao et al. (2021) found the 54 to 93 percent spread from the opening across orderings of four fixed SST-2 examples for GPT-3 2.7B, and a similar spread across choices of which four examples to show. Lu et al. (2022) showed that the problem appears at every model size they tested, that a good order for one model is not a good order for another, and that in a true few-shot setting there is no labeled development set to choose an order with. Their fix ranks candidate orders by the entropy of the model's predictions on a set of unlabeled probe inputs the model generates itself, preferring orders whose predictions are not collapsed onto one label, which improved GPT-family models by 13 percent relative on average across eleven text classification tasks.

Labels. Zhao et al. traced most of the variance to three biases. Majority label bias means the model favors whichever label appears most often in the prompt. Recency bias means it favors labels near the end, and it can beat the majority. With three positive examples followed by one negative, GPT-3 2.7B predicted "negative" for nearly 90 percent of inputs. Common token bias means it favors label words that are frequent in pretraining text. Min et al.'s random-label result, as we saw, says the correctness of the labels matters less than you would think, at least for models of that generation.

Format. The words around the examples matter as much. Sclar et al. (2024) varied only meaning-preserving details such as separators, capitalization, and spacing (Answer: versus answer -), and found accuracy differences of up to 76 points for LLaMA-2-13B on a single task. The sensitivity remained with larger models, more examples, and instruction tuning, and the best format for one model was only weakly correlated with the best for another.

Our label-copying head shows the first two biases, and makes the mechanism visible. Here it is on all 24 orders of two demonstration sets, one balanced and one with three positives and a negative.

The balanced set spans 68 to 94 percent across orders, and the unbalanced one 50 to 94 percent, a coin flip for its worst order. The last line reproduces Zhao et al.'s observation in miniature. With three positives and one negative at the end, the head's prediction for an input that matches nothing is 74 percent negative, because recency outweighs the majority.

Contextual calibration

That last line also suggests the fix. If you show the model a content-free input, such as "N/A" or the empty string, in the same prompt, an unbiased model should be indifferent between the labels. Whatever it predicts instead is the bias of this particular prompt, its examples, their order, and its format. Zhao et al.'s contextual calibration measures that bias and divides it out. With $\hat{\mathbf{p}} \in \R^{\lvert \mathcal{Y} \rvert}$ the model's distribution over labels for a real input and $\hat{\mathbf{p}}_{\text{cf}}$ its distribution for the content-free input (they average three, "N/A", "[MASK]", and the empty string), set

In plain words, divide each label's probability by how much the prompt favors that label on its own, then renormalize. The content-free input becomes exactly uniform by construction. It costs one extra forward pass per prompt, needs no labeled data, and raised GPT-3's and GPT-2's average accuracy by up to 30 points absolute in their experiments while shrinking the variance across prompts. It is not magic. It removes a bias that is the same for every input, so it can't fix a prompt whose examples are unhelpful, and a content-free input that is not really neutral (some tasks have no good one) calibrates toward the wrong point.

The calibrated lines of the cell show what it does for our head, 91 to 96 percent for every order of either set, and a standard deviation across orders five to ten times smaller. Try it yourself below. Move the recency preference, switch to the unbalanced set, and watch the spread across orders, then turn on calibration.

import numpy as np

def order_sensitivity(predict, X_demo, y_demo, X_test, y_test):
    raise NotImplementedError

import numpy as np

def calibrate(probs, probs_cf):
    raise NotImplementedError

def calibrate_logits(logits, logits_cf):
    raise NotImplementedError

Chain of thought, a preview

Some tasks can't be answered in one step, whatever the prompt. Ask for the answer to a multi-step word problem directly and the model has a fixed number of layers in which to compute it, one forward pass per output token. Wei et al. (2022) found a remedy with no training at all. Write the few-shot demonstrations so that each one shows its working before its answer.

Q: Roger has 5 tennis balls. He buys 2 cans of 3 tennis balls each. How many tennis balls does he have now?
A: Roger started with 5 balls. 2 cans of 3 balls is 6 balls. 5 + 6 = 11. The answer is 11.

Q: The cafeteria had 23 apples. They used 20 for lunch and bought 6 more. How many apples do they have?
A:

The model imitates the format, writes out intermediate steps, and only then the answer. With eight such exemplars, PaLM 540B went from 17.9 percent on the GSM8K math benchmark with ordinary few-shot prompting to 56.9 percent. Kojima et al. (2022) then found that a single phrase works even with no exemplars. Appending "Let's think step by step" took InstructGPT (text-davinci-002) from 17.7 to 78.7 percent on MultiArith and from 10.4 to 40.7 percent on GSM8K. Both papers found the effect depends on scale. Wei et al. saw gains only around 100B parameters and up, and smaller models wrote fluent but illogical chains.

Why does it help? Each generated token is one more forward pass, so writing out steps gives the model more computation per answer and lets it store intermediate results in the text, where later steps can read them, attention being the model's working memory. That is a hypothesis with theoretical support rather than a measured mechanism in any particular model. Merrill and Sabharwal (2024) proved that, under standard complexity conjectures, a Transformer of fixed depth that may write a number of intermediate tokens linear in its input can solve problems it cannot solve when it must answer immediately, and a written chain is not guaranteed to reflect how the model actually reached its answer. Chain of thought is where prompting meets reasoning. §18.1 develops it properly, §18.2 samples many chains and votes, and §18.5 trains models to produce good chains with reinforcement learning, which is how today's reasoning models work.

What a base model can do from a prompt, and what it can't

Everything so far applies to a base model, a model like ours that was trained only to continue text. Prompting a base model means writing the beginning of a document whose most likely continuation is the answer you want. That's why few-shot prompts work so well on base models. A list of "Review, Sentiment" pairs is a document, and continuing it is the most natural thing in the world.

It also explains what goes wrong. Type an instruction like "Write a short poem about the sea." into a base model and it may continue with more instructions, as if it were reading a list of writing prompts, or with a forum reply, or with the poem, depending on which documents in its training data began that way. It has no notion of a turn, so it rarely stops where you want, and nothing in pretraining taught it to prefer a helpful answer over a merely likely one. Our tiny GPT is the extreme case. Whatever you type into the generator of §6.5, it writes something sonnet-shaped, because the only documents it knows are sonnets. And it can't even copy. We fed it random strings repeated twice and its loss on the second copy was no lower than on the first, because sonnets almost never repeat a long stretch of text, so nothing ever rewarded an induction head.

A chat model is the same network after post-training (§14.1). Supervised fine-tuning on conversations teaches it to follow instructions and respect a chat template with roles and turns (§14.3), and preference training teaches it which of several answers people prefer (§16.1). The effect is large. Ouyang et al. (2022) found that people preferred the outputs of a 1.3B InstructGPT model to those of the 175B GPT-3 base model, despite a hundredfold difference in size. Yet most of the knowledge was already in the base model. Lin et al. (2024) compared Llama-2 with Llama-2-chat token by token and found that they agree on the top-ranked token at most positions, the differences being mostly stylistic tokens such as discourse markers and safety disclaimers. With a system-style instruction and as few as three fixed, carefully written example conversations, a base model prompted their way came close to the aligned models on their evaluation.

The calibration row deserves a sentence. The GPT-4 technical report showed that the pretrained model's confidence on multiple-choice questions tracked its accuracy closely, and that after post-training it no longer did. That's one reason likelihood-based evaluation of a chat model can mislead, and why §19.8 returns to calibration when it discusses hallucination.

How to evaluate a prompt honestly

Prompt engineering is model selection with a very large hypothesis space, and it inherits every way model selection can fool you. Perez et al. (2021) made the point sharply. Many papers that called themselves few-shot had chosen their prompts using large held-out labeled sets, which a real few-shot user doesn't have. When they restricted selection to the few labeled examples actually available, using cross-validation or minimum description length, the chosen prompts were only marginally better than random choices, and much worse than prompts chosen with held-out data. Their conclusion was that prior work significantly overestimated the true few-shot ability of language models.

The mechanism is the winner's curse. Pick the best of many prompts on a small development set and its score there is partly luck, which won't come back on fresh data. Here it is with our toy, choosing the best of 24 orders on 16 labeled examples, 500 times over.

The chosen order looks like 98 percent on its 16 examples and delivers 93 percent, still useful, since selection beat the average order's 74 percent by a lot, but the reported number was five points too high. With 24 candidates and 16 examples that is a mild case. Prompt searches over thousands of variants on a hundred examples can be far worse. Here's the checklist we use.

• Separate selection from measurement. Choose prompts, examples, and orders on one set and report on another that you looked at once. If you only have a few labeled examples, say so and report the cost of selection honestly.
• Report a distribution, not a point. Evaluate several formats, several example sets, and several orders, and report the mean and the spread, as Sclar et al. recommend. A comparison between models with one hand-picked prompt each is a comparison between prompts.
• Use the same prompts for every model you compare, and know that a format that suits one model may not suit another, so a fair comparison averages over formats.
• Put error bars on accuracy and compare models on the same questions. With $n$ questions the standard error of an accuracy $p$ is $\sqrt{p(1-p)/n}$, and a paired test on the same items is far more sensitive than comparing two separate intervals (§0.11).
• Match the scoring to the model. Score base models by likelihood or with few-shot formats, and chat models with their chat template and a parser for their answers (§6.6). Evaluating a chat model without its template, or a base model with one, measures the mismatch.
• Check for contamination. A benchmark whose questions were on the web may be in the pretraining data, and then few-shot accuracy measures memory. Deduplication against benchmarks (§8.3) and held-out, freshly written test items are the defenses, and the problem gets a full treatment where contamination in evaluation is discussed.
• Fix the decoding. Temperature, stop sequences, and maximum length change scores, so record them with the prompt.

The habits generalize. The most reliable evaluations are built from your own real failure cases and kept fixed while you iterate, which is the discipline behind building your own evaluations.

The real way, a few-shot classifier on GPT-2

Here is the whole pipeline on a real pretrained model with the Hugging Face transformers library, which CI can't download, so it is shown and not run. It builds a 4-shot sentiment prompt, scores the two label words at the last position, and applies contextual calibration with the three content-free inputs of Zhao et al. Both label words are single GPT-2 tokens with their leading space, which is why the prompt ends at Sentiment: with no trailing space (the tokenization detail from §6.6).

To measure order sensitivity on a real model, loop over the permutations of demos, recompute p_cf for each, and score a labeled test set with and without calibration, exactly as order_sensitivity does in Exercise 6.8e. On a model this small, expect large swings between orders and a clear gain from calibration. On a current instruction-tuned model, expect smaller swings, but measure rather than assume.

Read more

• Brown et al. (2020). Language Models are Few-Shot Learners. The paper that named few-shot prompting, with zero-, one-, and few-shot results across model sizes.
• Olsson et al. (2022). In-context Learning and Induction Heads. The phase change, the in-context learning score, and six lines of evidence, honestly graded.
• Xie et al. (2021). An Explanation of In-context Learning as Implicit Bayesian Inference. In-context learning as inferring a latent concept, with the GINC dataset.
• Min et al. (2022). Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?. Random labels, and what demonstrations actually contribute.
• Zhao et al. (2021). Calibrate Before Use: Improving Few-Shot Performance of Language Models. Majority-label, recency, and common-token bias, and contextual calibration.
• Lu et al. (2022). Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity. Order sensitivity across model sizes and entropy-based order selection.
• Garg et al. (2022). What Can Transformers Learn In-Context? A Case Study of Simple Function Classes. Transformers trained to learn linear functions, trees, and small networks in context.
• von Oswald et al. (2023). Transformers Learn In-Context by Gradient Descent. The linear-attention construction and trained weights that match it.
• Akyürek et al. (2023). What learning algorithm is in-context learning? Investigations with linear models. Gradient descent, ridge, and least squares inside trained Transformers.
• Shen et al. (2024). Do pretrained Transformers Learn In-Context by Gradient Descent?. Where the gradient-descent story fails for real language models.
• Wei et al. (2023). Larger language models do in-context learning differently. Flipped and semantically unrelated labels, and how scale changes the answer.
• Sclar et al. (2024). Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting. Formatting changes worth up to 76 points, and FormatSpread.
• Perez et al. (2021). True Few-Shot Learning with Language Models. Why prompt selection on held-out data overstates few-shot ability.

Watch next

▶

▶

▶