A circuit is the small part of a trained network that carries out one behavior, together with the evidence that this part, and not the rest, is what does it. This section is about finding circuits and proving them. The method is intervention. You change one internal value while the model runs and watch whether the output follows. Weights, attention patterns, and attributions can suggest a story, but only an intervention shows that a part causes the behavior.

Circuits turn "the model said Mary" into "these heads, in this order, made it say Mary". That is the kind of claim an auditor needs before trusting what a model does inside, and the kind you need to fix a behavior instead of guessing at it. The tools of this section are the everyday instruments of interpretability teams, and interviews for those teams ask about each of them.

Here is the question that runs through the section. GPT-2 small reads "When Mary and John went to the store, John gave a drink to" and continues with "Mary". The task is called indirect object identification, or IOI. The indirect object (IO) is the person who receives the drink, Mary. The subject (S) is the person who gives it, John, who is named twice.

A logit is the score the model gives a token before the softmax turns the scores into probabilities. On 100,000 sentences built from 15 templates like this one, GPT-2 small gives the indirect object a larger logit than the subject 99.3 percent of the time (Wang et al. 2022). The model has 144 attention heads in 12 layers. Which of them do this, and how would you prove it?

The previous section gave a first clue. Direct logit attribution takes the final logit difference, the logit of "Mary" minus the logit of "John", and splits it into one term per head and per MLP, the amount each one writes straight into the output (§23.3). It points at a few late heads that write "Mary". But a head that never writes to the output can still be essential, because later heads read what it wrote. Direct logit attribution sees only the last step of a computation, and the question here is about all of it.

Section 23.1 ranked evidence on a ladder of four rungs. Association means some quantity tracks the output. A direct effect is what a part writes straight into the output. A causal effect is what changes downstream when you set an internal value yourself. A mechanism is a full account of the steps, confirmed by the interventions it predicts. This section climbs from direct effects to causal effects and then to mechanisms.

Here is the plan. We first say exactly what a circuit is. Then we build the tools in order. Activation patching copies one value from one run into another, ablation switches a part off, and path patching follows one route through the network. Composition scores read the same wiring from the weights, and attribution patching makes the whole sweep cheap with one gradient. Every tool is checked on a small hand-built model whose answer we know. Then we follow the tools to the IOI circuit in GPT-2 small, to automated circuit discovery, and to the one trained network whose algorithm was reverse-engineered completely.

What counts as a circuit

A Transformer is built from parts that read and write. The parts are the token and position embeddings, each attention head, and each MLP, the two-layer feed-forward block in every layer. Every part reads the residual stream, the running vector at each position to which every layer adds its output (§5.1). Because all writes are added into the same stream, any earlier part can feed any later one.

So we can draw the forward pass as a computational graph. Its nodes are the parts, possibly split by position, so that head 9.6 at the last token is a different node from head 9.6 at the first. (We name heads layer.head, counting from 0, so 9.6 is head 6 of layer 9.) Its edges are the reads, one from every node to every later node. A circuit is a piece of this graph.

In plain words, a circuit is a part of the network plus a promise. The promise is that you can switch off the rest of the network and the behavior barely changes. Two views run through the section. The first intervenes on activations and asks which parts do the work on the inputs we care about. The second reads the weights and asks which parts could. An interviewer who asks "what is a circuit?" wants both halves of the answer, the subgraph and the test that backs it.

Interventions, the evidence that matters

Observing a model tells you what goes along with the answer. Watching attention patterns, training probes, and computing direct logit attributions are all observations. Intervening tells you what causes the answer. An intervention sets one activation $\mathbf{h}$ to a value $\mathbf{v}$ of our choosing, written $\mathbf{h} \leftarrow \mathbf{v}$, and lets everything downstream recompute as usual. If the output moves, the activation lies on a causal path to the output, at least for the inputs we used.

To choose the value we need a second input. Take a clean prompt $x_{\text{clean}}$ on which the model shows the behavior. Take a corrupted prompt $x_{\text{corr}}$ that differs from it in exactly the detail the behavior depends on. Write $\mathbf{h}_{\text{clean}}$ and $\mathbf{h}_{\text{corr}}$ for the value of one activation on the two runs.

Then pick a metric $m$, one number computed from the output. The usual metric is the logit difference $\Delta = z_a - z_b$. Here $z_a$ is the logit of the correct token $a$, and $z_b$ is the logit of a foil $b$, the most tempting wrong answer. In IOI the correct token is the indirect object and the foil is the subject.

A three-number example

Before any model, here is a computation small enough to do in your head. The input $x$ sets an activation $h_1 = x$. That feeds a second activation $h_2 = 2h_1$, and the output is $y = h_2 + h_1$. So $h_1$ reaches $y$ along two paths, directly and through $h_2$. Take $x_{\text{clean}} = 1$ and $x_{\text{corr}} = 0$, so $y_{\text{clean}} = 3$ and $y_{\text{corr}} = 0$.

Direct logit attribution in §23.3 measures only the last kind of number, the effect along the direct edge into the logits. The tools below measure the others. In an interview, this toy is the fastest way to show that you know a total effect from a direct one.

Activation patching in two directions

Activation patching runs one prompt with a single activation copied in from the other prompt's run. There are two ways to do it, and they answer different questions.

In plain words, denoising asks "is this activation enough to bring the answer back?" and noising asks "can the model do without it?". In the three-number example, denoising $h_2$ gives $2/3$. Noising $h_2$ runs the clean input with $h_2 \leftarrow 0$, so $y = 1$, which also destroys $2/3$ of the behavior. In that linear toy the two directions agree. In real networks they often do not.

They disagree when parts work together. Suppose the behavior needs two activations at once, an AND. Each one alone is necessary, so noising either one gives a large effect. Neither alone is sufficient, so denoising either one gives a small effect. If either activation is enough on its own, an OR, the reverse holds. We will meet an AND in the hand-built model below, and an OR when backup heads appear in GPT-2. So a patching result must always say which direction it used.

Choosing the corruption and the metric

Two choices decide whether the numbers mean anything.

The corruption. The corrupted prompt should change only the detail the behavior depends on, and it should still look like an ordinary input. For IOI, the simplest choice replaces the second mention of the subject with a third name. "John gave a drink to" becomes "Paul gave a drink to", and the sentence no longer says who should receive the drink. We call this the ABC corruption and use it for the small model below. Wang et al. (2022) used a stronger version, their ABC distribution, which replaces all three names with three unrelated random names, as in "When Anna and Omar went to the store, Paul gave a drink to". Either way no name repeats.

Swapping tokens for other tokens of the same kind like this is called symmetric token replacement. The alternative is to add Gaussian noise to some embeddings. That produces activations the model never saw in training, and it changes several things at once.

The metric. Just before the unembedding sits the final LayerNorm, a normalization that centers the stream and divides it by its spread. Once that spread is held fixed, the logit difference is a linear function of the final stream (§23.3). So the effects of different parts add up. A partial restoration shows up in proportion. The probability of the correct token instead passes the logits through a softmax, which flattens near 0 and 1. A patch that restores half of the logit difference can then show up as almost nothing. A probability also cannot fall below zero, so it can hide parts that push toward the wrong answer.

So a patching result should always come with its corruption and its metric. The first thing an interviewer will probe is why you chose them.

A model small enough to read completely

To check these tools against a known answer, we need a model whose mechanism we know because we wrote it. The mini-IOI model is a three-layer Transformer with one attention head per layer and no MLPs. Its weights are set by hand, like the induction head built in §4.7. Each head does one step of the IOI algorithm. Head 1 marks names that appear twice. Head 2 carries the fact "this name is repeated" to the last position. Head 3 copies to the output the name that is not repeated.

The vocabulary has 16 tokens. A BOS (beginning of sequence) token starts every prompt. There are ten names, and four filler tokens f that stand in for the other words of a sentence. An END token sits at the position where the model predicts. Prompts look like

or the same with $A$ and $B$ swapped in the first clause. The correct answer at END is the name that appears once, $A$, the indirect object. The foil is the repeated name $B$, the subject. We call the position of the first $B$ S1 and that of the second $B$ S2. The ABC corruption replaces the $B$ at S2 with a third name $C$.

The residual stream at each position is 54 numbers in five blocks laid side by side, $[\,\text{TOK} \mid \text{POS} \mid \text{DUP} \mid \text{INH} \mid \text{OUT}\,]$. TOK is a one-hot code of the token (16 numbers) and POS a one-hot code of the position (8 numbers). DUP, INH, and OUT have one slot per name (10 numbers each), and the heads write into them. As everywhere in the book from Chapter 4 on, token vectors are rows, and a matrix $\mathbf{W} \in \R^{d_{\text{in}}\times d_{\text{out}}}$ acts as $\mathbf{x}\mathbf{W}$.

Recall from §4.2 that a head splits into two parts that can be read separately. The QK circuit $\mathbf{W}_{QK} = \mathbf{W}_Q\mathbf{W}_K^\top$ decides where the head looks, since the score from position $t$ to position $j$ is $\mathbf{x}_t\mathbf{W}_{QK}\mathbf{x}_j^\top$. The OV circuit $\mathbf{W}_{OV} = \mathbf{W}_V\mathbf{W}_O$ decides what it moves, since attending to position $j$ adds $\mathbf{x}_j\mathbf{W}_{OV}$ to the output. Here the scores are not scaled by $1/\sqrt{d_k}$, and attention is causal.

The unembedding reads only OUT. In the last row, INH stands for the 10 numbers in the INH slot at END, which are close to 1 for the name head 2 flagged and 0 for the others. So the name mover's query gives an unflagged name $+c_3$ and a flagged one $-c_3$. At END it scores $+c_3$ for $A$, $-c_3$ for both copies of $B$, and 0 for the five other positions. With $c_3 = 4$ it puts weight $e^4/(e^4 + 5 + 2e^{-4}) \approx 0.92$ on $A$. With gain $g = 4$ the logit difference is about $4 \times 0.92 = 3.66$.

The three heads are wired the way Wang and colleagues found GPT-2's heads wired. Head 2's keys and values read what head 1 wrote, and head 3's query reads what head 2 wrote. When a later head's query, key, or value reads an earlier head's output, the heads are said to compose. We will measure this later as Q-, K-, and V-composition. The cell builds the model and checks it.

The model is right on every prompt. Its logit difference is 3.659 on the clean prompts and 0 on the ABC prompts. On an ABC prompt no name repeats, so the name mover splits its attention equally among $A$, $B$, and $C$.

Direct logit attribution gives the duplicate-token and S-inhibition heads exactly zero, because neither writes anything the unembedding reads. Yet without them the model cannot tell $A$ from $B$. That is the gap this section closes. (Section 23.3 ran a variant with five names, a final LayerNorm, and a small direct write from the S-inhibition head. Here that head writes nothing the unembedding reads, which keeps every number below exact.)

Patching the mini-IOI model

Read the left map from the top. At the embeddings and after layer 1, only S2 matters, since it is the only position that differs. After layer 2 the information has split across two positions. Denoising END alone restores 0.52 of the logit difference. Denoising S2 alone gives $-0.35$, worse than doing nothing. After layer 3 everything sits at END.

The right map tells the same story head by head. Patching the duplicate-token head's output at S2 restores 0.52. Patching the S-inhibition head's output at END restores 0.52. Patching the name mover's output at END restores everything.

The next cell puts denoising and noising side by side after layer 2, under both metrics.

After layer 2, S2 and END form an AND. Noising S2 destroys 48 percent of the behavior and noising END destroys 135 percent, so each is necessary. (More than 100 percent means the name mover then prefers $B$.) Denoising one of them restores about half at most, so neither is sufficient. Only both together restore everything.

Under the probability metric the same patches look different. Denoising END restores 0.52 of the logit difference but only 0.16 of the probability. The probability of $A$ is 0.15 on the corrupted run and 0.72 on the clean one, and half the logit difference moves it only a short way along the softmax. A probability heatmap would make the S-inhibition head look minor.

The widget runs the same experiments, with the residual stream on top and the head outputs below. Switch between denoising and noising, between the two metrics, and between swapping the S2 name and Gaussian noise at two scales.

Turn on Gaussian noise. It corrupts the embeddings of all three names, so it no longer asks whether S2 repeats a name, and the map changes completely. At large noise scales some effects fall far outside 0 to 1, a sign of activations unlike anything the model was built to process. In a hand-built model we know which picture is right. In a trained model we would not, which is why the corruption is part of the experiment and must be reported with it.

import numpy as np

def patch_effects(run, clean, corr, sites, metric, direction="denoise"):
    raise NotImplementedError

Ablations, and what to replace an activation with

Patching swaps in a value from a second, carefully matched prompt. Often we only want to know what happens when a part is switched off, which is what the knockouts in the definition of a circuit do. Ablating a part means switching it off. But switching off still means replacing the activation with something, and the choice of something matters.

Zero is arbitrary. Later layers often use a part's average output as a constant input, in effect a bias. Zeroing removes that constant along with the information, so zero ablation tends to overstate importance. Wang et al. (2022) found that it gave noisy results, perhaps for this reason.

Mean ablation keeps the constant and removes only what varies across $\mathcal{D}_{\text{ref}}$. So the choice of $\mathcal{D}_{\text{ref}}$ is the real decision. Wang and colleagues took means over their ABC distribution, with three unrelated names, one mean per template. A mean over the IOI prompts themselves would keep facts that hold on every one of them, such as "some name is repeated". The mean can also be a vector that no single input ever produces.

Resample ablation always uses a real activation from a real input. That is why causal scrubbing, a method we meet near the end of the section, is built on it. The price is variance, so it is averaged over many draws.

In the mini-IOI model all three agree for the S-inhibition head. On ABC prompts no name repeats, so the head attends to BOS and writes almost nothing. Zero, the mean, and a resampled value are then all about the same. The exercise builds a head where they disagree.

import numpy as np

def ablation_drops(run, clean_set, ref_set):
    raise NotImplementedError

So when you read or write an ablation result, ask what the activation was replaced with, and over which distribution the mean was taken. A zero ablation without a reason is a red flag, in a paper and in an interview.

Causal tracing in GPT-2 XL

The first large-scale use of patching in a language model was about facts. Meng et al. (2022) asked where GPT-2 XL, a 1.5-billion-parameter model, recalls that the Space Needle is in Seattle. They corrupted the prompt by adding Gaussian noise to the embeddings of the subject's tokens ("Space Needle"). Then they restored one clean hidden state, the residual stream at one layer and one position, into the noisy run. How much of the right answer comes back shows where the fact flows.

In plain words, causal tracing is denoising with probability as the metric. The IE is the numerator of the denoising effect, and $\operatorname{IE}/\operatorname{TE}$ is $\operatorname{eff}(\mathbf{h})$ for a single prompt.

The paper found two sites. A late site, near the prediction, was expected. An early site, in the middle layers at the last token of the subject, was new. That localization is what the same paper's rank-one model editing method (ROME) used to choose which MLP's weights to edit to change a stored fact. Section 23.7 takes up editing, along with evidence that editing success does not track tracing results as closely as one might hope.

Path patching

Patching an activation carries every path that leaves it. In the mini-IOI model the duplicate-token head restores 0.52 when patched. That number does not say whether it acts through the S-inhibition head, through the name mover directly, or straight on the logits. To ask about one path, let the clean value travel along that path and hold every other path fixed.

In plain words, path patching sends the clean signal down one road and closes the others. In the three-number example, it is exactly the computation that gave 2 for the path $h_1 \to h_2 \to y$. Wang and colleagues ran it in the noising direction, starting from the IOI prompt and patching in activations from the matching prompt of their ABC distribution. The direction changes only which run is the base.

The duplicate-token head has no direct effect and no effect through the name mover. Its whole effect of 0.52 flows through the S-inhibition head. The S-inhibition head in turn acts only through the name mover, and the name mover acts only on the logits. Three path patches recover the wiring diagram we built, which no single heatmap could. This is how real circuits are traced, one receiver at a time, working back from the logits.

import numpy as np

def path_patch(run_with_overrides, clean, corr, sender, receiver):
    # placeholder: activation patching of the sender, which also counts every other path
    v_clean, v_corr = run_with_overrides(clean), run_with_overrides(corr)
    return run_with_overrides(corr, {sender: v_clean[sender]})["out"] - v_corr["out"]

Reading composition from the weights

Interventions tell us which paths a model uses on a set of prompts. In a Transformer with attention and no MLPs, the possible paths can also be read from the weights, before running anything. This is the program of Elhage et al. (2021). They carried over to Transformers the circuits agenda that Olah et al. (2020) began on vision models, which reads a network's algorithm off its weights.

One layer is a bigram table plus skip-trigrams

Strip a Transformer to attention only, with no MLPs and no LayerNorm. For the derivation, drop the position embeddings too. Adding them changes the rows of the embedded stream but not the argument.

Here are the symbols. $\mathbf{X}_{\text{tok}} \in \{0, 1\}^{T\times V}$ holds the one-hot tokens as rows, for $T$ positions and a vocabulary of $V$ tokens. $\mathbf{E} \in \R^{V\times d}$ is the embedding and $\mathbf{W}_U \in \R^{d\times V}$ the unembedding. Head $i$ has the attention pattern $\mathbf{A}^{(i)} \in \R^{T\times T}$ and the OV circuit $\mathbf{W}_{OV}^{(i)} \in \R^{d\times d}$. The heads of a layer add up (§4.5), so for input stream $\mathbf{X}$ the layer's output is $\sum_i \mathbf{A}^{(i)}\mathbf{X}\mathbf{W}_{OV}^{(i)}$.

In plain words, each term is a table indexed by tokens. The $V\times V$ matrix $\mathbf{E}\mathbf{W}_U$ is a bigram table. It maps the current token to next-token logits, the direct path of §5.1.

Each head adds two more $V\times V$ tables. The QK table $\mathbf{E}\mathbf{W}_{QK}^{(i)}\mathbf{E}^\top$ says how much a current token $w_t$ wants to attend to an earlier token $w_j$. The OV table $\mathbf{E}\mathbf{W}_{OV}^{(i)}\mathbf{W}_U$ says how attending to $w_j$ changes the logits. Together they implement skip-trigrams, rules of the form "if $w_j$ appeared earlier and the current token is $w_t$, raise the logit of $v$". For example, an earlier "keep" with a current "in" can raise "mind".

The factored form has a cost, which Elhage et al. (2021) point out. A head that raises both "keep ... in mind" and "keep ... at bay" must also raise "keep ... in bay" and "keep ... at mind". All these tables can be read off the weights without running the model. The copying score of §6.8, which asks whether a head's OV table maps each token to itself, reads exactly this table.

Two layers add virtual heads and three kinds of composition

Add a second layer with heads $j$. Its input is the first layer's output stream, $\mathbf{X}^{(1)} = \mathbf{X}^{(0)} + \sum_i\mathbf{A}^{(i)}\mathbf{X}^{(0)}\mathbf{W}_{OV}^{(i)}$. Substituting it into the second layer gives four kinds of terms in the logits,

The last term is the subtle one. It looks exactly like a single head with attention pattern $\mathbf{A}^{(j)}\mathbf{A}^{(i)}$ and OV circuit $\mathbf{W}_{OV}^{(i)}\mathbf{W}_{OV}^{(j)}$, which Elhage and colleagues call a virtual head. (The earlier head's matrix comes first, because a row vector meets each matrix in turn from the left.) This is V-composition, the second head moving what the first head moved.

The expansion is exact, but it hides a dependence. The pattern $\mathbf{A}^{(j)}$ is computed from $\mathbf{X}^{(1)}$, so the second layer's queries and keys can read layer-1 outputs too. That is Q-composition and K-composition. It lets layer 2 compute attention patterns no single layer can. The induction head of §4.7 is the standard case, since its keys read what a previous-token head wrote. The cell checks the expansion on a random two-layer model.

The four terms add up to the logits to rounding error. The virtual heads are far from negligible on random weights, with a root-mean-square size of 1.24 against 3.68 for the direct path.

Composition scores

A composition score measures, from the weights alone, how strongly a later head reads what an earlier one writes. Here is where the matrix comes from. The later head's scores are $s_{tj} = \mathbf{x}_t\mathbf{W}_{QK}^{(2)}\mathbf{x}_j^\top$. Suppose the key-side stream $\mathbf{x}_j$ contains a write $\mathbf{y}_j\mathbf{W}_{OV}^{(1)}$ from the earlier head, where $\mathbf{y}_j$ is the blend of stream vectors that the earlier head read from position $j$. Then the score gains

so the matrix $\mathbf{W}_{QK}^{(2)}\mathbf{W}_{OV}^{(1)\top}$ governs K-composition. The same computation on the query side, $(\mathbf{y}_t\mathbf{W}_{OV}^{(1)})\mathbf{W}_{QK}^{(2)}\mathbf{x}_j^\top$, gives $\mathbf{W}_{OV}^{(1)}\mathbf{W}_{QK}^{(2)}$ for Q-composition. The value side gives $\mathbf{W}_{OV}^{(1)}\mathbf{W}_{OV}^{(2)}$ for V-composition.

To turn a matrix into one number we use the Frobenius norm $\lVert\mathbf{M}\rVert_F = \sqrt{\sum_{ij}M_{ij}^2}$, the length of the matrix read as one long vector. Dividing by the norms of the two factors removes their scale.

In plain words, a composition score measures how much of what head 1 writes lands where head 2 reads, with its queries, its keys, or its values. The scores depend only on the products $\mathbf{W}_Q\mathbf{W}_K^\top$ and $\mathbf{W}_V\mathbf{W}_O$. That is as it should be, since §4.2 showed that the individual factors have no meaning of their own.

What values should we expect? Write $\lVert\mathbf{M}\rVert_2$ for the spectral norm of $\mathbf{M}$, its largest singular value, which is the most it can stretch a unit vector (§0.10).

In plain words, a perfect one-direction match scores 1. A perfect match spread over $r$ equal directions scores only $1/\sqrt r$. For two unrelated random heads, such as heads at initialization, the mean square of the score is exactly $1/d$ whatever their head width. For heads of 8 or more dimensions the score itself sits within a percent of $1/\sqrt d$. Only very narrow heads score noticeably less on average (about $0.8/\sqrt d$ for width 1, where the score is the absolute cosine between two random directions).

Parts 2 and 3 together explain why raw scores look small. A head pair that composes perfectly through $r$ equally used directions scores only $1/\sqrt r$. So the only meaningful comparison is with random heads of the same shapes. Elhage et al. (2021) subtract that random baseline, measured empirically, before reporting scores.

The cell checks the example and the random baseline.

The random scores match $1/\sqrt d$, and their mean square matches $1/d$ to within half a percent, as part 3 predicts. In the mini-IOI model the scores light up exactly where the design put them.

The duplicate-token head feeds the S-inhibition head's keys (0.312) and values (0.316). The S-inhibition head feeds the name mover's queries (0.283). Every other entry is 0, against a random baseline of 0.136. The weights alone recover the wiring that path patching found by intervention.

In a trained model the two views can disagree. Weights say what could flow, on any input. Interventions say what does flow, on the prompts we chose. Careful work uses both, and an interviewer who asks "how would you know two heads compose?" wants to hear both.

import numpy as np

def composition_scores(W_Q1, W_K1, W_V1, W_O1, W_Q2, W_K2, W_V2, W_O2):
    raise NotImplementedError

def random_baseline(d, d_head, trials, rng):
    raise NotImplementedError

Attribution patching

Activation patching costs one forward pass per site, and at real scale that is the bottleneck. Attribution patching replaces the sweep with one gradient (Nanda 2023). Here is how large the gap is.

Here is the idea. Fix one site, and view the metric of the corrupted run as a function of the value $\mathbf{v}$ placed at that site, $f(\mathbf{v}) = m(x_{\text{corr}};\ \mathbf{h} \leftarrow \mathbf{v})$, with everything downstream recomputed. Denoising measures $f(\mathbf{h}_{\text{clean}}) - f(\mathbf{h}_{\text{corr}})$. If $f$ were a straight line between the two values, its slope at one end would give the whole change.

In plain words, the estimate for a site is "how far the patch moves the activation" dotted with "how much the metric cares about that activation on the corrupted run". The gradient for every site comes out of one backward pass on the corrupted run. Backpropagation computes the derivative of one number with respect to every intermediate activation at once. So attribution patching costs two forward passes, to cache $\mathbf{h}_{\text{clean}}$ and $\mathbf{h}_{\text{corr}}$ everywhere, and one backward pass, whatever the number of sites.

You have met this estimate before. In §23.1, gradient times input explained an output by a first-order Taylor expansion, and here the expansion is taken around the corrupted activation. The exact path integral is then integrated gradients along the straight line from $\mathbf{h}_{\text{corr}}$ to $\mathbf{h}_{\text{clean}}$, exact at the price of many gradients per site. Later work linearizes at the clean run instead, which estimates noising effects with the same derivation (Syed et al. 2023; Kramár et al. 2024).

The cells below compare the estimate with true patching on a random two-layer attention-only model, for every query, key, and head output at every position. A sharpness parameter multiplies the attention scores, so larger values push the softmax toward one-hot weights. NumPy has no automatic differentiation, so the cell gets each directional derivative $\boldsymbol\delta\cdot\nabla f$ from a central finite difference, which matches the gradient to many digits. The PyTorch block further down gets all of them from one backward pass.

With mild attention, scores scaled by 0.5, the estimate tracks the truth with correlation 0.99. With sharp attention, scale 4, the correlation falls to 0.50. The last-layer head outputs stay exactly on the diagonal at every setting. The metric is a linear readout of them, so $\nabla^2 f = \mathbf{0}$ there.

Where the linear approximation fails

The remainder term says where to expect trouble, wherever the metric curves sharply between the two values of the activation. Kramár et al. (2024) found two systematic failures. Both produce false negatives, sites with large true effects that attribution patching scores near zero.

Attention saturation. Queries and keys feed a softmax. Suppose the clean attention weight is near 1 and the corrupted one near 0, or the reverse. The softmax's derivative vanishes as its weights approach one-hot (§4.3). So the gradient at either end is nearly flat, and the linear estimate misses the jump. Their QK fix computes the attention pattern that the patched query or key actually produces, which is cheap since it needs only that head's scores. It linearizes only the rest of the model, $\widehat{\operatorname{AtP}}_{\text{fix}} = (\mathbf{A}_{\text{patched}} - \mathbf{A}_{\text{corr}})\cdot\nabla_{\mathbf{A}}m$. In the cell, the fix lifts the correlation at sharpness 4 from 0.50 to 0.73. It makes the layer-2 queries and keys exact, since nothing after their softmax is nonlinear.

Cancellation. A part's direct and indirect effects can nearly cancel. Then a small relative error in the linearized indirect part swamps the sum, and the estimate lands near zero.

The widget below runs the same comparison on a model of the same kind, with sliders. Each dot is one query, key, or head output at one position, placed across by its true patching effect and up by its estimate. Filled dots belong to layer 2 and hollow ones to layer 1. A large effect counts as missed when it is over a fifth of the largest true effect but its estimate is under half of it.

Raise the sharpness and watch every site whose effect passes through a softmax scatter away from the diagonal, while the layer-2 head outputs stay on it. Turn on the QK fix and the layer-2 queries and keys snap back onto the diagonal. Layer-1 sites can still miss, queries and keys as well as head outputs, because layer 2's softmax still lies after them. Raise the corruption size instead, and every curved site drifts, as the $\lVert\boldsymbol\delta\rVert^2$ in the remainder says it should. The practical rule follows. Rank sites with gradients, then verify the top candidates with real patching.

Patching with hooks in PyTorch

In PyTorch, patching is a forward hook, a function attached to a module that runs after the module and may return a replacement output (§0.9). Attribution patching is one backward() call with the activations' gradients retained.

On this model the estimate correlates 0.99 with the 80 true patches. The last layer's estimates are exact to about $10^{-7}$, for the price of three passes instead of 80.

import numpy as np

def attribution_patch(model, x_clean, x_corr):
    raise NotImplementedError

The indirect object identification circuit in GPT-2 small

Back to the question we opened with. Wang et al. (2022) built the IOI dataset from 15 templates filled with random single-token names, places, and objects. They measured the behavior by the logit difference between the indirect object and the subject at the final position, END. Then they looked for the heads that produce it.

The algorithm is the one the mini-IOI model hard-codes. Find the names, mark the one that is repeated, and copy the other. What makes the result remarkable is how GPT-2 learned it. It learned it from web text, spread it over 26 heads, made it redundant, and added heads that push against it.

How the circuit was found

Wang and colleagues worked backward from the logits.

1. Path patching from each head to the logits at END, in the noising direction with ABC prompts, found the heads with large direct effects. They were positive for 9.6, 9.9, and 10.0 and negative for 10.7 and 11.10. Their attention patterns, and the projections of their outputs onto the name unembeddings (direct logit attribution), showed what they do.
2. Path patching from every earlier head into the name movers' queries found the S-inhibition heads.
3. Path patching into the S-inhibition heads' values found the duplicate token and induction heads. Behind the induction heads' keys sat the previous token heads, K-composition as in §4.7.

Each class was then checked with targeted experiments. For example, duplicate token heads were shown to attend to earlier copies of the current token on random sequences, not only on IOI sentences.

The paper is also candid about what it left out. All its interventions were on attention heads, never on MLPs. Knocking out any single MLP other than the first left the task intact, but knocking out all of them after the first destroyed it. So the circuit is a statement about heads, given working MLPs.

Faithfulness, completeness, and minimality

A circuit that reproduces the model's average output could still be the wrong explanation. It might capture only one of two redundant mechanisms, for instance. Wang and colleagues proposed three tests. Write $\bar\Delta(C)$ for the mean logit difference over the IOI distribution when the circuit $C$ runs with everything outside it knocked out (their $F(C)$). Write $C \setminus K$ for $C$ with a further set $K$ of nodes knocked out.

In plain words, faithful means the circuit can do the job alone. Complete means nothing outside it can quietly do the same job. Minimal means nothing inside it is dead weight.

Completeness is a search over exponentially many sets $K$, so it can only be refuted, never fully verified. Wang and colleagues sampled $K$ three ways, at random, as whole classes of heads, and by a greedy search that tries to make the incompleteness score as large as possible. The first two found only small scores. The greedy search found sets with incompleteness up to 3.09, 87 percent of the original logit difference. The authors report this as a gap in their understanding. For minimality, every head in the circuit had a score of at least 1 percent of $\bar\Delta(M)$ for a suitable $K$.

So the IOI circuit is faithful and minimal, and its completeness is open. In an interview, that is the honest one-line summary of the best-known circuit in a real language model.

Backup heads and self-repair

When Wang and colleagues knocked out all three name mover heads at once, the logit difference dropped by only 5 percent. Other heads, the backup name movers, started doing the name movers' job. That is the OR met with the two directions of patching, where either part is enough on its own. So knocking out any one name mover makes the name movers look far less important than they are.

This behavior is called self-repair, later parts of a network compensating when an earlier part is removed. McGrath et al. (2023) found that ablating one attention layer of a language model (Chinchilla 7B) makes later layers compensate, which they call the Hydra effect. It appears even in models trained without dropout, the training trick of randomly zeroing parts of a network (§0.7). Wang and colleagues had guessed that dropout was the cause.

Self-repair is why single ablations understate importance. A part can be essential to how the model normally works and still look unimportant when removed alone, because something else covers for it. It is also why the measured effect of a head depends on what else has been ablated, which is what the completeness test probes.

Reproducing it at home

The code below runs both patching sweeps on GPT-2 small with TransformerLens, a library whose models expose every activation as a named hook. It needs a download, so it is shown here rather than run.

Automating circuit discovery

The IOI circuit was found by hand, one path patch at a time. Conmy et al. (2023) observed that one step of that work follows a fixed recipe, deciding which edges of the computational graph matter for a behavior. They automated it.

Their measure of quality is the KL divergence, a measure of how far one probability distribution is from another (§0.5). An edge is removed by making its receiver read the sender's value from the corrupted run, as in path patching.

In plain words, ACDC prunes edges greedily, from the output back, keeping only those whose removal would change the output distribution noticeably. The threshold $\tau$ trades the circuit's size against its fidelity.

ACDC still costs one forward pass per edge it tries. Edge attribution patching (EAP) scores every edge at once with the attribution-patching formula applied to edges. An edge's score is the change in the sender's output times the gradient at the receiver's input. The top-scoring edges are kept, for two forward passes and one backward pass in total. Syed et al. (2023) found that, averaged over tasks, it recovered known circuits better than ACDC. They scored each method by its ROC curve, the fraction of the known circuit's edges found against the fraction of other edges wrongly kept as the threshold varies, and compared the areas under the curves. At scale, the recipe is again to rank with gradients and verify with patching.

import numpy as np

def acdc(nodes, edges, run, tau):
    return list(edges)                    # placeholder: keeps every edge

The exercise shows ACDC's main weakness in miniature. With redundant paths, a greedy pass can keep either one depending on the order, and the threshold decides whether a weaker backup counts as part of the circuit. Conmy et al. (2023) report that in one of their experiments the order of parents changed the result. An interviewer who asks how you would automate circuit discovery will follow up on exactly this.

Testing a whole hypothesis

Patching one site, or one path, tests one claim. A circuit is a claim about many parts at once and about what each of them computes. Two lines of work test whole hypotheses, causal scrubbing and causal abstraction. Both replace activations with their values on other inputs that the hypothesis says should not matter, and check that the output survives.

A fully reverse-engineered algorithm

Every circuit so far was a partial explanation, a set of parts and a story. One case went all the way. Nanda et al. (2023) trained a one-layer Transformer on modular addition, computing $a + b \bmod p$, and explained exactly how it finds the answer.

The algorithm they found is called Fourier multiplication. In plain words, the model turns each number into positions on a few circles, adds the angles, and checks which answer sits at the resulting angle on every circle.

In symbols, the embedding maps $a$ and $b$ to $\cos(w_k a), \sin(w_k a), \cos(w_k b), \sin(w_k b)$ for each key frequency. Attention and the MLP compute $\cos(w_k(a + b))$ and $\sin(w_k(a + b))$ from them with the angle-addition identities, for example $\cos(w_k(a + b)) = \cos(w_k a)\cos(w_k b) - \sin(w_k a)\sin(w_k b)$. The unembedding then reads off, for each candidate answer $c$, the logit

One frequency already has its maximum in the right place, so why use five? Because the margin matters. The cell builds the logits from the five key frequencies for all $113^2$ pairs.

The strongest evidence was an ablation in Fourier space. Removing the five key frequencies dropped performance to chance, while removing all the other frequencies slightly improved it.

The explanation then became a measuring instrument. The restricted loss keeps only the key frequencies, and the excluded loss removes them. Both change smoothly long before test accuracy jumps. With them, training splits into three phases, memorization, circuit formation, and cleanup, in which weight decay removes the memorizing parts. The sudden jump in test accuracy happens during cleanup, after the general algorithm is already in place. So grokking, sudden on the outside, is gradual on the inside. That is what a complete mechanistic explanation can buy, a progress measure nobody could have defined without it.

Where circuit analysis breaks

The table collects the methods of the section, what each answers, what it costs, and how it fails.

Read more

• Elhage et al. (2021). A Mathematical Framework for Transformer Circuits. Path expansion, virtual heads, Q-, K-, and V-composition, composition scores, and induction heads in attention-only models.
• Wang et al. (2022). Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small. The largest hand-found circuit in a real language model, path patching, and the faithfulness, completeness, and minimality tests.
• Meng et al. (2022). Locating and Editing Factual Associations in GPT. Causal tracing, the total and indirect effects of hidden states, and the localization that ROME builds on.
• Heimersheim and Nanda (2024). How to use and interpret activation patching. The practical guide, with denoising versus noising, sufficiency versus necessity, and AND and OR circuits.
• Zhang and Nanda (2023). Towards Best Practices of Activation Patching in Language Models: Metrics and Methods. How the choice of corruption and metric changes the answer, and why to prefer token replacement and logit difference.
• Kramár et al. (2024). AtP*: An efficient and scalable method for localizing LLM behaviour to components. Attribution patching studied systematically, its two failure modes, and the QK fix and GradDrop that repair them.
• Conmy et al. (2023). Towards Automated Circuit Discovery for Mechanistic Interpretability. ACDC, the greedy edge-pruning algorithm, validated against hand-found circuits.
• Nanda et al. (2023). Progress measures for grokking via mechanistic interpretability. A complete reverse-engineering of a trained network, and how the explanation turned into progress measures for its training.

Watch next

▶

▶

▶

▶