Python · cell 1
An attribution graph is a map of how a model computed one answer on one prompt. Its nodes are the input tokens, the model's candidate next tokens, and, in between, features, directions in the model's activations that each stand for one recognizable concept such as "Texas". Its edges say how much each node pushed each later node, and each node's input is exactly the sum of the edges into it. This section builds such a map from scratch, measures how much of the computation it captures, and tests it by intervening on the model. Then it looks at what maps of a frontier model have shown, at how interpretability is used to audit real models in 2026, and at what is still unsolved.
Here is the question that motivates it. Ask Claude 3.5 Haiku to complete Fact: the capital of the state containing Dallas is and it answers "Austin". It could get there in two steps, from Dallas to Texas and then from Texas to its capital. Or it could have seen enough similar sentences to jump straight from "Dallas" and "capital" to "Austin". Both stories give the same answer, so no test of the model's outputs on this prompt can tell them apart.
The tools of this chapter so far cannot tell them apart either. Activation patching (§23.4) copies an activation from one run into another and sees whether the answer follows. It shows which components and positions matter, but an intermediate fact like "Texas" would be computed inside the MLP layers, as a pattern spread over thousands of neurons, and patching does not say what that pattern means. A probe (§23.2) is a small classifier trained to read a variable from the activations. A probe for "Texas" could show that the information is present without showing that the answer uses it. Steering (§23.7) adds a direction to the activations and watches the output change. It shows that one direction matters, which is not the same as tracing a computation.
The missing ingredient came from §23.6, which learned dictionaries of sparse features that read like concepts. This section stitches such features into a graph of one whole forward pass. §23.1 ranked interpretability evidence on a ladder of four rungs, association, direct effect, causal effect, and mechanism. On that ladder an attribution graph is a hypothesis about mechanism. It becomes evidence only after interventions on the real model confirm it, and we keep the two apart throughout. This matters in practice because the same tools are now used to audit frontier models before release, where a convincing but wrong story is worse than none.
The plan has three steps. Replace every MLP with an interpretable stand-in. Freeze the parts of the computation we are not trying to explain. Add back whatever the stand-in gets wrong, so nothing is lost. After these three steps, every feature's input is an exact sum of contributions from earlier features, and those contributions are the edges of the graph.
First, the conventions. The chapter writes vectors as rows. A token's residual stream, the running vector that every layer reads from and adds to (§5.1), is a row vector, and a weight matrix acts as . The model has blocks, each an attention sublayer followed by an MLP sublayer.
We need names for three vectors at each position . Write for the stream after blocks, so is the token embedding plus position. Write for the stream inside block , after its attention and before its MLP. Write for what the MLP adds, so .
The MLP does not read directly. A normalization layer first divides it by a per-token scale (§5.2). For RMSNorm the scale is the root mean square of the vector, and for LayerNorm it is the standard deviation after centering. So the MLP reads . Normalization layers also multiply by a learned gain, one number per coordinate, and we absorb it into the next weight matrix by scaling that matrix's rows, which changes nothing. One warning about letters. In this section and also name a source node and a target node of the graph, which is why positions are written . A scale always carries a layer superscript, and a node never does.
A transcoder is a wide, sparse MLP trained to predict an MLP's output from that MLP's input (§23.6, after Dunefsky et al. 2024). It has many more hidden units than the MLP it imitates, and on any one input almost all of them are zero. Each hidden unit is a feature. Feature computes a preactivation , the dot product of the input with its encoder column, and passes it through a threshold. Its activation is the result, and when it is nonzero the feature adds times its decoder row to the output. Stacking the features, and .
The attribution-graph paper writes the transcoder without biases and uses a sharper threshold, the JumpReLU, (Ameisen et al. 2025). It keeps a preactivation unchanged when is above the threshold and sets it to zero otherwise. For example, with it maps 0.4 to 0 and 1.3 to 1.3. We do the same.
Why a transcoder and not a sparse autoencoder (SAE) on the residual stream? An SAE decomposes what the stream holds at one layer. To get from SAE features at one layer to SAE features at the next, you still pass through the MLP, a nonlinear map. So the effect of one SAE feature on another is not a fixed number you can read off. A transcoder replaces the MLP. Its features take the MLP's input and produce its output, so they stand in for the MLP's nonlinearity instead of having to pass through it. Two transcoder features in different layers then interact only through the residual connections and attention. Both of those are linear once the attention weights and the normalization scales are held fixed. Everything in this section rests on that linearity, and an interviewer will ask you to explain where it comes from.
In a deep model, several MLP layers in a row often work on the same thing. Take the prompt Zagreb:Croatia::Copenhagen: in an 18-layer model. With one transcoder per layer, the graph shows a path of length 7 made entirely of "Copenhagen" features. The stream grows larger with depth, and the normalization divides every read by that growing size, so an early write fades unless later layers keep amplifying it. The graph then repeats the same fact seven times. A cross-layer transcoder (CLT) lets one feature write to every later layer at once. On the same prompt it collapses the chain into features at the first layer (Ameisen et al. 2025).
In plain words, each feature looks at the stream once, at its own layer, and may then leave a different message in every MLP output above it. A per-layer transcoder is the special case in which every cross-layer decoder is zero.
The payoff is shorter explanations. On a capital-city analogy prompt, the average path from embedding to logit had length 2.3 in a graph built from a 10-million-feature CLT, against 3.7 for per-layer transcoders with the same total number of features (Ameisen et al. 2025). The CLT was also cheaper for its quality. A 1-million-feature CLT beat a 10-million-feature set of per-layer transcoders at every sparsity level compared, on two measures, the error of reconstructing the MLP outputs and the KL divergence (the standard measure of how far one probability distribution is from another) between the next-token distributions of the model and of the replacement. It used 5 to 10 times less training compute.
The price is that a CLT's causal structure differs from the model's. If the model really runs a chain of amplifications over many layers, the CLT hides the chain inside one feature. That is a small instance of mechanistic unfaithfulness, a stand-in that gets the outputs right by a different route, and we will meet it again among the open problems. In an interview, be ready to give both sides, shorter paths and less compute against a structure the model does not have.
Swap every MLP for the CLT and you get a replacement model, a whole network built from interpretable parts. It is imperfect, because each layer's small reconstruction error feeds the next layer and the errors compound. So for one prompt we go further. We freeze the attention patterns and the normalization scales, the nonlinear parts we are not explaining, at the values they took on that prompt. Then we add back each layer's error as a constant. The result reproduces the original model exactly on that prompt.
On the prompt the local replacement model reproduces every activation and every logit of the original model, by construction. Each MLP output is rebuilt as , and everything else is the original computation with its recorded values. The authors compare it to a Taylor expansion, which is exact at one point and drifts as you move away. Exact agreement on says nothing about whether the two models use the same mechanism. Measuring that is the job of the interventions later in the section.
How good is the plain replacement model, without the errors? The largest CLT of the 18-layer research model gave the same top next token as the underlying model on 50 percent of a set of pretraining-style prompts. The prompts were chosen so that the model is correct but less than 80 percent confident, which leaves out the easy tokens. A stand-in that disagrees with the model on half of these tokens cannot explain them on its own, which is why graphs are built from the local model and not the plain one.
Two measures in the next box need names. The normalized reconstruction error is the squared error of the reconstruction divided by the squared size of the true output, , averaged over tokens and layers. It is roughly the fraction of the MLP output that the CLT misses. The L0 of a dictionary is the average number of features active on one token.
Freeze the attention patterns and the normalization denominators, and the stream between two MLPs becomes a linear map. Go through the pieces one at a time. The residual connection is the identity. Attention with a fixed pattern is a fixed weighted average of earlier positions followed by the head's OV map, the matrix that turns what a head reads into what it writes (§4.2). Dividing by a frozen scale is multiplying by a constant, and a LayerNorm's centering step is linear too. The MLP outputs are no longer computed from the stream at all. They are whatever the active features write, plus a constant error. So the only nonlinearity left anywhere is each feature's own JumpReLU.
In general, write for the matrix that maps a small change written into to the resulting change of , with everything frozen and the MLP outputs held fixed. Block sends a change to
where is the frozen weight with which head of block at position attends to position , and is the frozen scale of the attention input. (A LayerNorm would first center , which is also linear.)
In plain words, a change either stays where it is, through the residual, or hops to a later position through one head, scaled by the attention weight and the OV map. Composing blocks to multiplies these maps. Expanding the product gives one term per path, a choice at each block of the identity or one head. §23.4 wrote a whole attention-only model as this kind of sum over paths. Here the patterns are frozen at their values on one prompt instead of being treated as constant on every prompt.
With everything frozen, each block is an affine map of the stream. The model's only inputs are now the embeddings, the features, and the errors, so the stream is a linear function of those three. Concretely, is the embedding pushed through blocks to , plus, for every layer , whatever was written into pushed through blocks to .
The writes into are , position by position. Pushing a write at position to position is right-multiplication by , so
Multiply on the right by to get . Collect, for each feature , the terms that carry its activation. They come from every layer with , which is the sum in . The embedding and error terms are items 1 and 2. Only active features appear, because for the others.
For the derivative form, is linear in once the frozen quantities are constants, and the coefficient of is . ∎
In plain words, on this one prompt, every feature's input is a sum of contributions, one per upstream node. Each contribution is the upstream node's activation times a number fixed by the prompt's frozen attention and norms. Nonlinearity enters only when the sum passes through the feature's threshold. A real model also has constant terms that depend on no node, such as biases and normalization shifts. Their contribution reaches along the same linear paths, so an exact account either gives them input nodes of their own or reports the edges as summing to minus that constant. Our toy and the PyTorch block below have no such terms.
Two consequences matter in practice. The first is cost. A vector-Jacobian product is one backward pass of autograd from a single scalar, and it gives that scalar's gradient with respect to every earlier activation at once (§0.3). A stop-gradient treats a quantity as a constant during that backward pass, which is what detach() does in PyTorch. Inject at the target's site and run one backward pass with stop-gradients on everything frozen. Every source's edge is then the dot product of what the source writes with the gradient where it writes. So building a graph costs one backward pass per target node, and the number of passes grows linearly with the number of active features.
The second is that logits are attributed the same way. Ameisen et al. (2025) attribute the target logit minus the mean logit over the vocabulary, so that an edge measures how much a source favors this token over the rest.
A third point is what the decomposition does not need, a baseline. Integrated gradients (§23.1) also splits an output into contributions that add up exactly, but only relative to a baseline, a reference input you must choose, and the answer changes with that choice. Here the sum is exact with no baseline at all. The cost was paid up front, in freezing the parts we are not explaining.
Notice what the graph accounts for. It shows the information that frozen attention carried from one position to another, the OV half of each head. It never shows why a head attended where it did, the QK half, which sets the attention pattern from the query and the key. We come back to that gap after the case studies.
The cell stores the edges in a matrix indexed (target, source), so row holds the edges into , and checks the sums. We will reuse this graph twice more.
The last three lines compute an influence, which we define in a moment. First, let us build a real graph.
To see the whole pipeline, we build a three-layer model whose computation we know, because we designed it to contain both stories of the opening. It reads three tokens, "capital", "Dallas", and "is", one per position. Each coordinate of its stream has a name, so we can read it. Coordinates written in capitals, such as DALLAS, are the ones the features write and read.
The stream also carries a large constant "bulk" coordinate, so normalization is real but changes little when one feature moves. The first cell lays out the stream and the transcoder.
The underlying model's MLPs are close to the transcoder but not equal to it. They are 5 percent stronger, and they add a small random linear term to every output. When we raise the argument hidden above zero, they also contain a layer-2 unit that writes TEXAS and that the transcoder lacks. We call it the missing unit. Every decoder here writes only to its own layer, so this is a per-layer transcoder, the special case of a CLT. Exercise 23.8a handles cross-layer decoders.
Now run the prompt, read the features off the underlying model, and form the error nodes, true MLP output minus what the transcoder writes.
Of its three output tokens, the model gives "Austin" probability 0.90. The plain replacement model, with no error terms, already drifts to 0.869. The errors are small, at most 0.083 in size against stream vectors of size about 8. Some layer-2 and layer-3 features also fire at earlier positions ("say Austin" at "Dallas", for example). Nothing after the layer-2 head moves information between positions, so those features cannot affect the answer at the last position, and the graph will give them zero influence.
Next comes the backward pass with everything frozen. It returns the gradient of one target's preactivation with respect to the stream after every earlier layer.
Each edge is what the source writes, dotted with the gradient where it writes. One backward pass per target fills a whole row of the (target, source) matrix, and the theorem says every row must sum to its target's preactivation.
The sums agree to rounding error, about . Let us check one edge by hand against the path expansion. The "Dallas" feature at position "Dallas" reaches "Texas" at position "is" along exactly one path. It goes through its decoder, then the layer-2 head with attention weight 0.45 (divided by the frozen scale of the attention input), then Texas's encoder over its own frozen scale.
Every number is exact. The transcoder's approximation has not gone away, but it is now on display, as error edges we can measure.
This is the edge computation of the theorem, for a model whose features may write to several layers. The setup builds a frozen model with layers, positions, and one attention head per layer. The head's normalization scale is already folded into its frozen pattern, so it reads the raw stream. The stream starts at the embeddings X0. Layer then does three things.
sigma[l][c].f[l] of shape (T, F), decoders W_dec[l'][l], and errors err[l].Layers are numbered from 0. A feature in layer therefore sees decoder writes from layers .
Implement edge_weights(model). It returns a dictionary that maps every feature target ("feat", l, t, i), active or not, to a dictionary of its incoming edges. (A preactivation splits into edges whether or not it passes the threshold.) The sources are keyed ("emb", s) for each position's embedding, ("err", l_s, s) for each error with , and ("feat", l_s, s, j) for each active feature with . The tests check that each target's edges sum to its preactivation, that an edge scales with the source's activation and never the target's, and that a source with no path to a target gets an edge of zero.
In a real model you never write the backward pass by hand. You run the model once and record each attention pattern and normalization denominator. Then you rerun it with those quantities detached and with each MLP output replaced by . Here and are leaf tensors, inputs that autograd computes gradients for. In a real codebase, forward hooks, functions that PyTorch calls on a module's output, do the swapping (§0.9).
The frozen model is linear in its inputs with no constant term. For such a function the output equals the sum, over all inputs, of input times gradient. For example, at gives . That is the theorem again. The block also shows the contrast. With the softmax and the norms left live, gradient times input does not add up.
With everything frozen, the target's preactivation of about 2.6079 equals the sum of its incoming edges to six decimals, even though the transcoder is random and the errors are large. With the softmax and the norms live, the same sum comes to about , nowhere near the target. Exactness comes from the freezing, not from a good transcoder. A good transcoder only makes the error edges small.
A graph with a hundred features per token and millions of edges is not an explanation. We need to rank nodes by how much they matter to the output, through every path, and keep the few that matter most. Direct edges into the logit are not enough, since a feature two hops back can matter more than one that touches the logit directly. Ameisen et al. (2025) use a measure that adds up paths.
Take every edge's absolute value, so that a strong negative edge counts as strong. Then normalize each node's incoming edges to sum to 1. Call the result , indexed (target, source), so row says what fraction of 's input strength comes from each source. A path is a chain of edges, and its strength is the product of the entries along it. In the hand graph, 's inputs have absolute sizes 3, 1, and 1, so and .
Entry is a sum over sequences of edges from to , each term the product of its edge strengths. So it is the total strength of the paths of length , and summing over counts every path once.
A strictly lower triangular matrix has , since each multiplication pushes the nonzero entries one more diagonal below the main one, and the longest path has at most edges. Let be the smallest power with . Then
so is invertible with inverse (§0.10). Subtracting the term gives . ∎
The identity is the finite case of the Neumann series , which for a general matrix needs every eigenvalue to be smaller than 1 in absolute value. Here nilpotency, some power being exactly zero, makes it exact.
Influence also obeys a conservation law, and it is the key to the scores below.
Let and . Multiplying by gives . Sum both sides over all nodes .
Since is 1 for every node with inputs and 0 for an input node, . So . For an input node , because . ∎
In plain words, start at logit with probability and walk backward. At each node, pick an incoming edge with probability equal to its normalized strength. Every walk ends at an embedding or an error node. For an input node, is the chance the walk ends there. For any other node, it is the chance the walk passes through it, since a walk in an acyclic graph visits each node at most once.
Now we can say how much of the computation the features explain. The error nodes are the unexplained inputs, so we ask how much influence reaches the logits from them.
In the hand graph the replacement score is . Only takes input from the error node, so completeness is . The open-source circuit-tracer library computes completeness with the sum running over every node, counting each input node as fully explained. That gives a slightly higher number, here , so say which convention you use when you report one.
Implement error_nodes(y_true, y_tc). Both arrays have shape (L, T, d), the true MLP outputs and the transcoder reconstructions at each layer and position. Return (err, rel). Here err holds the error vectors, so that y_tc + err == y_true, and rel holds their relative sizes at each site, shape (L, T), with 0 where .
Then implement completeness(E, is_error, influence, logit_weights) exactly as defined above. E is the signed (target, source) edge matrix, is_error a boolean mask of error nodes, influence the , and logit_weights the . Average only over nodes that have incoming edges.
Pruning keeps the nodes that carry most of the influence. Sort all non-logit nodes by , inputs included. Keep the smallest prefix whose cumulative influence reaches a threshold, 0.8 of the total by default, and drop the features outside it. Embeddings and error nodes stay in the ranking, so they use up part of the budget, but they are never dropped, since they are the graph's inputs. Logit nodes are kept separately, the top tokens covering 95 percent of the probability, at most 10. (The open-source circuit-tracer library always keeps the embeddings and logits but may prune error nodes, so check which rule a tool uses.)
Ameisen et al. (2025) then recompute influence on the pruned graph and prune edges the same way at a higher threshold of 0.98, scoring each edge by its normalized weight times its target's influence. They report that this typically cuts the node count about tenfold, while completeness falls by only about 20 percent, and the edge count far more, about 500-fold, depending on the prompt's length.
Here are influence, pruning, and both scores on our two-hop graph.
So pruning trades completeness for legibility, and what it cuts is not always noise. When you read a pruned graph, ask what was cut and how much influence it carried.
The widget is this graph. Node size is logit influence, and edge width is the size of the edge. You can turn up the missing unit, prune, and click a feature to switch it off, halve it, or double it. The bars then compare the graph's linear prediction with the model rerun.
The next cell turns the missing unit up. The answer stays the same, but more and more of the TEXAS signal comes from a unit the transcoder lacks, so it reaches "say Austin" through the error node.
This is why graphs are always reported with their error nodes and their scores. The widget shows the same thing from the intervention side. Switch Texas off. At a missing unit of 0 the model's probability of Austin falls from 0.90 to 0.35, because "say Austin" drops below its threshold. The graph flags that crossing but cannot follow it. At 1.0 Austin barely moves, in the graph or in the model, because the missing unit keeps writing TEXAS.
Now press Swap Texas for California. At a missing unit of 0 the model answers Sacramento with probability 0.92. At 1.0 it still prefers Austin, 0.59 against 0.35, and at 2.0 Austin keeps 0.83, because the missing unit goes on writing TEXAS where no feature can be switched off. An intervention on features moves only the part of the computation that the features carry. The part on validating a graph, below, works through both interventions.
Implement logit_influence(E, logit_ids, probs). Take absolute values of the signed (target, source) edge matrix E. Normalize each node's incoming edges to sum to 1, leaving rows with no inputs at zero. Form and return for every node, where logit_ids are the logit nodes and probs their probabilities.
Then implement prune(influence, keep_frac, always_keep). It returns a boolean mask that keeps the smallest set of nodes, taken in order of decreasing influence, whose cumulative influence reaches at least keep_frac of the total, together with every node in always_keep (logits, embeddings, errors). As in the text, the ranking and the total run over all nodes, so an always_keep node with large influence uses up part of the budget. The tests compare your influence with a brute-force enumeration of every path.
Even pruned, a real graph has hundreds of features, and many do the same job on the prompt. Three features might fire on "Digital" in different contexts, and for an acronym prompt all that matters is that the word starts with D. Analysts group such features into a supernode and treat the group as one unit.
The authors are candid that this step is manual. They group by similar contexts, similar effects on the logits, or similar edges, depending on the claim being made. They aim for members that promote each other and push downstream nodes the same way. They tried automated methods, such as clustering the decoder vectors or the graph's edge matrix, and report that none covered the range of groupings their claims needed. Supernodes are where human judgment enters the method, and a reviewer should ask how they were chosen.
An attribution graph is a set of claims about the local replacement model, such as "feature A raises feature B, which raises logit X". The model we care about is the original one, and it may compute differently. So every claim is checked by perturbing the original model and seeing whether the effect matches.
The intervention is the feature clamping of §23.7. Set a feature's activation to a new value. You can suppress it to zero, multiply it by a factor (often ), or inject a feature that was not active. Replace the feature's decoded write in the MLP output by the new value's write, and run the original model forward, with its real MLPs, attention, and norms.
The graph makes its own prediction for that change. Push the change through the edges. Each active downstream feature passes its input change on at slope 1, the JumpReLU's slope above threshold. Read off the change of each logit. If the model moves as predicted, the graph's claim survives. If not, either the transcoder's mechanism is not the model's, or the change pushed some feature across its threshold, where no linear account can follow it.
The cell predicts four interventions with the graph and reruns the model for each. It flags any active feature that the change would push below its threshold. Then it performs a swap. It reads the California feature's activation off a prompt with "Oakland" in place of "Dallas", switches Texas off, and writes California in its place.
The three cases are the three kinds of validation evidence. Small, graded perturbations test the edges themselves. Large ones test the claimed causal role (does removing Texas remove Austin?), even when the size of the effect leaves the linear regime. A swap tests that the intermediate variable is what the graph says it is, since replacing it with another state changes the answer to that state's capital. The swap is the strongest of the three, and it is the test used on the frontier model below. An interviewer who hears "the graph shows X" should ask which of the three tests backed it.
The setup defines a one-position version of the two-hop toy and its graph G. G is a dictionary with the feature names in layer order followed by two logits, the clean activations f, preactivations h, thresholds theta, clean logits z, and the virtual weights W[t, s] between graph nodes (zero where there is no edge). run_model(changes) runs the underlying model, norms and thresholds live, with some features set to new activations.
Implement predict_intervention(G, changes). It propagates the changes linearly through the graph and returns (z_pred, flagged). z_pred is the predicted logits. flagged lists, in the order of G["names"], the active features (not themselves changed) whose predicted preactivation falls to or below their threshold. Then, in a comment, explain why no amount of linear propagation through the graph can predict the swap of Texas for California in the cell above.
How well does this hold up at scale? The next box collects the measurements. One term needs a definition first. A Spearman rank correlation is the ordinary correlation computed on ranks instead of values, so it is 1 when two lists put items in the same order, whatever the sizes.
A companion paper applied the method to Claude 3.5 Haiku (Lindsey et al. 2025). Each case below is a pruned graph plus interventions in the original model.
This is the real-scale version of our toy, with the same three kinds of evidence. It answers the opening question. The model does compute "Texas" on the way, and it also takes the shortcut. The other case studies follow, each in brief.
Planning in poems. Given the first line of a rhyming couplet, "He saw a carrot and had to grab it,", the model continues "His hunger was like a starving rabbit". The features that decide "rabbit" are active on the newline token before the second line begins. Features for the rhyme "it" activate candidate end words, "rabbit" and "habit". These then shape the whole line, including the words before the end (the "like a" is written toward the planned word).
Interventions matter only at that planning site, the newline, and not at the tokens around it. Suppressing "rabbit" makes the model write toward "habit", and injecting a "green" feature makes it write a line ending in "green". The authors found planned-word features in about half the poems they examined. Injecting "rabbit" or "green" into a random sample of 25 poems made the model end the line with the injected word in 70 percent of cases.
Shared multilingual features. Asked for the opposite of "small" in English, French, and Chinese, the model uses language-specific features at the input and the output. In the middle layers it uses largely shared, language-independent features. These shared circuits are more prominent in Haiku than in a smaller, less capable model. English is also privileged, in that the shared "say large" features often push English outputs more directly than outputs in other languages.
Addition in parallel. For "36+59=", one path works at low precision, roughly 36 plus roughly 60 giving a sum near 92. Another computes the ones digit exactly, a number ending in 6 plus one ending in 9 giving one ending in 5. The two combine to give 95. Asked how it got the answer, the model describes carrying the one, the algorithm taught in school, which is not what its features do. The model's account of its own computation is not a readout of that computation.
The addition graphs contain intermediate "lookup table" features, such as one for a number ending in 6 added to a number ending in 9. A small exact result about thresholded features, from the methods paper (Ameisen et al. 2025), explains why such a step is needed at all.
Proof. Additivity gives . The left side exceeds , so the two terms on the right cannot both be at most . ∎
So no single feature can fire on "sum is 5" for exactly the pairs that sum to 5. Firing on 1+4 and 2+3 forces it to fire on 1+3 or on 2+4. Computing a property of a sum therefore takes an intermediate step that combines the operands first, which is what the lookup-table features do.
Knowing what you know. The model has a default circuit, active on any Human and Assistant exchange, that pushes toward "I can't answer". Features for known entities, such as Michael Jordan, inhibit it, which lets the answer through. For an unknown name like "Michael Batkin" nothing inhibits it, and the model declines. Activating the known-answer features on the Batkin prompt made the model hallucinate a sport.
A natural misfire showed up when the model was asked for a paper by Andrej Karpathy. It recognized the name, so the known-entity features fired and switched the default off. It then named "ImageNet Classification with Deep Convolutional Neural Networks", a famous paper Karpathy did not write. Chapter 19 drew the lesson for hallucination in §19.8. Related entity-recognition directions were also found with sparse autoencoders, and they can steer a model to refuse questions about known entities or to invent facts about unknown ones (Ferrando et al. 2024).
Refusals and a jailbreak. A request for an ad about mixing bleach and ammonia is refused. Features for the danger of mixing cleaning chemicals feed a "harmful request" cluster, which feeds refusal features. The graphs suggest that general harmful-request features are built during fine-tuning from many specific harm features learned in pretraining.
In one partially effective jailbreak, the request is hidden in the initials of other words. The model assembles the hidden word letter by letter without ever representing the request as a request to make the thing. So the harmful-request features fire only weakly on the hidden word, and they turn on fully only once the model's own output has spelled out what is being made. The model then refuses only at the next sentence boundary, where features for starting a new sentence let it change course. §19.3 covers attacks at the level of threat models and defenses.
Faithful and unfaithful chains of thought. On a problem it can solve, the square root of 0.64, the graph shows the model doing what its chain of thought says, computing the square root of 64 on the way. Asked for the cosine of a large number it cannot compute, it claims to have used a calculator, and the graph shows no sign of a calculation. The authors call this bullshitting, in the philosopher Harry Frankfurt's sense of speech produced with no regard for whether it is true. Given a hint of the answer, the model works backward from the hint to choose the intermediate value that will lead there, which is motivated reasoning. Changing the hinted answer changes the chain of thought to match. This is the mechanistic side of the monitoring problem of §19.6.
A hidden goal. One model had been trained to exploit known biases of reward models (Marks et al. 2025). Its main reward-model-bias feature fired on all 100 Human and Assistant prompts tested and on 1 of 100 prompts without that format. The authors read this as the bias having been absorbed into the model's representation of the Assistant persona. The graphs show these features feeding the biased behaviors, and suppressing them inhibits those behaviors.
Because patterns are frozen, a graph shows what each head moved and never why it attended there. Sometimes the choice of where to attend is the interesting part of the computation, as two examples from Kamath et al. (2025) show. On "I always loved visiting Aunt Sally. Whenever I was feeling sad, Aunt", the graph says the model predicts "Sally" because "Sally" appeared earlier. That misses the induction mechanism of §4.7, a head that finds the previous occurrence of the current token and copies the token that followed it. On a multiple-choice question, the graph says the model answers "B" because of a feature on the token after "(B)". That skips how the model knew that 1945 was right.
QK attributions, from the same paper, fill part of the gap. Before the softmax, a head's score from query position to key position is bilinear in the two normalized streams, . Here is the head's QK circuit (§4.2) and is the key width. (With rotary position encodings the matrix depends on the distance between the two positions, but for a fixed pair it is still one matrix, and the split below still holds.)
With the norm scales frozen, each normalized stream is a sum of node contributions, one for each feature write, embedding, and error that reaches it. Write on the query side and on the key side. Then, exactly,
with one term for each pair of a query-side and a key-side node. In plain words, the score splits into a table of pairwise interactions, and the largest entries say which pair of features made the head look where it did.
The method confirmed the induction story, with query-side features for the current token meeting key-side "preceded by that token" features. It also found new mechanisms. Key-side features "tag" a word so that a later query asking for its opposite can find it. Multiple-choice heads attend to the option marked by "correct answer" features. Some heads check whether a statement agrees with the context. A second problem is saying which heads carried each edge of a graph. Between two transcoder features many layers apart, the number of paths through heads grows exponentially with the gap. So the authors also built graphs that record attributions at every layer, from features of the residual stream itself. That gives up exact linearity through the MLPs, but each edge then passes through only one layer of heads.
In May 2025 Anthropic released the circuit-tracing code as an open-source library, with an interactive graph explorer on Neuronpedia. It came with demonstrations of multi-step reasoning and multilingual features in two small open models, Gemma-2-2b and Llama-3.2-1b. Anyone can now build, annotate, and test attribution graphs on open models, including the interventions described above.
The more consequential change is that interpretability tools are now used in audits, investigations of whether a model has undesired objectives before it ships. Three lines of work made that possible.
Auditing games. An auditing game trains a model with a planted problem and asks auditors to find it. In one study, the planted problem was a hidden objective, exploiting known biases of the reward models used in training (Marks et al. 2025). Four teams investigated it without being told the objective. The three teams given the model's weights and training data uncovered it, using sparse autoencoders, behavioral attacks, and analysis of the training data. The fourth, with only query access to the model, did not. A game like this measures audit methods, because the team that built the model knows the right answer.
Auditing agents. Follow-up work gave a language-model agent the auditors' tools, a chat interface to the model, data analysis, and interpretability tools (Bricken et al. 2025). §24.1 explains how such agents run. One investigator agent solved the same game 10 to 13 percent of the time under realistic conditions. It tended to fixate on its early hypotheses. A "super-agent" that read many agents' reports and investigated further raised the solve rate to 42 percent.
Reading activations in words. Two lines of work skip features and train a language model to describe activations directly. Activation oracles are language models trained with supervised data to answer arbitrary natural-language questions about another model's activations, which are fed in as extra inputs (Karvonen et al. 2025). They matched or beat the best earlier method on three of four auditing tasks, such as uncovering secret knowledge or misalignment introduced by fine-tuning. They did so without ever training on activations of the fine-tuned models they audited.
Natural language autoencoders pair two models, both initialized as copies of the target model (Fraser-Taliente, Kantamneni, Ong et al. 2026). An activation verbalizer maps an activation to text, and an activation reconstructor maps the text back to an activation. They are trained jointly with reinforcement learning to reconstruct residual-stream activations through the text. Nothing in the objective asks for faithful or readable explanations, yet by the authors' measures the text grows more informative over training.
Training models to be readable. Another approach changes the model instead of the microscope. In Transformers whose weights are almost all zero, about 1 in 1,000 nonzero in the sparsest models, task circuits are roughly 16 times smaller than in dense models of comparable pretraining loss (Gao et al. 2025). Their neurons and residual channels often match natural concepts. Sparsity trades capability for interpretability, and scaling such models beyond tens of millions of nonzero parameters while keeping them interpretable is open.
Parameter decomposition splits the weights rather than the activations. It writes the weights as a sum of components chosen to reproduce the network, to be few per input, and to be simple. It was introduced on toy models as attribution-based parameter decomposition (Braun et al. 2025), and a stochastic version made it cheaper and less sensitive to hyperparameters (Bushnaq et al. 2025).
Benchmarks. MIB, a mechanistic interpretability benchmark, scores methods on how precisely and concisely they recover the causal pathways or causal variables of four tasks in five models (Mueller et al. 2025). Attribution and mask-optimization methods did best at locating circuits. For finding causal variables, a supervised method did best. It is called distributed alignment search, and it searches for a subspace of the activations that behaves like a known variable under interventions. SAE features did no better than individual neurons there, which echoes the mixed evidence on SAEs in §23.6. Multimodal models raise the same questions with image features in the stream, which the CLIP chapter meets as multimodal neurons and typographic attacks.
The table compares the approaches of the chapter by what each gives you and how each fails.
| approach | what you get | needs | main failure mode |
|---|---|---|---|
| probes (§23.2) | whether a variable is decodable, and a direction | labels for the variable | decodable is not used |
| activation and path patching (§23.4) | causal effect of components | contrastive prompts | off-distribution patches, heads only |
| SAE or transcoder features (§23.6) | an unsupervised vocabulary of directions | a large training run | dark matter, splitting and absorption |
| attribution graphs | a prompt-specific mechanism through features | a CLT, a backward pass per node | error nodes, frozen attention, about a quarter of prompts |
| activation oracles, NLAs | readable hypotheses about activations | a second model trained on the first | confabulation, no mechanism |
| weight-sparse models | small, legible circuits by construction | training a new model | capability cost, unclear scaling |
| parameter decomposition | mechanisms as components of the weights | heavy optimization | toy-scale so far |
The field's own map of what is unsolved is a long review written by many of its researchers (Sharkey et al. 2025). Each problem below has already shown up in this section, and a strong interview answer names the problem, the evidence for it, and why it is hard.