How to read what a trained network computes, from the outside and from the inside. Attributions, probes, lenses, causal patching, circuits, superposition, sparse dictionaries, steering, and attribution graphs, each derived, implemented, and tested against what it can and cannot prove.

This chapter opens the model up. Chapter 22 judged a model by its outputs alone, treating it as a black box (§22.1). That has a limit. A behavioral test sees only the inputs someone thought to try, so it cannot show that a behavior is absent or say why a model succeeds. This chapter reads the computation itself, in the Transformer the book has built. Its residual stream is a running sum that every attention head and MLP writes into (§5.1), its heads move information between positions (§4.5), and its output layer turns the final vector into next-token scores (§6.2). Each section recaps the piece it needs.

Interpretability is the study of how a trained network computes its outputs. We can read every weight of a network, run it on any input, and record every number it computes along the way, and still not know how it reaches its answer. A trained network is a program that nobody wrote. This section explains why it is worth knowing how that program works, what would count as an explanation, and how far we can get from the outside, by asking which parts of the input an output depended on. It ends where the outside view stops working, which is where the rest of the chapter begins.

Here is the problem in its sharpest form. Two teams train image classifiers to spot horses. Both score 95 percent on a test set drawn the same way as the training data. One model looks at the animal. The other looks at a small copyright tag in the corner of the photo, which happens to sit on many of the horse pictures. This is close to a real case. Lapuschkin et al. (2019) found it in a classifier trained on PASCAL VOC 2007, a standard benchmark of labeled photos. A source tag sat on about one fifth of its horse images, and the classifier had learned to look for it. Pasting the tag onto a photo of a car made it call the car a horse.

No test set drawn from the training distribution can tell these two models apart, because on that distribution they are equally accurate. Two other kinds of evidence can. You can collect new photos in which the tag and the horse come apart, if you know to look for the tag. Or you can look at how each model computes its answer and see which one reads the tag. This chapter is about the second kind of evidence.

The section runs in this order. We first build the horse problem in two dimensions, small enough to check every number. Then we say what interpretability is for, what counts as an explanation, and how strong different kinds of evidence are. Then come the tools that work from the outside, called attributions, which split the credit for one output among the inputs. We build three of them (gradients, integrated gradients, and Shapley values), prove what each guarantees, and show how each fails. We end with the tests that catch attributions that ignore the model, and with why even a perfect attribution cannot say what the model computed.

Two models that no test set can tell apart

Each example in our toy problem has a label $y \in \{0, 1\}$, which we also write as a sign $s = 2y - 1 \in \{-1, +1\}$. It has two features.

• Feature $x_1$ is the cause, with noise, $x_1 = 1.645\,s + \epsilon$ with $\epsilon \sim \mathcal{N}(0, 1)$. Guessing the label from the sign of $x_1$ is right with probability $\Phi(1.645) = 0.95$. Here $\Phi$ is the standard normal CDF, the probability that a standard normal variable falls below its argument.
• Feature $x_2$ is the tag. It equals $s$ with probability $\rho$ and $-s$ otherwise, plus a little noise, so $x_2 = \pm s + 0.3\,\epsilon'$. In training the tag agrees with the label 95 percent of the time, $\rho = 0.95$.

We fit three logistic regressions. A logistic regression scores an input with a weighted sum $\mathbf{w}\cdot\mathbf{x} + b$, called the logit, and predicts class 1 when the logit is positive. It is trained by minimizing the cross-entropy loss (§0.2). Model A sees only $x_1$, model B only $x_2$, and model C sees both. We score each one twice. The first test set is fresh data from the same distribution. The second is shifted, so that the tag now points the wrong way, $\rho = 0.05$.

Here the mechanism is a weight vector, and reading it settles the question. Model A puts weight 0 on the tag and model B puts weight 0 on the cause, by construction. Model C puts 2.63 on the tag next to 3.18 on the cause. Nothing about C's test score warned us, but its weights did.

The rest of the chapter is about getting that knowledge when the mechanism is not two numbers but billions. In the widget, you set how often the tag agrees with the label in training and at test, and watch the three models. Its table also shows how much of each model's credit goes to the tag $x_2$ on data like training, measured with integrated gradients, a method this section builds later.

The horse classifier and model C are cases of shortcut learning. Geirhos et al. (2020) gave that name to decision rules that do well on the standard test set and fail on harder or real-world inputs, and their survey collects many cases. A classifier recognizes cows by the grass around them and fails on a cow at the beach. A pneumonia detector trained on chest X-rays from several hospitals learned to tell which hospital a scan came from, partly from a metal token that technicians place on the film (Zech et al. 2018). It then used each hospital's pneumonia rate as a shortcut, and did worse on hospitals it had not seen.

In practice, this changes how you select and ship models. Held-out accuracy can prefer the model with the shortcut, as it preferred C. An interviewer will ask how you would find a spurious feature without data from the shifted distribution. One good answer is to test behavior under changes that should not matter. Another is to measure what the model's output depends on. This section builds both.

What interpretability is for

The word interpretability means little until you say what you want from it, because papers ask it for many different things (Lipton 2016). This book uses it for three purposes, and each has a published case.

Debugging. A model can be right for the wrong reason, and the reason predicts where it will fail. In the same paper as the horse tag, Lapuschkin et al. studied a network trained by reinforcement learning to play the Atari game Pinball. It had learned to nudge the table, gently enough never to trigger the game's tilt lockout, so that the ball kept passing over a scoring switch. That is an exploit of the game's physics, not good play. It is a case of specification gaming, behavior that satisfies the letter of an objective and misses its intent (§19.1). The same section describes goal misgeneralization, in which a model stays capable on new inputs while pursuing a goal other than the intended one. You can see either failure coming only if you can see what the model is keying on.

Safety. A behavioral evaluation can show that a model does something. It can never show that a model never does it. An evaluation measures what the model does on the prompts someone thought to try, so it is a lower bound on what the model can do (§14.1). A backdoor is a hidden behavior that fires only when a specific trigger appears in the input. A model with a backdoor, or one that behaves differently when it suspects a test, passes every test that misses the trigger. Looking inside can catch what tests miss.

• §19.6 describes models trained with deliberate backdoors that survived safety training. A linear probe scores the model's internal vector by its dot product with a fixed direction. One probe, whose direction came from a single pair of generic texts unrelated to the backdoor, separated prompts on which the model would defect from the rest with an AUROC of 99.3 percent. The AUROC is the probability that the probe scores a random defecting case above a random normal one, so 50 percent is chance (§0.11).
• In a blind auditing game, Marks et al. (2025) trained a model with a hidden objective and gave it to four teams. The three teams given the weights and the training data found the objective. The team that could only send prompts and read answers through an API, the access an outside auditor would have, did not.

Science. A network that solves a problem has found some algorithm for it, and reading that algorithm can teach us something. Small networks trained on algorithmic tasks sometimes memorize their training set first and generalize only much later, a pattern Power et al. (2022) called grokking (§9.6). Nanda et al. (2023) reverse-engineered a small Transformer that had grokked addition modulo a prime. It represents each number as an angle on a circle, at a few frequencies, and adds numbers by adding angles. Knowing the algorithm let them build measures that track hidden progress long before the model generalizes, and §23.4 takes that algorithm apart.

All three uses need an explanation of how the model computes its output, sharp enough to make a prediction. Here is the definition the chapter works with. It uses the word activations, which means the vectors of numbers a network computes at its internal layers on a given input.

In plain words, an explanation is a hypothesis about the model's insides that you could test by surgery. "The model reads the tag" predicts that covering the tag changes the answer, and it does. A story that predicts nothing you could check is not yet an explanation.

Levels of explanation

A system that processes information can be described at three levels. The top level says what function it computes. The middle level says which variables it keeps and which steps transform them. The bottom level says how those steps are physically built. An answer at one level does not settle the others. The neuroscientist David Marr (1982) made this argument for vision, and his levels map onto a trained network like this.

In a Transformer, the attention heads move information between positions (§4.5), and the MLPs are the two-layer networks that transform each position on its own (§5.3). Two models can agree at the top level and differ completely below it, as models A and B did on the training distribution.

Behavioral interpretability stays in the top row and treats the model as a black box. Mechanistic interpretability tries to reverse-engineer the bottom two rows, the way one might turn a compiled binary back into source code. Olah et al. (2020) set out three working claims for that program, and the rest of this chapter tests them. Networks are built from meaningful features, internal variables that each stand for one thing. Features connected by weights form circuits. And similar features and circuits recur across different models, a claim they called universality.

The ladder of evidence

Methods also differ in how strong a conclusion they allow. The chapter climbs a ladder of four rungs, and every later section says which rung it stands on.

Input attributions sit on the first rung, and the reason is worth stating carefully. Some of them do intervene, but on the input. Covering part of an image and watching the score drop is a causal statement about which inputs matter. As evidence about the internal computation it is only association, because many different internal mechanisms produce the same attributions, as the end of this section shows. When you read an interpretability result, ask which rung its evidence reaches.

Behavioral tests come first

Before opening anything, test behavior systematically. Ribeiro et al. (2020) adapted software testing to language models, in a method they called CheckList, with three kinds of test.

• A minimum functionality test checks simple cases the model must get right. "I didn't love it" is not positive.
• An invariance test makes a change that should not matter, such as swapping one city name for another, and checks that the prediction stays put.
• A directional expectation test makes a change that should move the prediction one way, such as adding "the service was awful", and checks that it does.

With these tests, the team that owned a heavily tested commercial sentiment model found new bugs it could act on. §22.6 uses the same three test types to build a product's eval suite. Behavioral tests are cheap, and they find real failures. An invariance test is the cheapest way to check for a shortcut you already suspect. What they cannot do is find a shortcut nobody thought of, or prove that one is absent.

Faithful, not merely plausible

An explanation can fail in a way that is easy to miss. It can look right to a person and still not describe what the model did.

A plausible but unfaithful explanation is worse than none, because it creates confidence. A model trained on human-written explanations that writes a fluent justification after the fact is plausible by construction, and nothing guarantees that it is faithful. Attention weights were long read as explanations because they look like one. §3.6 explains why they are not, on their own. Quite different attention patterns can give the same prediction, and information mixes across positions at every layer.

So how do you test faithfulness, if not by looking? Jacovi and Goldberg collected the tests in the literature and the assumptions behind them, and three tests carry most of the weight in practice.

1. Same predictions, same explanation. If two models make the same predictions everywhere but a method explains them differently, the method is unfaithful to at least one of them. Jacovi and Goldberg call the assumption behind this test the model assumption.
2. Destroyed model, different explanation. A model whose weights have been randomized must not get the same explanation as the trained one. The model-randomization test later in this section runs this check.
3. Intervention. Change what the explanation says matters, and check that the output moves as the explanation predicts. This is the strongest test, and it follows straight from the definition of an interpretability claim. Every section of this chapter ends up using it.

In an interview, "how would you know your explanation is right?" has the third test as its answer.

Attributing an output to its inputs

The simplest question about one prediction is which inputs it depended on. An attribution answers with one number per input, the credit that input gets for the output. Credit has to be measured against something. So we compare the input with a baseline, a reference input that stands for "the information removed".

Here is a tiny case first. Take a linear model $f(\mathbf{x}) = 2x_1 - x_2 + 0.5$, the input $\mathbf{x} = (3, 1)$, and the baseline $\mathbf{x}' = (0, 0)$. The output goes from $f(\mathbf{x}') = 0.5$ to $f(\mathbf{x}) = 5.5$, a change of 5. The natural split gives $x_1$ the amount $2\times 3 = 6$ and $x_2$ the amount $-1\times 1 = -1$. These add up to the change, 5. For a nonlinear model the split is no longer obvious, and the methods below are different answers to how to make it.

In general we explain a scalar output $f : \R^n \to \R$. For a classifier we pick the logit of the predicted class, not its softmax probability (the pitfall box later in this section says why). We fix the input $\mathbf{x} \in \R^n$ and a baseline $\mathbf{x}' \in \R^n$, such as a black image, the all-zero embedding, or the training mean.

Three methods come first, because they are cheap and nearly everyone uses them.

The gradient. The saliency map of Simonyan et al. (2013) is the gradient $\nabla f(\mathbf{x})$, how fast the output changes as each input moves. Backpropagation computes all $n$ entries in one backward pass, which costs about as much as two forward passes, whatever $n$ is (§0.3). The gradient answers "what if input $i$ moved a little?" That is a different question from "how much did input $i$ contribute?"

Gradient times input. The first-order Taylor expansion of $f$ around $\mathbf{x}$ (§0.2) says $f(\mathbf{x}') \approx f(\mathbf{x}) + \nabla f(\mathbf{x})\cdot(\mathbf{x}' - \mathbf{x})$. Rearranged,

In plain words, each input's share of the change is its slope times how far it moved. The $i$-th term, $\partial_i f(\mathbf{x})\,(x_i - x'_i)$, is the attribution. With the baseline $\mathbf{x}' = \mathbf{0}$ it is the familiar "gradient times input". It is exact when $f$ is linear, as in the tiny case above, and only approximate otherwise.

Occlusion. Zeiler and Fergus (2014) slid a gray square over an image and recorded how much the class score dropped. For one input at a time, $a_i = f(\mathbf{x}) - f(\mathbf{x}_{[i \leftarrow x'_i]})$, where $\mathbf{x}_{[i \leftarrow x'_i]}$ is $\mathbf{x}$ with entry $i$ replaced by its baseline value. It costs $n + 1$ forward passes. It asks a real counterfactual question, but only about one input at a time.

Gradients go blind when a unit saturates

This example, from Sundararajan et al. (2017), is one that every interviewer in this area knows.

In plain words, the gradient is local. It reports the slope where you stand. A unit has saturated when its output has stopped responding to its input. Then the slope is zero however much that input mattered on the way there. ReLUs past their kink, sigmoid and tanh units near their flat tails, and softmax probabilities near 1 all saturate, and trained networks are full of them.

The last line is the fix, and it is worth doing by hand. Instead of the slope at $x$, average the slope over the whole straight path from the baseline to the input. On the path from 0 to 2 the slope is 1 for the first half and 0 for the second, so the average is $\tfrac12$. Then $\tfrac12\times(2 - 0) = 1$, exactly the change in the output. That average is the next method, integrated gradients.

import numpy as np

def grad_times_input(f_grad, x, x0):
    raise NotImplementedError

def occlusion(f, x, x0):
    raise NotImplementedError

Integrated gradients

Sundararajan et al. (2017) turned the hand calculation above into a method and, more importantly, into a theory. Attribution methods had mostly been judged by how good their pictures looked, which is plausibility. They wrote down properties an attribution should have, and found a method that has them.

In plain words, walk from the baseline to the input in a straight line. Average the sensitivity to input $i$ along the way, and multiply by how far input $i$ moved. Gradient times input uses the sensitivity at the end of the walk only. Integrated gradients uses all of it. On the saturated ReLU it gives $2\times\tfrac12 = 1$, the right answer.

The reason it gets the right answer is a theorem. The attributions always add up to the change in the output.

Proof. The function $g$ has one variable. By the chain rule (§0.2), wherever $f$ is differentiable at $\boldsymbol\gamma(\alpha)$,

since $\boldsymbol\gamma'(\alpha) = \mathbf{x} - \mathbf{x}'$. Integrate both sides over $\alpha$ from 0 to 1, and swap the finite sum with the integral. The right side becomes $\sum_i \operatorname{IG}_i(\mathbf{x})$. The left side is $\int_0^1 g'(\alpha)\,d\alpha = g(1) - g(0) = f(\mathbf{x}) - f(\mathbf{x}')$, by the fundamental theorem of calculus. That theorem applies on each of the finitely many pieces between kinks, where $g$ is differentiable with a bounded derivative. The pieces add up because $g$ is continuous at the kinks. (For a ReLU network, $g$ is piecewise linear along the segment, so the integral is a sum of finitely many slopes times lengths.) ∎

This is the gradient theorem for line integrals, the multivariable form of the fundamental theorem of calculus. It buys the property the gradient lacked, which Sundararajan et al. call Sensitivity(a). Suppose $\mathbf{x}$ and $\mathbf{x}'$ differ in one input only and $f(\mathbf{x}) \ne f(\mathbf{x}')$. Every other input gets zero, because its factor $x_j - x'_j$ is zero. So completeness forces the one input that moved to get the whole difference, which is not zero. The saturated ReLU is that situation with $n = 1$.

The widget draws the function along the path, with its derivative below. Integrated gradients is the shaded area under the derivative between $x'$ and $x$, which equals the distance moved times the average derivative. The dashed box is gradient times input, the derivative at $x$ times the distance moved. The widget estimates the area with $K$ rectangles, each sampled at its midpoint, and its bottom panel shows how far the estimate is from the true change with midpoints and with left ends. Press Walk the path to watch the area pile up as a dot moves from $x'$ to $x$, and stop growing wherever $f$ is flat.

The axioms

Completeness is one of six properties Sundararajan et al. asked for. Here they all are, for an attribution method $\mathbf{a}$. Interviewers ask for them by name.

1. Completeness. $\sum_i a_i = f(\mathbf{x}) - f(\mathbf{x}')$.
2. Sensitivity(a). If $\mathbf{x}$ and $\mathbf{x}'$ differ in one input and the outputs differ, that input gets nonzero credit.
3. Sensitivity(b), called dummy in the economics of cost sharing. If $f$ does not depend on input $i$ at all, $a_i = 0$.
4. Implementation invariance. Two networks that compute the same function get the same attributions, however differently they are built.
5. Linearity. The attributions for $c_1 f_1 + c_2 f_2$ are $c_1$ times those for $f_1$ plus $c_2$ times those for $f_2$.
6. Symmetry preservation. If swapping inputs $i$ and $j$ leaves $f$ unchanged, and $\mathbf{x}$ and $\mathbf{x}'$ each have equal values in positions $i$ and $j$, then $a_i = a_j$.

Integrated gradients has all six. The plain gradient has implementation invariance, because the chain rule gives the same derivative whatever intermediate variables a network uses. But it fails Sensitivity(a), as the ReLU showed.

Computing it, and checking that you did

The integral is estimated with a Riemann sum, an average of the integrand at $K$ points on the path. With the midpoint rule the points are $\alpha_k = (k + \tfrac12)/K$ for $k = 0, \dots, K-1$, and

The left rule uses $\alpha_k = k/K$ and the right rule uses $(k+1)/K$, which is the form Sundararajan et al. wrote down. All $K$ gradients come from one batched backward pass, since the points on the path are independent inputs.

Completeness now doubles as a convergence check. The completeness gap is $\sum_i \widehat{\operatorname{IG}}_i - \big(f(\mathbf{x}) - f(\mathbf{x}')\big)$, where $\widehat{\operatorname{IG}}$ is the Riemann-sum estimate. It is zero for the exact integral, so whatever is left is discretization error. Sundararajan et al. found that 20 to 300 steps brought the sum within 5 percent of the difference, and recommended checking this every time. How fast the gap shrinks depends on the rule.

Here is a small network where the difference between the methods is not subtle.

The code writes vectors as rows, as the rest of the book does. So a layer computes $\mathbf{x}\mathbf{W}$, and $\mathbf{W}_1 \in \R^{2\times 2}$ maps inputs to hidden units. Now for the convergence rates. The next cell builds a random tanh network with eight inputs and 32 hidden units, and explains one input against the zero baseline with both rules.

Each fourfold increase in $K$ cuts the left-rule gap by a factor of about 4, from $9.4\times 10^{-3}$ at $K = 16$ to $2.4\times 10^{-3}$ at 64. It cuts the midpoint gap by about 16, from $9.2\times 10^{-5}$ to $5.8\times 10^{-6}$, as the fact box predicts. At $K = 64$ the midpoint rule is about 400 times more accurate than the left rule, for the same 64 gradient evaluations. In practice, use midpoints and always report the gap.

import numpy as np

def integrated_gradients(f, grad_f, x, baseline, steps=64, rule="midpoint"):
    raise NotImplementedError

Choosing the baseline

Every attribution in this section is relative to a baseline, and changing the baseline changes the answer. Sundararajan et al. recommend a baseline whose output is near zero and that carries no signal, so that the attributions describe the input rather than the baseline.

• For images it is often a black image.
• For text models it is often the all-zero embedding vector. That is not a real input, but it works because unimportant tokens tend to have small embeddings.
• Other common choices are a blurred copy of the input, uniform noise, and the training mean. Expected gradients (Erion et al. 2021) averages integrated gradients over many baselines drawn from the training data.

None of these is neutral. A black baseline cannot credit black pixels for anything, since they do not move along the path. A dark object on a dark background gets no attribution however much it mattered. The baseline is part of the question being asked, "compared with what?" Choose it on purpose and report it.

Shapley values

Integrated gradients needs gradients, and a smooth way to move an input partway. Many inputs are not like that. A word is in a sentence or it is not, and a feature in a table is observed or missing. For these the natural tool comes from cooperative game theory. Lloyd Shapley (1953) asked how to split a team's payoff fairly among its players, and his answer is the Shapley value.

Here is the difficulty in one example. Take $f = x_1x_2$ at $\mathbf{x} = (2, 3)$ with baseline $(0, 0)$, so the output goes from 0 to 6. Bring in $x_1$ first, with $x_2$ still at its baseline, and the output stays 0. Bring in $x_1$ after $x_2$, and the output jumps from 0 to 6. What $x_1$ contributed depends on who was already there. Shapley's answer is to average over every order of arrival.

To say this in symbols, treat the $n$ inputs as players $N = \{1, \dots, n\}$. A coalition is a subset $\mathcal{S} \subseteq N$. A value function $v$ assigns a number $v(\mathcal{S})$ to every coalition, the output when only the inputs in $\mathcal{S}$ are present, with $v(\varnothing)$ the output when none of them are. When player $i$ joins coalition $\mathcal{S}$, its marginal contribution is $v(\mathcal{S}\cup\{i\}) - v(\mathcal{S})$.

In plain words, assemble the team in a random order and credit each player with what the team gained when it arrived. Then average over orders, so that nobody is favored by arriving early or late. For $f = x_1x_2$ above, $x_2$ arrives before $x_1$ in half the orders, so $\phi_1 = \tfrac12\times 6 = 3$, and likewise $\phi_2 = 3$.

When the game is built from a model with a single baseline, as below, efficiency is completeness under another name, and dummy is Sensitivity(b).

Efficiency has a short reason. In any one order of arrival, the marginal contributions telescope to $v(N) - v(\varnothing)$, so their average over orders does too. The derivation below proves all four properties and shows that no other method has them.

From a model to a game

To use any of this, a model and an input have to be turned into a game. That is a modeling choice with real consequences, and there are three common choices.

• The single-baseline game. A coalition $\mathcal{S}$ keeps its inputs at their values in $\mathbf{x}$ and sets the rest to the baseline, $v(\mathcal{S}) = f(\mathbf{x}_\mathcal{S}, \mathbf{x}'_{\bar{\mathcal{S}}})$, where $\bar{\mathcal{S}}$ is the complement of $\mathcal{S}$.
• The conditional game. Lundberg and Lee (2017) defined SHAP values as the Shapley values of $v(\mathcal{S}) = \mathbb{E}[f(\mathbf{X}) \mid \mathbf{X}_\mathcal{S} = \mathbf{x}_\mathcal{S}]$. This averages the model over plausible values of the missing inputs, given the present ones.
• The interventional game. The conditional expectation is hard to estimate. So in practice Lundberg and Lee, and most libraries, assume the inputs are independent and use $v(\mathcal{S}) = \mathbb{E}_{\mathbf{X}'}[f(\mathbf{x}_\mathcal{S}, \mathbf{X}'_{\bar{\mathcal{S}}})]$, with $\mathbf{X}'$ drawn from a background dataset. This is the single-baseline game averaged over many baselines.

The choice changes the answer. Under the conditional game, an input the model ignores can receive credit because it is correlated with one the model uses, since knowing it changes the expected output. Under the interventional game it cannot, because then the dummy axiom applies to the model itself. Sundararajan and Najmi (2020) showed that these and other versions give very different attributions. That takes much of the force out of the uniqueness theorem. Shapley values are the unique fair split of a given game, and the game is yours to choose. An interviewer who asks "what does SHAP assume?" wants this answer.

On one kind of model, every method in this section gives the same answer.

In plain words, when the model has no interactions and no saturation, there is only one sensible answer, and every method finds it. Everything interesting about attribution happens in the nonlinear parts. This is also why the widget near the start of this section can compute the credit for its three logistic models as each weight times the input's distance from the training mean.

The cell checks this by enumerating every coalition.

The two methods agree on this function, but they are different methods. Sundararajan et al. gave the example that separates them.

The cost, and how to pay less

Exact Shapley values need $v$ on all $2^n$ coalitions. Each one is a model evaluation, or for the interventional game an average of several. For $n = 20$ features that is $2^{20} \approx 10^6$ evaluations per explained input. For the 50,176 pixels of a 224 by 224 image, or the tokens of a long prompt, it is out of the question. Two estimators make it practical.

Permutation sampling. Draw $M$ random orders and average each player's marginal contribution over them. Each order costs $n + 1$ evaluations and gives an unbiased estimate of every $\phi_i$ at once. By the central limit theorem the error falls like $1/\sqrt{M}$. The cell compares this with the exact values on a quadratic model of 10 inputs, where the exact computation needs $2^{10} = 1{,}024$ coalitions. It encodes coalitions as bitmasks, integers whose bit $j$ is set when input $j$ is present.

The root-mean-square error over 40 repetitions goes from 0.066 to 0.033 to 0.016. It halves with each fourfold increase in $M$, the $1/\sqrt{M}$ rate. (The table of all coalition values is a shortcut for this demonstration. A real sampler evaluates the model only on the coalitions its orders visit, $M(n + 1)$ of them.)

Regression. A second estimator fits a weighted linear model to the outputs on sampled coalitions, and with the right weights its coefficients are the Shapley values. Lundberg and Lee found it more sample-efficient than permutation sampling, because one regression estimates all $n$ values jointly. The box below gives the weights.

import numpy as np
from math import comb

def shapley(v, n):
    raise NotImplementedError

Back to the shortcut, with a real network

The opening example was too easy, because a linear model shows its mechanism in its weights. A neural network does not. Its first-layer weights can be large for an input the network ignores, if the units reading that input sit saturated on every example and so pass on a constant. They can also be small for an input it relies on. This is where attributions earn their keep. In the next exercise you get two small tanh networks with equal accuracy on the training distribution, and one of them secretly reads the tag. You may not use shifted data.

import numpy as np

def uses_shortcut(model, X):
    # placeholder that looks at the inputs, not at the model
    return bool(np.abs(X[:, 1]).mean() > np.abs(X[:, 0]).mean())

The held-out test is the point. The model you flag from in-distribution attributions alone is the one that falls apart when the tag flips, at about 10 percent accuracy against 92 percent for the other. The exercise checks that fact only after you have answered. In practice this is the workflow for a suspected shortcut when no shifted data exists. Attribute on the data you have, and see which inputs carry the decision.

Do attributions explain the model?

Integrated gradients is complete, and its axioms are reassuring. But they are properties of a formula, not evidence that a map tells you about the model. In vision, many saliency methods were judged by whether their maps looked like the object, which is plausibility again. Adebayo et al. (2018) designed two tests that check whether a map depends on the thing it claims to explain.

• The model-parameter randomization test. Replace the trained network's weights with random values, either from the top layer down, one layer after another (cascading), or one layer at a time (independent). Recompute the attributions after each step and measure how similar they are to the original maps. A method that explains the model must change when the model is destroyed.
• The data randomization test. Train the same architecture on randomly permuted labels, so that it memorizes noise. Compare its maps with those of the model trained on real labels. A method that explains what the model learned must differ between the two.

How you compare maps matters too. Adebayo et al. used the Spearman rank correlation, the ordinary correlation coefficient computed on the ranks of the values rather than on the values themselves. It asks whether two maps order the inputs the same way. They reported it on absolute values and on signed values, because the sign of a map flips easily under randomization while its ranking by magnitude may not.

The cell runs the cascading test on a small network trained on 16 inputs whose scales differ by a factor of 15, a stand-in for the bright and dark regions of an image. Only inputs 3, 7, and 11 determine the label. We randomize the top layer, then the top two, then all three, and compare each method's map with its original. The third method returns the input itself and uses no model, so it plays the part of the edge detector.

Passing this test is necessary, not sufficient. A method can depend on the weights and still describe them badly. In practice, run the randomization test before you trust any attribution method on a new model. An interviewer will ask what the test can rule out (methods that ignore the model) and what it cannot (methods that depend on the model but misdescribe it).

import numpy as np

def randomization_test(attr_fn, layers, X, reinit):
    raise NotImplementedError

Attributions at real scale

Integrated gradients and Shapley values are widely used in practice. Their best-known applications show both what they do well and where they stop.

A Transformer's input is a sequence of token IDs. These are discrete, so its attributions are computed with respect to the input embeddings instead, the vectors each token is mapped to before the first layer. Each token's attribution is the sum of the attributions of its embedding coordinates. The baseline is the zero embedding or the embedding of a padding token, and completeness holds for the whole sequence.

Here is that computation in PyTorch, for a tiny model that averages its token embeddings and scores the average with an MLP. The whole path goes through the model as one batch, so one backward pass returns all $K$ gradients. Summing the outputs before calling backward is valid, because each path point's output depends only on its own input. (The model has no batch statistics.)

In production code you would use a library. Captum, PyTorch's attribution library (Kokhlikyan et al. 2020), provides IntegratedGradients for inputs and LayerIntegratedGradients for an internal layer such as the embedding. Its attribute method takes the baseline, the target class, the number of steps (default 50), the quadrature rule, and a flag that returns the completeness gap. Its default rule is Gauss-Legendre quadrature, which places the points unevenly and converges faster than the midpoint rule on smooth integrands.

For models with special structure, Shapley values can be exact and fast. For tree ensembles, an algorithm that walks the trees computes them in polynomial time (Lundberg et al. 2020). For linear models they have the closed form above.

One more method points toward the next section. Kim et al. (2018) asked not which pixels mattered but whether a human concept, such as "striped", mattered. Their method, called TCAV (testing with concept activation vectors), works in three steps.

1. Collect examples of the concept and random counterexamples, and record a layer's activations on both.
2. Train a linear classifier on those activations to separate them. The vector perpendicular to its decision boundary is the concept's direction in that layer.
3. The concept's importance for a class is the fraction of that class's inputs whose logit increases when the activation moves along that direction, the sign of a directional derivative.

The attribution is now to an internal direction rather than to an input. That only makes sense if the network represents the concept as a direction. Whether it does is the subject of §23.2.

The table collects the methods of this section. It is a good page to have in your head for an interview.

Where the outside view stops

Suppose an attribution method passed every test above. It would still answer only the question it was built for, which inputs this output depended on. It would not say what the model computed with them. That is a different question, and here is the shortest demonstration.

An attribution at one point is a summary of a function, and many functions share a summary. Inside a network the gap is wider still, since the same function can be computed by many different arrangements of weights. Three further limits follow from the same fact.

• An attribution explains one output on one input. The questions at the start of this section are about the computation across inputs. Will it fail when the tag flips? Does it have a backdoor? What algorithm did it learn?
• Attributions are taken with respect to inputs, so they cannot name the intermediate variables a model computes, such as "this token is a repeated name" or "the subject of the sentence is a city in France".
• They inherit the model's behavior off the training distribution, through baselines, paths, and occlusions.

So we have to look inside. The obvious first move is to record the activations of individual neurons, the single coordinates of a layer's output, and ask what each one responds to by finding the inputs that activate it most. Bolukbasi et al. (2021) showed how that goes wrong. They took neuron 221 in the last layer of BERT, a Transformer text encoder, and found its top-activating sentences in the Quora Question Pairs dataset of 164,246 questions. The top sentences were questions about the meaning of song titles, a clean concept. On a question-answering set drawn from Wikipedia (58,645 sentences), the same neuron's top sentences began with dates ("On 16 June 2006, ..."). On BookCorpus, a collection of novels, a third pattern appeared. Each dataset alone told a convincing story, and the stories disagreed.

That standard governs the rest of the chapter, and the later sections take it up in order. Probing asks what information the activations contain. The logit lens reads them in the model's own output vocabulary. Activation patching intervenes on them, and its fastest variant, attribution patching, is gradient times input applied to an activation instead of an input. Superposition explains why single neurons like neuron 221 so often mean several things at once. Whether a measurement measures what it claims is the same question of validity that §22.1 asked about benchmarks.

Read more

• Sundararajan, Taly, Yan (2017). Axiomatic Attribution for Deep Networks. Integrated gradients, the axioms, the saturation example, the uniqueness argument, and practical advice on baselines and step counts.
• Lundberg and Lee (2017). A Unified Approach to Interpreting Model Predictions. SHAP values, the Shapley kernel that turns LIME into KernelSHAP, and the independence approximation used in practice.
• Sundararajan and Najmi (2020). The many Shapley values for model explanation. Why the choice of game matters, with the conditional and interventional versions compared.
• Adebayo et al. (2018). Sanity Checks for Saliency Maps. The model- and data-randomization tests and which methods pass them.
• Jacovi and Goldberg (2020). Towards Faithfully Interpretable NLP Systems: How should we define and evaluate faithfulness?. Faithfulness versus plausibility, and the assumptions behind faithfulness tests.
• Geirhos et al. (2020). Shortcut Learning in Deep Neural Networks. The survey of right-for-the-wrong-reason models across vision and language.
• Olah et al. (2020). Zoom In: An Introduction to Circuits. The case for mechanistic interpretability as a science, with the features, circuits, and universality claims.
• Bolukbasi et al. (2021). An Interpretability Illusion for BERT. Why a neuron's top-activating examples on one dataset can tell the wrong story, and why every later method must be tested across distributions.

Watch next

▶

▶

▶

▶