Chapter 23 · Section 23.2 · 85 min

Probing

A probe is a small classifier that reads one property out of a network's hidden activations. The network stays frozen, and only the classifier is trained. It is the simplest way to ask what a trained model carries inside it, and one of the most used tools in interpretability. It is also the easiest one to over-read. A probe that succeeds shows that a property can be decoded from one place in the network. It does not show that the network uses the property, and much of this section is about telling the two apart.

Here is a question a probe can answer. Li et al. (2022) trained a small GPT, a Transformer that predicts the next token, on nothing but lists of Othello moves. Each move is a square name such as "C4", so a game is a list like "C4, E3, F6". The model never sees a board. Its only job is to name a legal next move, and its top guess is illegal only 0.01 percent of the time. To play that well it seems to need the board, which squares are black, which are white, and which are empty. We cannot ask the model whether it keeps a board. We can record its activations on thousands of games and train a classifier to predict each square's color from them. If that classifier succeeds at layer 6, what have we learned about the model?

§23.1 led to this question. Its tools, the input attributions, say which input tokens an output depended on. They never say what the model computed from those tokens, and to see that we have to read the activations. That section also ranked evidence on a ladder of four rungs. From weakest to strongest they are association, direct effect, causal effect, and mechanism. A probe stands on the lowest rung. It shows that something in the activations tracks a property, without changing anything inside the model.

The section runs in this order. We train a linear probe honestly and ask what its accuracy means, which leads to control tasks. We derive the best linear probe and find that the most accurate reading direction is not the direction the model writes the concept along. We erase a concept in closed form and prove that no linear classifier can recover it. We state the linear representation hypothesis, which explains why linear probes work as often as they do, with its evidence and its counterexamples. We end with probes that need no labels, probes as safety monitors, the statistics of probing many layers, and a probe on every layer of a small Transformer in PyTorch.

A probe is a classifier on frozen activations

A probing experiment has three steps. First, pick a trained model and one place in it, called a site, such as the residual stream after layer 6 at the last token. Second, run the model on many inputs and record the activation vector at that site for each one. Third, label each input with the property you care about, and train a classifier that maps activation to label. The model's weights never change.

The site is usually the residual stream. §5.1 described it as the running sum that every block of a Transformer reads from and adds its output to. Throughout this chapter, $\mathbf{x}^{(\ell)}_t \in \R^d$ is the residual stream at position $t$ after $\ell$ blocks, in a model of width $d$ . So $\mathbf{x}^{(0)}_t$ is the token embedding plus the position embedding. And $\mathbf{x}^{(L)}_t$ , after all $L$ blocks, is the last stream before the final normalization $\operatorname{LN}_f$ , as in §5.6.

Activations are rows. A dataset of $N$ of them stacks into a matrix $\mathbf{X} \in \R^{N\times d}$ , one activation per row, and a probe's weight vector $\mathbf{w} \in \R^d$ scores a row as $\mathbf{x}\cdot\mathbf{w}$ . Inside derivations we write vectors as columns, as the papers do, so the same score is written $\mathbf{x}^\top\mathbf{w}$ .

📐 Probe

Fix a trained model and a site, for example $\mathbf{x}^{(\ell)}_t$ . Collect examples $(\mathbf{x}_n, y_n)$ , $n = 1, \dots, N$ , where $\mathbf{x}_n \in \R^d$ is the activation at that site on input $n$ and $y_n$ is a property of input $n$ . A probe is a model $g$ trained to predict $y$ from $\mathbf{x}$ with the network's weights frozen, and evaluated on examples held out by the unit the claim is about (a sentence, a document, a game, a word type, a prompt template). A linear probe for a binary property $y \in \{0, 1\}$ is

\hat p = \sigma(\mathbf{x}\cdot\mathbf{w} + b), \qquad \mathbf{w} \in \R^d,\ b \in \R,

with $\sigma(z) = 1/(1 + e^{-z})$ the sigmoid. For $K$ classes it is $\operatorname{softmax}(\mathbf{x}\mathbf{W} + \mathbf{b})$ with $\mathbf{W} \in \R^{d\times K}$ and $\mathbf{b} \in \R^K$ .

In plain words, a probe asks whether a property can be read off one place in the network by a reader of fixed, usually tiny, capacity. Only the reader is trained, never the network. Here is a linear probe with numbers. Take $d = 2$ , $\mathbf{w} = (2, -1)$ , and $b = 0.5$ . The activation $\mathbf{x} = (1, 0.5)$ gets the score $2 - 0.5 + 0.5 = 2$ and the probability $\sigma(2) = 0.88$ , so the probe says the property is present.

The words "held out by the unit the claim is about" matter. If the claim is about games, the test games must be games the probe never saw in training, not just new positions from the same games.

📜 Where probing comes from

The idea is older than Transformers. §3.4 met it in ELMo, a language model built from two layers of bidirectional LSTMs, where simple probes on each layer read part of speech best from the lower layer and word sense best from the upper one. Alain and Bengio (2016) put linear "classifier probes" on every layer of the image classifiers Inception v3 and ResNet-50, and found that the classes became steadily more linearly separable with depth. Probing then became the main tool for asking what pretrained language models know. Tenney, Das, and Pavlick (2019) found BERT's layers handling the steps of the classical NLP pipeline in their textbook order, part of speech, then parsing, named entities, semantic roles, and coreference. Hewitt and Manning (2019) went beyond labels with a structural probe, a linear map under which squared distances between word vectors approximate distances in the sentence's parse tree. Belinkov (2021) surveys this literature and its debates. Linear probing is also the standard way to score self-supervised vision models (§28.6), where it measures the quality of a representation rather than what the representation encodes.

Training a linear probe

A linear probe is logistic regression with activations as its inputs. It is trained to make its predicted probabilities match the labels, with a small penalty that keeps the weights from growing large. §0.2 derived the key gradient. The sigmoid is the softmax of the two logits $(0, z)$ , so the derivative of one example's cross-entropy with respect to its logit $z$ is $\hat p_n - y_n$ , predicted minus observed. A probe adds a bias $b$ and an L2 penalty on the weights. The loss is the mean cross-entropy plus the penalty,

\mathcal{L}(\mathbf{w}, b) = \frac1N\sum_{n=1}^N\Big[-y_n\log\hat p_n - (1 - y_n)\log(1 - \hat p_n)\Big] + \frac{\lambda}{2}\lVert\mathbf{w}\rVert^2, \qquad \hat p_n = \sigma(z_n),\ z_n = \mathbf{x}_n\cdot\mathbf{w} + b .

The logit depends on the parameters through $\partial z_n/\partial\mathbf{w} = \mathbf{x}_n$ and $\partial z_n/\partial b = 1$ , and the penalty contributes $\lambda\mathbf{w}$ . The chain rule gives

\nabla_{\mathbf{w}}\mathcal{L} = \frac1N\sum_{n=1}^N(\hat p_n - y_n)\,\mathbf{x}_n + \lambda\mathbf{w} = \frac1N\mathbf{X}^\top(\hat{\mathbf{p}} - \mathbf{y}) + \lambda\mathbf{w}, \qquad \frac{\partial\mathcal{L}}{\partial b} = \frac1N\sum_{n=1}^N(\hat p_n - y_n).

In plain words, every example pushes the weights along its own activation, in proportion to how wrong its predicted probability is, and the penalty pulls the weights back toward zero.

Two facts from Chapter 0 say how to set this fit and why to trust it. First, the penalty is a prior belief that the weights are small. §0.4 showed that an L2 penalty is maximum a posteriori (MAP) estimation under a Gaussian prior. Multiply the mean loss by $N$ and the penalty becomes the negative log of independent $\mathcal{N}(0, \tau^2)$ priors on the weights, with $\lambda = 1/(N\tau^2)$ . So a larger $\lambda$ is a tighter prior. Second, the loss is convex, a bowl with no false bottoms. §0.2 showed that the Hessian of the cross-entropy part is $\mathbf{X}^\top\mathbf{D}\mathbf{X}/N$ , with $\mathbf{D}$ diagonal and nonnegative, which is positive semidefinite. With $\lambda > 0$ the loss is strictly convex, and with both labels present it has a minimum, which is then unique. Gradient descent with a small enough step converges to it from any start.

Five habits separate a probing result from a probing artifact.

Split by the unit of the claim. If test sentences are near copies of training sentences, or share a game, a template, or a document with them, the probe can succeed by recognizing the item rather than reading the property.
Standardize with training statistics only. Centering and scaling with the test set's mean and standard deviation leaks the test distribution into the probe.
Choose the regularization on a validation split, never on the test set.
Report more than accuracy. Give the majority-class rate next to it, the accuracy of always guessing the more common label. Give the AUROC, the probability that a random positive example scores above a random negative one, which needs no threshold. And give an interval (§0.11).
Compare against baselines. Run the same probe on the input embeddings ( $\ell = 0$ ), on a randomly initialized copy of the model, and on a control task, which the next subsection builds.

To try this we need activations where we know the truth. The cell builds the planted-feature stream, a synthetic stand-in for a residual stream that this chapter uses as its laboratory. Each activation is a sum of fixed feature directions with random strengths, plus a little noise,

\mathbf{x} = \sum_{i=1}^n a_i\mathbf{g}_i + \boldsymbol\epsilon .

The $\mathbf{g}_i$ are $n = 192$ random unit vectors in $d = 48$ dimensions. Each strength $a_i$ is zero with probability 0.9 and otherwise uniform on $[0, 1]$ , and $\boldsymbol\epsilon$ is Gaussian noise with standard deviation 0.05 per coordinate. There are more features than dimensions, which is the situation §23.5 studies. The concept we probe for is feature 0. It is on for half the examples, with a strength uniform on $[0.5, 1.5]$ , and off for the rest. We train the probe by full-batch gradient descent, check it against scikit-learn, and report accuracy, AUROC, and a bootstrap interval. The bootstrap resamples the 1,000 test examples with replacement 200 times and reports the middle 95 percent of the AUROCs it gets.

Python · cell 1

The probe reads the concept well. Held-out accuracy is 0.934 against a majority rate near 0.5, and the AUROC is 0.982, with a bootstrap interval from 0.976 to 0.988. Gradient descent lands on scikit-learn's solution. The last line holds a puzzle that we return to below. The probe's weight vector has cosine only 0.864 with $\mathbf{g}_0$ , the direction the concept was planted along. The plain difference between the two class means has cosine 0.997.

Warm-up

A logistic probe from scratch

Implement train_probe(X_tr, y_tr, X_te, y_te, lam, steps, lr), a logistic probe trained by hand.

Compute the per-coordinate mean mu and standard deviation sd (divisor $N$ , NumPy's default) of X_tr alone, and use them to standardize both splits.
Start from $\mathbf{w} = \mathbf{0}$ and $b = 0$ . Run steps steps of full-batch gradient descent with step size lr on the mean cross-entropy plus $\frac{\lambda}{2}\lVert\mathbf{w}\rVert^2$ , with $\lambda$ = lam. Do not penalize the bias.
Return (w, b, mu, sd, acc). Here w and b act on standardized inputs, and acc is the held-out accuracy of the rule "predict class 1 when $\hat p \gt 0.5$ ".

The tests shift the test set, so standardizing with the wrong statistics changes the answer.

import numpy as np

def train_probe(X_tr, y_tr, X_te, y_te, lam=1e-2, steps=300, lr=0.5):
    raise NotImplementedError

Tests, hints, verified solutions, and AI help, with Pro at $7.99 a month.

In practice, a probe result without its baseline and its interval is the first thing a reviewer or an interviewer will challenge. The next question is subtler. Even with a clean split and a good interval, how much of the accuracy belongs to the representation, and how much to the probe?

Control tasks show how much a probe memorizes

A probe is not a passive meter. It is a learner, and a flexible learner can learn the property by itself from almost any representation that keeps the inputs apart. Two words help here. A word type is an entry in the vocabulary, such as "dog". A token is one occurrence of a type in the text, so "the dog saw the cat" has five tokens and four types.

Here is a tiny case. A corpus has three word types, "the", "dog", and "barks", tagged determiner, noun, and verb. If each type's vector is distinct, a probe that memorizes "this vector is the, so determiner" tags every token correctly. It would also score perfectly on any other fixed assignment, say the as a verb and barks as a determiner. A perfect score then says only that the vectors tell the words apart. It says nothing about whether they encode part of speech.

Contextual word vectors are close to this case, because each one is nearly a unique fingerprint of its word. A large probe can reach high part-of-speech accuracy by memorizing word identities. Hewitt and Liang (2019) turned the arbitrary assignment into a measurement, called a control task.

In plain words, the control task has the same outputs and the same label frequencies as the real task, but its labels have nothing to do with language. The only way to solve it is to recognize each word's identity from its vector and memorize an arbitrary label for it. So control accuracy measures how much this probe can memorize on this representation. Selectivity is the part of the task accuracy that memorization cannot explain.

For this to work, train and test must share word types, as they do in the usual splits of the Penn Treebank, the standard corpus of tagged English sentences. A type never seen in training has an unpredictable control label. So control accuracy is capped at the fraction of test tokens whose type appears in training, plus chance on the rest.

Hewitt and Liang measured selectivity on ELMo, a language model built from two layers of bidirectional LSTMs. They compared linear probes with MLP probes, which have one hidden layer of ReLU units.

The control task also changed a conclusion about ELMo. Probes on its first layer beat probes on its second at part-of-speech tagging, the "lower layers are syntactic" finding of §3.4. Hewitt and Liang found the second layer more selective, 31.4 against 26.0 for the linear probe, at accuracies of 96.6 against 97.2. They proposed that the first layer's lead may come partly from word identity being easier to read off it, since it sits closer to the word embeddings. So an accuracy difference between layers is a difference in what the probe could exploit, not only in what each layer encodes.

The cell reproduces the effect on synthetic data. Four hundred word types each get a random vector in 32 dimensions. A binary property of each type is written along one direction $\mathbf{g}$ , and every token gets its own context noise. Both kinds of probe are ridge classifiers, least squares on targets of plus or minus one, which is faster than logistic regression and about as good here. The linear probe reads the raw activations. The MLP probe passes them through a fixed random layer of ReLU units and trains only the readout on top, a cheap stand-in for a trained one-hidden-layer network of the same width.

Python · cell 2

The linear probe gets 0.936 on the task and 0.575 on the control task, a selectivity of 0.361. As the MLP probe widens to 1,024 units, its task accuracy creeps up to 0.972 while its control accuracy climbs to 0.855, and selectivity falls to 0.117. That is the same shape as Hewitt and Liang's table.

The widget runs the same experiment with 48 word types, at four layers of a made-up network. In layer 0 the property is the product of two random signs, an XOR. Tokens that have it sit in two opposite corners of a plane, and tokens that lack it sit in the other two, so no single direction separates them. Layer by layer the property is rewritten, and by layer 3 it lies along one direction. The right panel draws each test token minus its word's own vector, so what is left is the property's code plus the token's context noise. The switch sets how many training tokens each word type gets.

Core

Control tasks and selectivity

Word types are the integers 0..V-1, and t_tr, t_te give the type of each token.

Implement control_map(types, y, V, rng). It returns an array of shape (V,) with one control label per type. Draw each label independently with rng from the empirical distribution of the token labels y.
Implement selectivity(probe, X_tr, t_tr, y_tr, X_te, t_te, y_te, V, rng) and return (acc_task, acc_control, sel). The argument probe(X_tr, labels_tr, X_te) trains a probe and returns its predictions on X_te. Build the control map once, from the training tokens, and use that one map to label both splits.

The setup provides a logistic probe, a 1-nearest-neighbour probe that can memorize anything, and a token generator.

import numpy as np

def control_map(types, y, V, rng):
    raise NotImplementedError

def selectivity(probe, X_tr, t_tr, y_tr, X_te, t_te, y_te, V, rng):
    raise NotImplementedError

Tests, hints, verified solutions, and AI help, with Pro at $7.99 a month.

In practice, use the simplest probe that works, and report a control task or another capacity check whenever the probe is more than linear. Interviewers ask what a control task is, and why an MLP probe with higher accuracy can be the worse experiment.

🔬 Measuring how easily, with description length

Selectivity needs a control task, and a control task exists for properties of word types but not for, say, "this sentence is true". Voita and Titov (2020) measured the same thing, how easily a probe extracts a property, in another way. They count the bits needed to send the labels to someone who already has the activations, using the probe as the code. This count is the description length of the labels given the representations.

In their online code, split the $N$ examples into blocks at $1 \le t_1 \lt t_2 \lt \cdots \lt t_S = N$ . Send the first $t_1$ labels with a uniform code, $t_1\log_2 K$ bits for $K$ classes. For each later block, train the probe on the first $t_i$ examples and send labels $t_i + 1$ through $t_{i+1}$ at the probe's cross-entropy,

L_{\text{online}} = t_1\log_2 K + \sum_{i=1}^{S-1}\ \sum_{n = t_i + 1}^{t_{i+1}} -\log_2 p_{\theta_i}(y_n \mid \mathbf{x}_n),

where $\theta_i$ is the probe trained on the first $t_i$ examples. The compression $N\log_2 K/L_{\text{online}}$ is large when the property can be learned from few examples. A well-organized representation offers that, and a memorized one does not. Voita and Titov found clear differences in description length between trained and randomly initialized representations, and between real and control tasks, in cases where accuracy showed little difference.

Pimentel et al. (2020) argued from the other side. If probing estimates the mutual information between a representation and a property, you should use the best probe you can, since a better probe gives a tighter estimate. But a contextual representation is computed from the whole sentence. By the data processing inequality of §0.5, which says that no processing adds information, it holds no more information about the property than the sentence does. Under their assumptions it holds exactly the same amount. Both papers conclude that "is the information there" is the wrong question, and that how easily it can be extracted is the one worth answering.

Three ways to choose the probe's direction

A linear probe is a direction plus a threshold. It scores an activation by its dot product with $\mathbf{w}$ and compares the score with the threshold. There are three standard ways to choose the direction. The difference of means takes the average activation of class 1 minus the average activation of class 0. Linear discriminant analysis (LDA) first undoes the spread that both classes share, then takes the difference of means. Logistic regression fits the direction by gradient descent, as in the first cell.

Now back to the puzzle from that cell. The logistic probe and the difference of means pointed in noticeably different directions, and the difference of means was closer to where the concept was planted. Four points in the plane show why.

✏️ Four points, two probes, by hand

Class 1 is $(3, 4)$ and $(-1, -4)$ , class 0 is $(1, 3)$ and $(-3, -3)$ . The class means are $\boldsymbol\mu_1 = (1, 0)$ and $\boldsymbol\mu_0 = (-1, 0)$ , so the mean difference is $\boldsymbol\delta = \boldsymbol\mu_1 - \boldsymbol\mu_0 = (2, 0)$ and the midpoint of the means is the origin. The difference-of-means probe scores $\boldsymbol\delta\cdot\mathbf{x}$ , which is $6$ and $-2$ for class 1 and $2$ and $-6$ for class 0. Thresholded at the midpoint's score, 0, it gets two of the four points wrong, accuracy 0.5.

Now measure the spread inside each class. The deviations from the class means are $\pm(2, 4)$ in class 1 and $\pm(2, 3)$ in class 0. The pooled within-class covariance measures how points spread around their own class mean, with both classes pooled. It averages the outer products of the deviations over the four points,

\boldsymbol\Sigma_w = \tfrac14\big[2(2, 4)^\top(2, 4) + 2(2, 3)^\top(2, 3)\big] = \begin{pmatrix} 4 & 7 \\ 7 & 12.5\end{pmatrix}, \qquad \det\boldsymbol\Sigma_w = 50 - 49 = 1,

so $\boldsymbol\Sigma_w^{-1} = \begin{pmatrix} 12.5 & -7 \\ -7 & 4\end{pmatrix}$ . The LDA direction is $\boldsymbol\Sigma_w^{-1}\boldsymbol\delta = (25, -14)$ . Its scores are $19$ and $31$ for class 1 and $-17$ and $-33$ for class 0, accuracy 1. The spread within each class runs roughly along $(1, 2)$ . LDA turns its direction until that shared spread is nearly invisible to it, and what is left is the difference between the classes.

Python · cell 3

The general statement is a classical theorem, which goes back to the statistician Ronald Fisher in 1936. It says when a linear probe is the best possible classifier, and which direction that probe uses. Write $\pi_1 = \mathbb{P}(y = 1)$ and $\pi_0 = 1 - \pi_1$ for the class priors, the fractions of each class, and $\Phi$ for the standard normal cumulative distribution function.

∴ The optimal linear probe for shared-covariance Gaussians

Suppose $\mathbf{x}\mid y \sim \mathcal{N}(\boldsymbol\mu_y, \boldsymbol\Sigma_w)$ for $y \in \{0, 1\}$ , with one positive definite covariance $\boldsymbol\Sigma_w$ for both classes, and let $\boldsymbol\delta = \boldsymbol\mu_1 - \boldsymbol\mu_0$ .

The log-odds are exactly linear, $\log\dfrac{\mathbb{P}(y = 1\mid\mathbf{x})}{\mathbb{P}(y = 0\mid\mathbf{x})} = \mathbf{x}^\top\mathbf{w}^\star + b^\star$ with $\mathbf{w}^\star = \boldsymbol\Sigma_w^{-1}\boldsymbol\delta$ and $b^\star = -\tfrac12(\boldsymbol\mu_1 + \boldsymbol\mu_0)^\top\boldsymbol\Sigma_w^{-1}\boldsymbol\delta + \log(\pi_1/\pi_0)$ . So the Bayes-optimal classifier is a linear probe, and logistic regression fit to such data converges to it.
With equal priors, take any direction $\mathbf{u}$ with $\mathbf{u}^\top\boldsymbol\delta \gt 0$ . The probe that scores $\mathbf{x}^\top\mathbf{u}$ and thresholds at the midpoint of the projected means has accuracy $\Phi\big(\tfrac12\,\mathbf{u}^\top\boldsymbol\delta/\sqrt{\mathbf{u}^\top\boldsymbol\Sigma_w\mathbf{u}}\big)$ . It is largest exactly when $\mathbf{u} \propto \boldsymbol\Sigma_w^{-1}\boldsymbol\delta$ , where it equals $\Phi(\kappa/2)$ with $\kappa^2 = \boldsymbol\delta^\top\boldsymbol\Sigma_w^{-1}\boldsymbol\delta$ , the squared Mahalanobis distance between the means.
The difference-of-means probe, $\mathbf{u} = \boldsymbol\delta$ , is optimal if and only if $\boldsymbol\Sigma_w^{-1}\boldsymbol\delta \propto \boldsymbol\delta$ , that is, if and only if $\boldsymbol\delta$ is an eigenvector of $\boldsymbol\Sigma_w$ . Isotropic noise, $\boldsymbol\Sigma_w \propto \mathbf{I}$ , is the common special case.

∂ Proof of the three parts

Part 1. The Gaussian log-density is $\log\mathcal{N}(\mathbf{x}; \boldsymbol\mu, \boldsymbol\Sigma_w) = -\tfrac12(\mathbf{x} - \boldsymbol\mu)^\top\boldsymbol\Sigma_w^{-1}(\mathbf{x} - \boldsymbol\mu) - \tfrac12\log\det(2\pi\boldsymbol\Sigma_w)$ . By Bayes' rule the log-odds are $\log\pi_1 - \log\pi_0$ plus the difference of the two log-densities. The determinant terms are equal and cancel because the covariance is shared. Expanding the quadratics, the terms $\mathbf{x}^\top\boldsymbol\Sigma_w^{-1}\mathbf{x}$ also cancel, and what is left is

-\tfrac12\Big[-2\mathbf{x}^\top\boldsymbol\Sigma_w^{-1}\boldsymbol\mu_1 + \boldsymbol\mu_1^\top\boldsymbol\Sigma_w^{-1}\boldsymbol\mu_1 + 2\mathbf{x}^\top\boldsymbol\Sigma_w^{-1}\boldsymbol\mu_0 - \boldsymbol\mu_0^\top\boldsymbol\Sigma_w^{-1}\boldsymbol\mu_0\Big] = \mathbf{x}^\top\boldsymbol\Sigma_w^{-1}\boldsymbol\delta - \tfrac12(\boldsymbol\mu_1 + \boldsymbol\mu_0)^\top\boldsymbol\Sigma_w^{-1}\boldsymbol\delta,

using $\boldsymbol\mu_1^\top\boldsymbol\Sigma_w^{-1}\boldsymbol\mu_1 - \boldsymbol\mu_0^\top\boldsymbol\Sigma_w^{-1}\boldsymbol\mu_0 = (\boldsymbol\mu_1 + \boldsymbol\mu_0)^\top\boldsymbol\Sigma_w^{-1}(\boldsymbol\mu_1 - \boldsymbol\mu_0)$ , which holds because $\boldsymbol\Sigma_w^{-1}$ is symmetric. The posterior is the sigmoid of these log-odds, which is the logistic model with parameters $(\mathbf{w}^\star, b^\star)$ . Since the logistic family contains the true conditional, its maximum-likelihood estimate is consistent and converges to $(\mathbf{w}^\star, b^\star)$ as $N$ grows.

Part 2. The score $s = \mathbf{x}^\top\mathbf{u}$ is Gaussian in each class, with means $\mathbf{u}^\top\boldsymbol\mu_1$ and $\mathbf{u}^\top\boldsymbol\mu_0$ and the same variance $\mathbf{u}^\top\boldsymbol\Sigma_w\mathbf{u}$ . With equal priors and equal variances the best threshold is the midpoint $m$ . A class-1 point is classified correctly with probability $\mathbb{P}(s \gt m) = \Phi\big((\mathbf{u}^\top\boldsymbol\mu_1 - m)/\sqrt{\mathbf{u}^\top\boldsymbol\Sigma_w\mathbf{u}}\big) = \Phi\big(\tfrac12\mathbf{u}^\top\boldsymbol\delta/\sqrt{\mathbf{u}^\top\boldsymbol\Sigma_w\mathbf{u}}\big)$ , and by symmetry a class-0 point with the same probability. Since $\Phi$ is increasing, maximize $J(\mathbf{u}) = (\mathbf{u}^\top\boldsymbol\delta)^2/(\mathbf{u}^\top\boldsymbol\Sigma_w\mathbf{u})$ , Fisher's criterion. Put $\mathbf{a} = \boldsymbol\Sigma_w^{1/2}\mathbf{u}$ and $\mathbf{c} = \boldsymbol\Sigma_w^{-1/2}\boldsymbol\delta$ , so that $\mathbf{u}^\top\boldsymbol\delta = \mathbf{a}^\top\mathbf{c}$ and $\mathbf{u}^\top\boldsymbol\Sigma_w\mathbf{u} = \lVert\mathbf{a}\rVert^2$ . By Cauchy-Schwarz, $J = (\mathbf{a}^\top\mathbf{c})^2/\lVert\mathbf{a}\rVert^2 \le \lVert\mathbf{c}\rVert^2 = \kappa^2$ , with equality exactly when $\mathbf{a} \propto \mathbf{c}$ , that is, $\mathbf{u} \propto \boldsymbol\Sigma_w^{-1}\boldsymbol\delta$ .

Part 3. By part 2 the direction $\boldsymbol\delta$ is optimal exactly when it is a positive multiple of $\boldsymbol\Sigma_w^{-1}\boldsymbol\delta$ , which says $\boldsymbol\Sigma_w\boldsymbol\delta = c\,\boldsymbol\delta$ for some $c \gt 0$ . ∎

In plain words, the best linear reader of a concept first whitens away the spread that the concept does not cause, then reads along the mean difference. To whiten is to change coordinates linearly so that a covariance becomes the identity (§0.10). Here the change is $\boldsymbol\Sigma_w^{-1/2}$ , after which each class is a round cloud and the difference of means is optimal by part 3. Undoing the change turns that direction into $\boldsymbol\Sigma_w^{-1}\boldsymbol\delta$ in the original coordinates. The number $\kappa$ is the Mahalanobis distance between the means, their distance measured in units of the within-class spread.

Part 2 makes the four points quantitative. Pretend each class were a Gaussian with the covariance $\boldsymbol\Sigma_w$ just computed. The difference of means $\mathbf{u} = (2, 0)$ has $\mathbf{u}^\top\boldsymbol\delta = 4$ and $\mathbf{u}^\top\boldsymbol\Sigma_w\mathbf{u} = 16$ , so its accuracy would be $\Phi(\tfrac12\cdot 4/4) = \Phi(0.5) = 0.69$ . LDA reaches $\kappa^2 = \boldsymbol\delta^\top\boldsymbol\Sigma_w^{-1}\boldsymbol\delta = (2, 0)\cdot(25, -14) = 50$ and accuracy $\Phi(\sqrt{50}/2) = \Phi(3.54) = 0.9998$ . The same means and the same spread give 69 percent along one direction and near certainty along the other.

One more identity says that it does not matter which covariance we whiten with, and erasure will use it.

Proof. The law of total covariance splits $\boldsymbol\Sigma$ into the average within-class covariance, $\boldsymbol\Sigma_w$ , plus the covariance of the class means about the overall mean, which is $\pi_1\pi_0\boldsymbol\delta\boldsymbol\delta^\top$ . Multiplying out, $\boldsymbol\Sigma\,(\boldsymbol\Sigma_w^{-1}\boldsymbol\delta) = \boldsymbol\delta + \pi_1\pi_0\boldsymbol\delta\,(\boldsymbol\delta^\top\boldsymbol\Sigma_w^{-1}\boldsymbol\delta) = (1 + \pi_1\pi_0\kappa^2)\,\boldsymbol\delta$ . Divide by the scalar and multiply by $\boldsymbol\Sigma^{-1}$ . ∎

How does the logistic probe fit in? Its L2 penalty decides where it lands. A strong penalty pulls it all the way to the difference of means, and a weak one lets it reach the LDA direction.

∴ Strong L2 turns a logistic probe into a difference of means

Let $(\mathbf{w}_\lambda, b_\lambda)$ minimize the mean cross-entropy plus $\frac{\lambda}{2}\lVert\mathbf{w}\rVert^2$ on a fixed dataset with both labels present, the bias unpenalized. Write $\bar y$ for the fraction of examples with $y = 1$ and $\hat{\boldsymbol\mu}_1, \hat{\boldsymbol\mu}_0$ for the sample class means. Then as $\lambda \to \infty$ ,

\lambda\,\mathbf{w}_\lambda \;\to\; \hat{\boldsymbol\sigma}_{xy} = \frac1N\sum_{n=1}^N(y_n - \bar y)\,\mathbf{x}_n = \bar y(1 - \bar y)(\hat{\boldsymbol\mu}_1 - \hat{\boldsymbol\mu}_0),

so the direction of $\mathbf{w}_\lambda$ tends to the difference of the class means. With $\lambda \to 0$ and many examples from shared-covariance Gaussians, it tends instead to the LDA direction, by part 1 of the theorem on the optimal linear probe.

∂ Proof of the strong-L2 limit

Setting the gradient to zero gives two conditions, $\lambda\mathbf{w}_\lambda = \frac1N\sum_n(y_n - \hat p_n)\mathbf{x}_n$ and $\sum_n(y_n - \hat p_n) = 0$ . Comparing the minimum with $\mathbf{w} = \mathbf{0}$ gives $\frac{\lambda}{2}\lVert\mathbf{w}_\lambda\rVert^2 \le \min_b\mathcal{L}(\mathbf{0}, b) \le \log 2$ , so $\mathbf{w}_\lambda \to \mathbf{0}$ . Every $\hat p_n$ then tends to the same value $\sigma(b_\lambda)$ , and the bias condition forces that value to $\bar y$ . Substituting $\hat p_n \to \bar y$ in the first condition gives the limit. Finally, $\frac1N\sum_n(y_n - \bar y)\mathbf{x}_n = \bar y\hat{\boldsymbol\mu}_1 - \bar y(\bar y\hat{\boldsymbol\mu}_1 + (1 - \bar y)\hat{\boldsymbol\mu}_0) = \bar y(1 - \bar y)(\hat{\boldsymbol\mu}_1 - \hat{\boldsymbol\mu}_0)$ . ∎

In plain words, a heavily penalized probe barely moves away from predicting the base rate, and its first small step from there is along the class-mean difference. So the penalty slides a logistic probe between the difference of means and the LDA direction. The next subsection shows that these two play different roles, one marks where the concept is written and the other is the best way to read it. When you compare probe directions across layers or papers, you must know where on that slide each one sits.

The most accurate probe is not the direction the model writes

The four points already showed that the best classifier can point away from the mean difference. Now we ask what each direction does when we add it to the activations. We need a toy where we know the answer. A concept is written along a unit vector $\mathbf{g}_c$ . A nuisance feature, unrelated to the label, is written along a unit vector $\mathbf{g}_f$ that overlaps it, $\mathbf{g}_c\cdot\mathbf{g}_f = 0.5$ . Overlaps like this are typical when a network stores more features than it has dimensions (§23.5). Each activation in $\R^{32}$ is

\mathbf{x} = (y - \tfrac12)\,\mathbf{g}_c + z\,\mathbf{g}_f + \boldsymbol\epsilon, \qquad z \sim \mathcal{N}(0, 4),\ \boldsymbol\epsilon \sim \mathcal{N}(\mathbf{0}, 0.09\,\mathbf{I}).

Here $y \in \{0, 1\}$ is the label and $z$ is the nuisance's random strength. Suppose the network downstream reads each feature with a readout that recovers that feature's value exactly and ignores the other. For two overlapping directions these readouts are the dual vectors $\mathbf{g}_c^\ast$ and $\mathbf{g}_f^\ast$ , defined by $\mathbf{g}_c^\ast\cdot\mathbf{g}_c = 1$ , $\mathbf{g}_c^\ast\cdot\mathbf{g}_f = 0$ , and the same with the roles swapped.

We fit the three probes. Then we steer with each one, which means adding a vector to the activations, as §23.7 does at scale. We read what the step does to both features through their dual vectors. Each step $\boldsymbol\theta$ points along the probe's own direction. It is scaled so that it moves the probe's score by the gap between the class means, as Marks and Tegmark (2023) do.

Python · cell 4

The difference of means points along $\mathbf{g}_c$ (cosine 0.998) but classifies poorly, 0.674, as part 2 of the theorem predicts (0.673). LDA and logistic regression tilt away from the nuisance and reach about 0.92, against a prediction of 0.925. Steering moves the concept by its full class gap in every case. Along the LDA or logistic direction, though, it also moves the nuisance by about half a unit (minus 0.49) for every unit the concept moves. The difference of means leaves the nuisance nearly alone, and its 0.05 is sampling error in the estimated means.

So the accurate probe is the right reader, and the difference of means is the right writer. They coincide only when the nuisance spread is orthogonal to the concept, which is part 3 of the theorem again.

The next cell checks the strong-L2 theorem on the same data. scikit-learn sets the penalty through $C = 1/(\lambda N)$ , so a small $C$ is a strong penalty.

Python · cell 5

The regularization sweep confirms the strong-L2 theorem. At $C = 10^{-5}$ , the strongest penalty, the logistic probe is the difference of means (cosine 1.000). As the penalty weakens it rotates to the LDA direction, reaching cosine 0.999 at $C = 0.1$ and 0.997 at $C = 10$ .

✏️ Truth directions in LLaMA-2, at real scale

Marks and Tegmark (2023) built datasets of simple true and false statements, such as "The city of Paris is in France", and probed LLaMA-2 7B, 13B, and 70B. They compared logistic regression with mass-mean probes, their name for the difference between the mean activations on true and on false statements. In the 13B and 70B models the two classified about equally well, including on held-out datasets. In the 7B model the mass-mean probe was less accurate. They then added each probe's direction, scaled as in the cell above, to the hidden states at the end of each statement. Their patching experiments had located the truth computation there. They measured how far the model's own TRUE or FALSE answer moved, normalized so that 1 is a full flip, on statements about Spanish-English translations that the probes had never seen. For LLaMA-2-13B with probes trained on statements about cities, pushing false statements toward true reached 0.77 with the mass-mean direction against 0.13 with the logistic one. Mass-mean directions were the more causal in 7 of their 8 conditions. Their recommendation is the theorem's split of roles. Classify with the whitened direction $\boldsymbol\Sigma_w^{-1}\boldsymbol\delta$ , which they call the IID mass-mean probe and which coincides with LDA, and steer with $\boldsymbol\delta$ .

The widget below runs the same comparison in two dimensions. The black dashed arrow is where the concept is written, the grey dotted one is the nuisance, and each colored arrow is one probe's direction. The bars give each probe's accuracy on new points and how far steering along it moves the nuisance for every unit it moves the concept.

Interview

Difference of means, LDA, and their thresholds

Implement mean_diff_probe(X, y) and lda_probe(X, y, shrink=0.0). Each returns (w, b) for the rule "class 1 when $\mathbf{x}\cdot\mathbf{w} + b \gt 0$ ", with the threshold at the projected midpoint of the two class means. The first uses $\mathbf{w} = \boldsymbol\mu_1 - \boldsymbol\mu_0$ . The second uses $\mathbf{w} = \boldsymbol\Sigma_s^{-1}(\boldsymbol\mu_1 - \boldsymbol\mu_0)$ . Here $\boldsymbol\Sigma_w$ is the pooled within-class covariance, built from the deviation of every point from its own class mean with divisor $N$ . The shrunk covariance $\boldsymbol\Sigma_s = (1 - s)\boldsymbol\Sigma_w + s\,\frac{\operatorname{tr}\boldsymbol\Sigma_w}{d}\mathbf{I}$ pulls it toward a multiple of the identity by the amount shrink $= s$ . One test has more dimensions than examples, where $\boldsymbol\Sigma_w$ alone cannot be inverted.

import numpy as np

def mean_diff_probe(X, y):
    raise NotImplementedError

def lda_probe(X, y, shrink=0.0):
    raise NotImplementedError

Tests, hints, verified solutions, and AI help, with Pro at $7.99 a month.

This distinction is a favorite interview question. Expect to be asked when the difference of means is Bayes-optimal, to derive LDA from two Gaussians, and to explain why the best classifier is not the best steering vector.

Erasing a concept to test whether it is used

A probe shows that a property is decodable. To ask whether the network uses it, remove the property from the activations and check whether the network's behavior changes. Elazar et al. (2021) called this amnesic probing. It raises a precise question first. What does it mean for a concept to be gone?

One natural answer is that no linear probe can find it anymore. That is the definition below. It quantifies over every linear probe and every reasonable training loss at once.

In plain words, after erasure no linear probe, trained with cross-entropy, squared error, hinge loss, or any other convex loss, can do better than ignoring $\mathbf{x}$ and predicting the base rate. That is a strong statement. The useful fact is that it reduces to one vector being zero. Write

\boldsymbol\sigma_{xy} = \operatorname{Cov}(\mathbf{x}, y) = \mathbb{E}[\mathbf{x}y] - \mathbb{E}[\mathbf{x}]\,\mathbb{E}[y] \in \R^d

for the cross-covariance of the activations with the label.

∂ Proof of the equivalence

(1 and 2.) Since $\mathbb{E}[\mathbf{x}y] = \pi_1\boldsymbol\mu_1$ and $\mathbb{E}[\mathbf{x}] = \pi_1\boldsymbol\mu_1 + \pi_0\boldsymbol\mu_0$ , we get $\boldsymbol\sigma_{xy} = \pi_1\boldsymbol\mu_1 - \pi_1(\pi_1\boldsymbol\mu_1 + \pi_0\boldsymbol\mu_0) = \pi_1\pi_0(\boldsymbol\mu_1 - \boldsymbol\mu_0)$ , and $\pi_1\pi_0 \gt 0$ .

(1 implies 3.) Take any affine $\eta(\mathbf{x}) = \mathbf{x}^\top\mathbf{w} + b$ and any loss $\ell$ convex in its first argument. Condition on the class and apply Jensen's inequality to the convex function $\mathbf{x} \mapsto \ell(\eta(\mathbf{x}), y)$ ,

\mathbb{E}\big[\ell(\eta(\mathbf{x}), y)\big] = \sum_{y'}\pi_{y'}\,\mathbb{E}\big[\ell(\eta(\mathbf{x}), y')\mid y = y'\big] \ \ge\ \sum_{y'}\pi_{y'}\,\ell\big(\eta(\boldsymbol\mu_{y'}), y'\big) = \mathbb{E}\big[\ell(c, y)\big], \qquad c = \eta(\boldsymbol\mu),

where the last step uses $\boldsymbol\mu_1 = \boldsymbol\mu_0 = \boldsymbol\mu$ . So $\eta$ is no better than the constant $c$ .

(3 implies 2.) Use the logistic loss, one of the convex losses that guardedness covers. Its best constant is $b_0 = \log(\pi_1/\pi_0)$ , where $\sigma(b_0) = \pi_1$ . Move from $(\mathbf{0}, b_0)$ a distance $t$ along the weights $\mathbf{w} = \boldsymbol\sigma_{xy}$ . The derivative of the expected loss at $t = 0$ is $\mathbb{E}[(\sigma(b_0) - y)\,\mathbf{x}^\top\boldsymbol\sigma_{xy}] = -\boldsymbol\sigma_{xy}^\top\boldsymbol\sigma_{xy}$ , using $\mathbb{E}[(\pi_1 - y)\mathbf{x}] = -\boldsymbol\sigma_{xy}$ . If $\boldsymbol\sigma_{xy} \ne \mathbf{0}$ this is negative, so a small step beats the best constant and $\mathbf{x}$ does not guard $y$ . ∎

In plain words, a linear reader can beat a constant guess only if the two classes sit at different average positions. Make the class means equal and every linear reader is blind. Many affine maps, a matrix times $\mathbf{x}$ plus a vector, make the means equal. Which one changes the representation least? The answer is called LEACE, for least-squares concept erasure.

∴ LEACE for one binary concept (Belrose et al. 2023)

Let $\boldsymbol\mu = \mathbb{E}[\mathbf{x}]$ , let $\boldsymbol\Sigma = \operatorname{Cov}(\mathbf{x})$ be positive definite, and let $\boldsymbol\sigma_{xy} \ne \mathbf{0}$ . The map

r(\mathbf{x}) = \mathbf{x} - \boldsymbol\sigma_{xy}\,\frac{\boldsymbol\sigma_{xy}^\top\boldsymbol\Sigma^{-1}(\mathbf{x} - \boldsymbol\mu)}{\boldsymbol\sigma_{xy}^\top\boldsymbol\Sigma^{-1}\boldsymbol\sigma_{xy}}

satisfies $\operatorname{Cov}(r(\mathbf{x}), y) = \mathbf{0}$ , so $r(\mathbf{x})$ linearly guards $y$ , and $\mathbb{E}[r(\mathbf{x})] = \boldsymbol\mu$ . Among all affine maps whose output linearly guards $y$ , it minimizes the mean squared change $\mathbb{E}\lVert r(\mathbf{x}) - \mathbf{x}\rVert^2$ , and the minimum is $\lVert\boldsymbol\sigma_{xy}\rVert^2/(\boldsymbol\sigma_{xy}^\top\boldsymbol\Sigma^{-1}\boldsymbol\sigma_{xy})$ . Belrose et al. prove that the same map is optimal for every norm induced by an inner product, the Mahalanobis norm included. For $k$ classes they give it as $r(\mathbf{x}) = \mathbf{x} - \boldsymbol\Sigma^{1/2}\mathbf{P}\boldsymbol\Sigma^{-1/2}(\mathbf{x} - \boldsymbol\mu)$ , with $\mathbf{P}$ the orthogonal projection onto the column space of $\boldsymbol\Sigma^{-1/2}\boldsymbol\Sigma_{xy}$ , where $\boldsymbol\Sigma_{xy} \in \R^{d\times k}$ is the cross-covariance with the one-hot label (pseudoinverses replace inverses when $\boldsymbol\Sigma$ is singular).

In plain words, LEACE reads the concept with the whitened reader $\boldsymbol\Sigma^{-1}\boldsymbol\sigma_{xy}$ and subtracts exactly that much along $\boldsymbol\sigma_{xy}$ . By the proof of the equivalence above, $\boldsymbol\sigma_{xy}$ is a positive multiple of the mean difference $\boldsymbol\delta$ , the direction the concept is written along. By the total-covariance theorem above, $\boldsymbol\Sigma^{-1}\boldsymbol\sigma_{xy}$ points along the LDA direction, the best reader. So LEACE removes the concept where it is written, by the amount the best probe reads. The four points from before make this concrete.

✏️ Erasing the concept from the four points, by hand

The four points have mean $\boldsymbol\mu = (0, 0)$ and half of them are in class 1. Their total covariance and cross-covariance are

\boldsymbol\Sigma = \tfrac14\sum_n\mathbf{x}_n\mathbf{x}_n^\top = \begin{pmatrix} 5 & 7 \\ 7 & 12.5\end{pmatrix}, \qquad \boldsymbol\sigma_{xy} = \tfrac14\sum_n(y_n - \tfrac12)\,\mathbf{x}_n = (0.5, 0),

which is $\pi_1\pi_0\boldsymbol\delta = \tfrac14(2, 0)$ , as the theorem says. With $\det\boldsymbol\Sigma = 13.5$ , the reader is $\boldsymbol\Sigma^{-1}\boldsymbol\sigma_{xy} = (6.25, -3.5)/13.5$ , and dividing by $\boldsymbol\sigma_{xy}^\top\boldsymbol\Sigma^{-1}\boldsymbol\sigma_{xy} = 3.125/13.5$ scales it to $\mathbf{a} = (2, -1.12)$ . So LEACE is

r(\mathbf{x}) = \mathbf{x} - (0.5, 0)\,(2x_1 - 1.12\,x_2) = (0.56\,x_2,\ x_2).

The concept lived on the first coordinate. LEACE does not delete that coordinate. It replaces $x_1$ by $0.56\,x_2$ , the best linear prediction of $x_1$ from $x_2$ (the slope $0.56$ is $7/12.5$ , the regression coefficient), and throws away only the part of $x_1$ that $x_2$ cannot explain. The erased points are $(2.24, 4)$ and $(-2.24, -4)$ in class 1 and $(1.68, 3)$ and $(-1.68, -3)$ in class 0. Both class means are now the origin. The mean squared change is $\lVert\boldsymbol\sigma_{xy}\rVert^2/(\boldsymbol\sigma_{xy}^\top\boldsymbol\Sigma^{-1}\boldsymbol\sigma_{xy}) = 0.25/(3.125/13.5) = 1.08$ . Deleting the first coordinate outright, the orthogonal projection along $\boldsymbol\delta$ , also equalizes the means, but it moves the points by $(9 + 1 + 1 + 9)/4 = 5$ on average.

Look at the erased points once more. A logistic probe trained on them learns $\mathbf{w} = \mathbf{0}$ , as the theorem promises. Yet the nonlinear rule $|x_2| \gt 3.5$ still separates the classes perfectly. Linear erasure removes linearly available information, and only that. One more subtlety hides here. Guardedness is a statement about convex losses, and accuracy is not one, so a single linear threshold, $x_2 \gt 3.5$ , still gets 3 of the 4 points right. No probe trained with a convex loss will find that threshold, but a search that tunes the threshold for accuracy can.

Python · cell 6

∂ Proof of LEACE, and why the binary formula is the general one

Guardedness and the mean. Covariance is linear and constants do not change it, so $\operatorname{Cov}(r(\mathbf{x}), y) = \boldsymbol\sigma_{xy} - \boldsymbol\sigma_{xy}\,\boldsymbol\sigma_{xy}^\top\boldsymbol\Sigma^{-1}\boldsymbol\sigma_{xy}/(\boldsymbol\sigma_{xy}^\top\boldsymbol\Sigma^{-1}\boldsymbol\sigma_{xy}) = \mathbf{0}$ , and $\mathbb{E}[\mathbf{x} - \boldsymbol\mu] = \mathbf{0}$ gives $\mathbb{E}[r(\mathbf{x})] = \boldsymbol\mu$ . By the previous theorem $r(\mathbf{x})$ linearly guards $y$ .

Optimality. Any affine map can be written $\mathbf{x} \mapsto \mathbf{x} - \mathbf{Q}(\mathbf{x} - \boldsymbol\mu) + \mathbf{c}$ for some matrix $\mathbf{Q}$ and vector $\mathbf{c}$ , and its output guards $y$ exactly when $\mathbf{Q}\boldsymbol\sigma_{xy} = \boldsymbol\sigma_{xy}$ . The mean squared change is $\mathbb{E}\lVert\mathbf{Q}(\mathbf{x} - \boldsymbol\mu)\rVert^2 + \lVert\mathbf{c}\rVert^2$ , because the cross term has mean zero, so take $\mathbf{c} = \mathbf{0}$ . What remains is $\operatorname{tr}(\mathbf{Q}\boldsymbol\Sigma\mathbf{Q}^\top) = \sum_i\mathbf{q}_i^\top\boldsymbol\Sigma\mathbf{q}_i$ over the rows $\mathbf{q}_i^\top$ of $\mathbf{Q}$ , and the constraint says, row by row, that $\mathbf{q}_i^\top\boldsymbol\sigma_{xy} = \sigma_i$ , the $i$ -th entry of $\boldsymbol\sigma_{xy}$ . The rows decouple. For each one, Cauchy-Schwarz applied to $\boldsymbol\Sigma^{1/2}\mathbf{q}_i$ and $\boldsymbol\Sigma^{-1/2}\boldsymbol\sigma_{xy}$ gives

\sigma_i^2 = \big(\mathbf{q}_i^\top\boldsymbol\sigma_{xy}\big)^2 \le \big(\mathbf{q}_i^\top\boldsymbol\Sigma\mathbf{q}_i\big)\big(\boldsymbol\sigma_{xy}^\top\boldsymbol\Sigma^{-1}\boldsymbol\sigma_{xy}\big),

with equality exactly when $\mathbf{q}_i \propto \boldsymbol\Sigma^{-1}\boldsymbol\sigma_{xy}$ . The constraint fixes the constant, $\mathbf{q}_i = \sigma_i\boldsymbol\Sigma^{-1}\boldsymbol\sigma_{xy}/(\boldsymbol\sigma_{xy}^\top\boldsymbol\Sigma^{-1}\boldsymbol\sigma_{xy})$ . Stacking the rows gives $\mathbf{Q} = \boldsymbol\sigma_{xy}\boldsymbol\sigma_{xy}^\top\boldsymbol\Sigma^{-1}/(\boldsymbol\sigma_{xy}^\top\boldsymbol\Sigma^{-1}\boldsymbol\sigma_{xy})$ , which is the map $r$ , and summing the bounds gives the minimum $\sum_i\sigma_i^2/(\boldsymbol\sigma_{xy}^\top\boldsymbol\Sigma^{-1}\boldsymbol\sigma_{xy})$ .

The general formula. With one concept, $\boldsymbol\Sigma_{xy} = \boldsymbol\sigma_{xy}$ and $\mathbf{P} = \mathbf{v}\mathbf{v}^\top/(\mathbf{v}^\top\mathbf{v})$ for $\mathbf{v} = \boldsymbol\Sigma^{-1/2}\boldsymbol\sigma_{xy}$ . Then $\boldsymbol\Sigma^{1/2}\mathbf{P}\boldsymbol\Sigma^{-1/2} = \boldsymbol\sigma_{xy}\boldsymbol\sigma_{xy}^\top\boldsymbol\Sigma^{-1}/(\boldsymbol\sigma_{xy}^\top\boldsymbol\Sigma^{-1}\boldsymbol\sigma_{xy})$ , the same $\mathbf{Q}$ . Read the general formula right to left and it says whiten (§0.10), project out the concept's direction in whitened space, and unwhiten. ∎

The matrix $\mathbf{Q} = \boldsymbol\sigma_{xy}\mathbf{a}^\top$ in LEACE satisfies $\mathbf{Q}^2 = \mathbf{Q}$ but is not symmetric. It is an oblique projection, one that removes a direction while measuring along a different one, and that is how it beats every orthogonal projection. Two simpler erasers make the comparison concrete.

Mean projection is the orthogonal projection that removes the unit vector along $\boldsymbol\delta$ . It also equalizes the means, so it also guards linearly. But it throws away all the variance along $\boldsymbol\delta$ , the nuisance's included. Its mean squared change is $\boldsymbol\sigma_{xy}^\top\boldsymbol\Sigma\boldsymbol\sigma_{xy}/\lVert\boldsymbol\sigma_{xy}\rVert^2$ . By Cauchy-Schwarz on $\boldsymbol\Sigma^{1/2}\boldsymbol\sigma_{xy}$ and $\boldsymbol\Sigma^{-1/2}\boldsymbol\sigma_{xy}$ , that is never less than LEACE's, with equality only when $\boldsymbol\sigma_{xy}$ is an eigenvector of $\boldsymbol\Sigma$ .
Projecting out the probe's weight vector removes the reading direction, so it destroys that one probe. But $(\mathbf{I} - \hat{\mathbf{w}}\hat{\mathbf{w}}^\top)\boldsymbol\sigma_{xy} \ne \mathbf{0}$ unless $\boldsymbol\sigma_{xy}$ is parallel to $\mathbf{w}$ , so a freshly trained probe finds what is left. Iterative nullspace projection, INLP (Ravfogel et al. 2020), repeats the step. It trains a new probe and projects out its direction, again and again, until accuracy reaches the majority rate. It gets there, but each round removes a whole direction, so it usually deletes far more than the concept.

The cell runs all three erasers on the toy from the last subsection, the concept and the overlapping nuisance.

Python · cell 7

LEACE and mean projection both drive the cross-covariance to rounding error and a freshly trained probe to 0.501, chance. LEACE does it with a mean squared change of 0.378, against 1.470 for mean projection, nearly four times less. Projecting out the logistic probe's direction changes the data least, 0.271, and fails, since a fresh probe still reaches 0.575.

Interview

Erase a concept in closed form

Implement leace_binary(X, y, ridge=1e-9) for activations X of shape (N, d) and binary labels y. Estimate from the data the mean $\boldsymbol\mu$ , the total covariance $\boldsymbol\Sigma$ (divisor $N$ , plus ridge times the identity), and the cross-covariance $\boldsymbol\sigma_{xy}$ . Return a function erase(Z) that applies $r(\mathbf{x}) = \mathbf{x} - \boldsymbol\sigma_{xy}\,\boldsymbol\sigma_{xy}^\top\boldsymbol\Sigma^{-1}(\mathbf{x} - \boldsymbol\mu)/(\boldsymbol\sigma_{xy}^\top\boldsymbol\Sigma^{-1}\boldsymbol\sigma_{xy})$ to every row of a new array Z, using those fitted statistics. The tests check zero cross-covariance, a fresh probe at chance, an unchanged mean, a smaller change than orthogonal mean projection, and the formula on new data.

import numpy as np

def leace_binary(X, y, ridge=1e-9):
    raise NotImplementedError

Tests, hints, verified solutions, and AI help, with Pro at $7.99 a month.

What does erasure prove? Erasing a concept and then running the rest of the model is an intervention, so it climbs the ladder to the causal rung. If the model's behavior degrades on exactly the inputs where the concept matters, the model was using it. §23.4 develops this kind of evidence with patching, and erasing a direction at every position becomes the directional ablation of §23.7.

Erasure has three limits. First, linear erasure removes only linearly available information, as the four points showed. A nonlinear probe may still succeed unless the two class-conditional distributions become identical. They do for shared-covariance Gaussians. An affine $r$ maps each class to a Gaussian, makes the two means equal, and keeps the two covariances equal, so the two classes end up with the same distribution. Second, a later layer may recompute an erased concept from other features. Third, an erased activation may lie off the distribution the model was trained on, so a change in behavior can come from the damage rather than from the missing concept.

In an interview, expect to be asked how LEACE differs from projecting out a probe's weights, and why the answer involves whitening. In practice, erasure is how a probing claim becomes a claim about use.

Why linear probes work so often

Why should a linear probe work at all? A network is free to store a concept in any curved, tangled way it likes. The linear representation hypothesis is the claim that many concepts a network uses are carried as directions in its activation space, so that how strongly a concept is present is how far the activation reaches along its direction. §1.5 found the first hint in word vectors. In the analogy $\mathbf{v}_{\text{king}} - \mathbf{v}_{\text{man}} + \mathbf{v}_{\text{woman}} \approx \mathbf{v}_{\text{queen}}$ , the relation male to female is one direction. §1.6 warned how easily two-dimensional pictures of such geometry mislead.

At the very end of a language model, linearity is exact. The logits are $z_v = \mathbf{h}\cdot\mathbf{u}_v$ . Here $\mathbf{h} = \operatorname{LN}_f(\mathbf{x}^{(L)}_t)$ is the normalized last stream, and $\mathbf{u}_v = (\mathbf{W}_U)_{:,v}$ is the column of the unembedding matrix for token $v$ (with no output bias, as in GPT-2). For two tokens $a$ and $b$ the softmax's shared denominator cancels, so

\log\frac{p(a)}{p(b)} = z_a - z_b = \mathbf{h}\cdot(\mathbf{u}_a - \mathbf{u}_b).

If $z_a = 3.1$ and $z_b = 1.1$ , the model puts $e^{2} \approx 7.4$ times as much probability on $a$ as on $b$ , whatever the other logits are. So, conditioned on the next token being $a$ or $b$ , the log-odds of $a$ are a linear probe with weight $\mathbf{u}_a - \mathbf{u}_b$ . Every binary choice between tokens is linearly readable from the last layer, by construction. The hypothesis is that concepts are linear in the middle of the network too, where nothing forces it.

"Concept is a direction" can mean three different things, and Park, Choe, and Veitch (2023) separated them. One needs a term first. Two concepts are causally separable when each can be changed without changing the other. Gender and language are causally separable. Changing "king" to "queen" changes the gender and keeps the language, and changing "king" to "roi" changes the language and keeps the gender.

📐 The linear representation hypothesis, three versions

Let $W$ be a binary concept (male to female, English to French, false to true), and let $(Y(0), Y(1))$ range over counterfactual pairs of output words that differ only in $W$ , such as ("king", "queen"). Following Park, Choe, and Veitch, vectors here are columns.

Unembedding version. There is a vector $\bar{\boldsymbol\gamma}_W$ such that $\mathbf{u}_{Y(1)} - \mathbf{u}_{Y(0)}$ is a positive multiple of $\bar{\boldsymbol\gamma}_W$ for (almost) every counterfactual pair.
Embedding version. There is a vector $\bar{\boldsymbol\lambda}_W$ such that adding a positive multiple of it to a context's representation raises the probability of $W = 1$ and leaves unchanged the probability of every concept that is causally separable from $W$ .
Probing version. There is a direction $\mathbf{r}$ such that $W$ is linearly decodable from the representation along $\mathbf{r}$ .

The three versions are different claims. The unembedding version is about outputs. The embedding version is about what an edit to the stream does. The probing version is about what a classifier can read. Park et al. show that the unembedding version yields a linear probe, on the logit scale, for each counterfactual pair, which generalizes the last-layer identity above. They also show that the embedding version is the property that steering relies on, the subject of §23.7. So a probing result alone tests only the third version, which says nothing about outputs or edits.

🔬 The causal inner product

Which inner product should measure angles between concept directions? Park et al. point out that Euclidean angles between unembedding vectors carry no meaning. The model's outputs do not change if the representation space is transformed by an invertible matrix and the unembeddings by its inverse transpose, and such a transformation changes every angle. So they define a causal inner product as one under which causally separable concepts are orthogonal. They show that

\langle\mathbf{u}, \mathbf{v}\rangle_{\mathrm{C}} = \mathbf{u}^\top\operatorname{Cov}(\mathbf{u}_\nu)^{-1}\mathbf{v}

is one such choice, where $\mathbf{u}_\nu$ is the unembedding vector of a token $\nu$ drawn uniformly from the vocabulary. (The paper writes $\gamma$ for unembedding vectors, a letter this book keeps for the norm's gain, and its argument leaves a positive diagonal rescaling undetermined.) Under this inner product the two representations are tied together, $\bar{\boldsymbol\lambda}_W = \operatorname{Cov}(\mathbf{u}_\nu)^{-1}\bar{\boldsymbol\gamma}_W$ . They confirmed the predictions on LLaMA-2 7B with concepts built from lists of word pairs.

Notice the shape of that formula. An inverse covariance turned the mean difference into the LDA reader, it turned $\boldsymbol\sigma_{xy}$ into LEACE's reader, and here it turns an output direction into an input direction. Whitening is how the written and read forms of the same concept are related.

Whether real networks are linear in the middle is an empirical question. The evidence for it is strong, and the clearest case is Othello.

✏️ Othello-GPT, a board read in the wrong basis

Li et al. (2022) trained an 8-layer GPT (8 heads, $d = 512$ ) on 20 million synthetic Othello games, with a vocabulary of the 60 playable squares, to predict legal moves. They probed every layer for the state of each of the 64 squares, black, white, or empty. Linear probes never got below about 20 percent error (20.4 at best, against 26.7 to 28.9 for the same probes on a randomly initialized network). Probes with one hidden layer reached 1.7 percent error at layer 7, while on the randomized network they stayed near 26. Editing the activations by gradient steps on these nonlinear probes changed the model's predicted legal moves as if the board had been edited. So the board looked real, and nonlinear.

Nanda, Lee, and Wattenberg (2023) changed the variables. The model plays both sides, so they labeled each square relative to the player about to move, as mine, theirs, or empty. A linear probe then reached 99.4 to 99.6 percent accuracy in the later layers, against about 75 percent for a linear probe on black, white, and empty. Adding the probe's "mine" or "empty" direction to the residual stream flipped or erased squares, and the model's legal-move predictions followed the edited board almost perfectly. The edits left about 0.1 wrong moves on average for flips and 0.02 for erasures, against about 2.7 with no edit. The representation was linear all along. Li et al. had probed in the wrong basis.

Othello-GPT has become the standard example in the debate over whether sequence models trained only to predict build internal models of the world that produced their data. A probe alone could not settle it. The interventions did.

The same pattern appears in models trained on text. Marks and Tegmark's truth directions generalize across datasets of true and false statements, and they flip the model's judgments when added. Gurnee and Tegmark (2023) fit ridge-regression probes on the last token of the names of places, people, and events in Llama-2 7B, 13B, and 70B. In Llama-2-70B at 60 percent of its depth, they recovered the latitude and longitude of world places with out-of-sample $R^2 = 0.911$ , the fraction of the variance on held-out names that the probe explains, and the death years of historical figures with $R^2 = 0.835$ . Probes with a hidden layer added little (0.926 for world places). Tigges et al. (2023) found one sentiment direction in GPT-2 and Pythia models. Ablating it removed 76 percent of the above-chance zero-shot accuracy on the Stanford Sentiment Treebank.

There are also clear counterexamples. Engels et al. (2024) used sparse autoencoders, dictionaries of sparse features learned from activations (§23.6), to search GPT-2 and Mistral 7B. They found features that are irreducibly two-dimensional, the days of the week and the months of the year arranged on circles. Interventions in Mistral 7B and Llama 3 8B showed that the circles are what the models use for modular arithmetic on days and months. A circle cannot be one direction. A linear probe can still read "is it Tuesday" from it, but the variable the model computes with, the position on the circle, lives in a plane. Othello's colors show a subtler failure. A concept can be linear in one choice of variables and nonlinear in another, and the hypothesis gives no rule for finding the right variables. In practice, finding them is most of the work, and a failed linear probe is not proof that a concept is absent.

⚠️ Five ways a probing result misleads

Leakage. The split shares documents, templates, or near-duplicate items, or the test set's statistics were used to standardize. The probe recognizes items instead of reading the property.
A proxy. The label correlates with something easier, such as sentence length, a token at a fixed position, or the prompt template, and the probe reads that. Test on data where the proxy is broken.
Capacity. An expressive probe learns the task itself. Report a control task or description length, and prefer the simplest probe that works.
The best of many. Reporting the best of dozens of layers, positions, and seeds inflates the result. The statistics subsection below puts a number on it.
Decodable, not used. Even a perfect, selective, generalizing probe is correlational. Only an intervention shows use.

An interviewer will ask you to state the hypothesis and to give one piece of evidence and one counterexample. The strongest answer names the three versions, uses Othello-GPT for the evidence and the circular days for the counterexample, and says which version each one tests.

Unsupervised probes find what is salient, not what is true

Every probe so far needed labels. Labels are exactly what we lack when we want to know what a model believes, rather than what we believe. Burns et al. (2022) proposed finding truth without labels by using its logical structure. A statement and its negation cannot both be true. So turn each yes-or-no question $q_i$ into a contrast pair of two texts, $x_i^+$ ending in "Yes" and $x_i^-$ ending in "No". Exactly one of the two is true, so a probe for truth should give them probabilities that add to one. The method searches for a probe with that property, and is called contrast-consistent search, CCS for short.

📐 Contrast-consistent search (CCS)

Let $\boldsymbol\phi(x) \in \R^d$ be a model's activation on text $x$ . Normalize each side of the pairs separately, $\tilde{\boldsymbol\phi}(x_i^+) = \boldsymbol\phi(x_i^+) - \boldsymbol\mu^+$ and $\tilde{\boldsymbol\phi}(x_i^-) = \boldsymbol\phi(x_i^-) - \boldsymbol\mu^-$ , where $\boldsymbol\mu^\pm$ is the mean over the $n$ questions (scaling by each side's standard deviation is optional). A probe $p(x) = \sigma(\boldsymbol\theta^\top\tilde{\boldsymbol\phi}(x) + b)$ is trained to minimize

\mathcal{L}_{\text{CCS}} = \frac1n\sum_{i=1}^n\Big[\big(p(x_i^+) - (1 - p(x_i^-))\big)^2 + \min\big(p(x_i^+), p(x_i^-)\big)^2\Big],

and predicts $\tilde p(q_i) = \tfrac12\big(p(x_i^+) + 1 - p(x_i^-)\big)$ .

In plain words, the loss has two terms. The first is consistency. It asks a statement and its negation to have probabilities that add to one. Alone it is minimized by $p = 0.5$ everywhere, so the second term, confidence, asks the probe to be sure about one of the two. For a pair with $p(x^+) = 0.9$ and $p(x^-) = 0.2$ , the consistency term is $(0.9 - 0.8)^2 = 0.01$ , the confidence term is $0.2^2 = 0.04$ , and the pair's loss is 0.05.

The per-side normalization matters. The most salient difference within each pair is that one text ends in "Yes" and the other in "No". A probe that read only that would satisfy both terms perfectly, and subtracting each side's own mean removes it. Across six models and ten question-answering datasets, CCS beat zero-shot prompting by 4 percent on average, and it stayed accurate when prompts pushed the models toward wrong answers.

Farquhar et al. (2023) then showed that the loss cannot tell truth from anything else.

Proof. The consistency term is $(h - (1 - (1 - h)))^2 = 0$ , and the confidence term is $\min(h, 1 - h)^2 = 0$ because one of $h$ and $1 - h$ is 0. Both terms are nonnegative, so 0 is the minimum. ∎

The theorem says the objective is underspecified. It does not say CCS will find a bad feature, since a linear probe on real activations can express only some of these $h$ . Their experiments showed that it often does. In one, they appended one of two random words, "Banana" or "Shed", to movie-review prompts, according to a random split of the questions. In another, they inserted "Alice thinks it is positive" or "negative" into the prompts. Both times CCS learned the distractor instead of the review's sentiment. So did two simpler unsupervised baselines, one that thresholds the top principal component of the activations and one that splits them into two clusters with k-means. The probe finds whatever binary feature flips most saliently between the two members of a pair. The research exercise reproduces this.

Research

Contrast-consistent search and its failure

Implement three functions.

ccs_loss(p_pos, p_neg) returns the CCS loss for two arrays of probabilities, one entry per pair.
normalize_pairs(Phi_pos, Phi_neg) centers each side of the pairs with its own mean.
fit_ccs(Xp, Xn, steps, lr, seed) fits $(\boldsymbol\theta, b)$ by full-batch gradient descent on the CCS loss and returns (theta, b). Start from theta = rng.normal(size=d) / np.sqrt(d) with rng = np.random.default_rng(seed), and b = 0. Derive the gradient by hand. When $p^+ = p^-$ exactly, let the confidence term act on $p^+$ .

The setup builds contrast pairs whose activations contain three directions. One is "ends in Yes", one is truth, and one is a distractor that also flips sign between the two members of a pair. It also provides best_of_restarts, which keeps the lowest-loss fit over several seeds. The last test reproduces the finding of Farquhar et al. When the distractor is more salient than truth, CCS finds the distractor.

import numpy as np

def ccs_loss(p_pos, p_neg):
    raise NotImplementedError

def normalize_pairs(Phi_pos, Phi_neg):
    raise NotImplementedError

Tests, hints, verified solutions, and AI help, with Pro at $7.99 a month.

There is a deeper problem. A model may represent several candidate truths at once, its own belief, what a character in the text believes, and what a typical writer would say. All of them are consistent, so a consistency loss cannot tell them apart. That is why interviewers ask what CCS finds, and why the honest answer is "the most salient consistent feature". In practice, an unsupervised probe is a way to find candidate directions, which then need labeled checks and interventions like any other probe.

Probes as safety monitors

Probes are cheap, one dot product per token, which makes them attractive as safety monitors that read a deployed model's activations and raise a flag. §19.6 described linear probes, built from generic contrast pairs, that flagged when deliberately backdoored "sleeper agent" models were about to act on their hidden trigger. §19.8 described semantic entropy probes, which predict from the hidden states of one generation whether the model is making an answer up.

Two cautions come with every such number. First, set the threshold on the data the monitor will actually see, at a fixed false-positive rate. §0.11 explains why a high AUROC can still mean many false alarms when the behavior is rare. Second, a monitor can be evaded once something optimizes against it. Bailey et al. (2024) showed that probes, sparse autoencoders, and out-of-distribution detectors on activations are all vulnerable to obfuscated activations, internal states that keep a harmful behavior while looking harmless to the monitor. Against probes trained to detect harmful requests, their attacks often cut recall from 100 percent to 0 while keeping a 90 percent jailbreak success rate.

The same holds for training. A model trained with a probe in its loss can learn to satisfy the probe instead of changing its behavior. A probe is a good monitor as long as nothing has been trained to beat it. In practice, that means a probe monitor is kept out of the training loop and its threshold is checked on fresh traffic. An interviewer will ask how you would validate one, and the last interview question below gives an answer. §23.6 returns to monitors when it compares probes with sparse-autoencoder features on the same detection task.

Doing the statistics honestly

A probing study usually tests many sites. Suppose you probe 32 layers at 3 token positions with 5 seeds, 480 probes in all, and report the best. Two questions decide whether the result means anything. How many test examples does one accuracy need, and how much does choosing the best of 480 inflate it? The cell answers both, the second by simulating 480 independent probes that are all at chance.

Python · cell 8

✏️ How many test items, and how much the best of 480 is inflated

To pin an accuracy near 90 percent to within plus or minus 2 points with 95 percent confidence, the normal interval of §0.11 needs $n \ge 1.96^2\cdot 0.9\cdot 0.1/0.02^2 \approx 864.4$ , so 865 test examples.

A Bonferroni correction tests each of $m$ comparisons at level $0.05/m$ , so that the chance of even one false alarm stays below 0.05. With 480 probes that is $p \lt 0.05/480 \approx 1.0\times 10^{-4}$ per probe, which needs $|z| \gt 3.88$ instead of the usual 1.96.

The best of 480 chance-level probes, each scored on 500 test items, has a median accuracy of 0.566 and exceeds 0.582 one time in twenty. Nothing was learned, yet the best number looks like a modest finding.

Real probes on neighboring layers are correlated, which shrinks the inflation but does not remove it. There are two remedies, and they can be combined. Pick the layer and position on a validation split and report the test accuracy of that one choice. Or correct for the number of comparisons. Interviewers like this question because it separates people who have run probing studies from people who have read about them.

Probing every layer in PyTorch

In a real model the activations come from forward hooks. A hook is a function PyTorch calls with a module's input or output during the forward pass, so it can record activations without changing the model (§0.9). The block below trains a two-layer Transformer on sequences of 12 tokens drawn from 6. At every position it must say whether more 0s than 1s have appeared so far. Then it probes $\mathbf{x}^{(0)}$ , $\mathbf{x}^{(1)}$ , and $\mathbf{x}^{(2)}$ for a related variable that the training labels never mention, whether a 0 occurred at an earlier position. It does this for the trained model and for an untrained copy.

🔥 In PyTorchfor a machine with PyTorch

The trained model's probes read the variable with accuracies 0.740, 0.966, and 0.962 at the three sites, against a majority rate of 0.539. Two baselines make those numbers interpretable. At $\mathbf{x}^{(0)}$ the probe already beats the majority rate, because the position embedding is there and later positions are more likely to have seen a 0. Both models score 0.740 there, which is what the best rule that sees only the position scores on these test sequences. A position with $t$ tokens before it has seen a 0 with probability $1 - (5/6)^t$ , which first passes one half at $t = 4$ , so that rule says "yes" from the fifth position on. And the untrained model reaches 0.893 and 0.913 after its blocks, because even random attention averages the earlier tokens' embeddings into each position. The trained model adds about five to seven points on top. Without the untrained baseline, "layer 1 represents whether a 0 occurred" would have credited learning with what the architecture does for free. In practice, every layer-by-layer probing plot should carry the same curve for an untrained copy of the model, and an interviewer who sees a plot without it will ask for it.

Choosing a method

method	what it measures	strengths	weaknesses
logistic probe	linear decodability at one site	standard, convex, gives a calibrated probability	weight vector is a reading direction, not the concept's direction
difference of means	the direction the classes differ in	no training, best steering vector, matches the write direction	poor classifier when nuisances overlap the concept
LDA (whitened) probe	Bayes-optimal linear reading for shared-covariance Gaussians	closed form, accurate, shrinkage handles $d \gt N$	needs a covariance estimate, same reading-direction caveat
MLP probe	nonlinear decodability	finds information that is there but not linear	can learn the task itself, low selectivity
control task and selectivity	how much of the accuracy is memorization	simple, interpretable	needs labels that belong to word types
description length (MDL)	how easily the property is extracted	works for any labels, stable	more training runs, less familiar
sparse probe	whether a few neurons carry the property	localizes to neurons	assumes neurons are meaningful units, which §23.5 questions
CCS	a consistent binary feature, without labels	needs no labels	finds the most salient flipped feature, which need not be truth
LEACE, INLP, amnesic probing	whether removing the property changes behavior	causal, closed form (LEACE)	linear only, can push activations off distribution

The sparse probe in the table is that of Gurnee et al. (2023), a linear probe allowed to use only $k$ neurons. In Pythia models from 70 million to 6.9 billion parameters, they found early layers representing many features with sparse combinations of neurons, and middle layers with neurons that seem dedicated to single features. It was an early sight of the superposition that §23.5 explains.

🧭 What to remember

A probe is a classifier trained on frozen activations. Train it with the gradient $\frac1N\mathbf{X}^\top(\hat{\mathbf{p}} - \mathbf{y}) + \lambda\mathbf{w}$ , split by the unit of the claim, standardize with training statistics, and report the majority rate, AUROC, an interval, and baselines (layer 0, an untrained model, a control task).
A probe can learn the task itself. Selectivity, task accuracy minus control-task accuracy, exposes it. On ELMo's part-of-speech tags it was 26.0 for a linear probe against 4.5 for an MLP probe, at nearly equal accuracy.
For shared-covariance Gaussians the optimal linear probe is LDA, $\boldsymbol\Sigma_w^{-1}(\boldsymbol\mu_1 - \boldsymbol\mu_0)$ , and the difference of means is optimal only when the mean difference is an eigenvector of $\boldsymbol\Sigma_w$ . The accurate probe is a reading direction, and the difference of means is the writing direction to steer with.
A concept is linearly erased exactly when its cross-covariance with the activations is zero. LEACE does it with the smallest change, subtracting along $\boldsymbol\sigma_{xy}$ what $\boldsymbol\Sigma^{-1}\boldsymbol\sigma_{xy}$ reads, and a nonlinear probe can still recover what it leaves.
The linear representation hypothesis is exact at the last layer for choices between tokens and well supported in the middle (Othello's mine-or-theirs board, truth, space and time, sentiment), with real exceptions such as the circular day and month features.
Unsupervised probes find the most salient consistent feature, not truth. Probes used as monitors stay reliable only as long as nothing is optimized against them.
A probe shows that information is decodable, in a basis we chose, not that the model uses it. Next, §23.3 reads the residual stream in the model's own output basis, with the unembedding as the probe.

Hewitt and Liang (2019). Designing and Interpreting Probes with Control Tasks. Control tasks and selectivity, with the ELMo experiments that showed how much of a probe's accuracy can be its own.
Belinkov (2021). Probing Classifiers: Promises, Shortcomings, and Advances. The survey of probing in NLP, its methodological debates, and the move from correlation to intervention.
Belrose et al. (2023). LEACE: Perfect linear concept erasure in closed form. The equivalence of linear guardedness and zero cross-covariance, the optimal eraser for every inner-product norm, and concept scrubbing.
Park, Choe, and Veitch (2023). The Linear Representation Hypothesis and the Geometry of Large Language Models. The unembedding and embedding versions of the hypothesis, their links to probing and steering, and the causal inner product.
Li et al. (2022). Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task. Othello-GPT, nonlinear board probes, and interventions that edit the board the model plays on.
Nanda, Lee, and Wattenberg (2023). Emergent Linear Representations in World Models of Self-Supervised Sequence Models. The same board, linear once squares are labeled relative to the player to move.
Marks and Tegmark (2023). The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets. Truth directions across datasets, mass-mean probes, and why the most accurate probe is not the most causal direction.
Farquhar et al. (2023). Challenges with unsupervised LLM knowledge discovery. Why contrast-consistent search finds the most salient feature, with the theorem and the experiments.

Made for larger screens

Probing

A probe is a classifier on frozen activations

Training a linear probe

A logistic probe from scratch

Control tasks show how much a probe memorizes

Control tasks and selectivity

Three ways to choose the probe's direction

The most accurate probe is not the direction the model writes

Difference of means, LDA, and their thresholds

Erasing a concept to test whether it is used

Erase a concept in closed form

Why linear probes work so often

Unsupervised probes find what is salient, not what is true

Contrast-consistent search and its failure

Probes as safety monitors

Doing the statistics honestly

Probing every layer in PyTorch

Choosing a method

Read more

Watch next