Superposition is how a neural network stores more features than it has dimensions. A feature is a property of the input that the network keeps track of, such as "this text is Korean" or "this image contains a cat face". The network gives each feature its own direction in a layer's space of activations. When there are more features than dimensions, those directions cannot all be at right angles, so reading out one feature picks up a little of the others. That leak costs little when features are rarely active at the same time, and a ReLU with a small negative bias after the readout removes most of it.

The idea explains a puzzle that blocks the simplest way of reading a network. Pick one neuron in the MLP of a small language model and collect the text that makes it fire hardest. Bricken et al. (2023) did this for a one-layer Transformer whose MLP has 512 neurons. One neuron responds to academic citations, English dialogue, HTTP requests, and Korean text. Vision models do the same. Neuron 4e:55 of the image classifier InceptionV1 fires for cat faces, cat legs, and the fronts of cars (Olah et al. 2020). Images synthesized to excite it show that it looks for each of these separately (eyes and whiskers, furry legs, shiny car fronts), not for one property they share. A neuron that responds to several unrelated things is called polysemantic.

Polysemantic neurons have already caused trouble in this chapter. In §23.1, neuron 221 in the last layer of BERT seemed to mean song titles on one dataset and dates on another, because its top-activating sentences changed with the data. In §23.4, circuits through attention heads came out clean, while circuits through MLP neurons were hard to describe at all. A unit that mixes four unrelated concepts looks like a bug. This section shows that it is a compression scheme, and that a layer that compresses this way must have polysemantic neurons.

Here is the plan. We define features and superposition, and ask which layers have meaningful coordinates. We prove that a linear model cannot compress features this way. Then we study the smallest model that can, a toy that squeezes $n$ features through $m \lt n$ dimensions and decodes them with a ReLU (Elhage et al. 2022). We solve its two-feature case exactly, train it and watch features arrange themselves into opposite pairs and pentagons, and measure how large the leak is at the width of a real model. We connect the story to compressed sensing, the mathematics of recovering a sparse signal from few measurements. We end with the evidence that real models are in superposition. Interviews for interpretability roles ask about each of these steps, most often the two-feature derivation and the link to sparse autoencoders.

How strong is the evidence for all this? §23.1 ranked interpretability evidence on a ladder, from association (some quantity tracks a concept) up to mechanism (a full account confirmed by the interventions it predicts). This section is a theory of how features are stored. In the toy model, where we know the true features, it is confirmed at the level of mechanism. In real models it is supported only indirectly.

As in the rest of this chapter, vectors are rows and a matrix acts on the right, $\mathbf{x}\mathbf{W}$. The papers we follow write columns, and we translate where it matters.

Features, polysemantic neurons, and superposition

§23.2 stated the linear representation hypothesis. It says that a concept a model uses is a direction in its space of activations. How strongly the concept is present is how far the activation points along that direction. Superposition takes this as given and asks a counting question. If a layer has $m$ dimensions, how many such directions can it use?

Start with a tiny case. A layer has two neurons and must represent three features. Give feature 1 the direction $(1, 0)$, feature 2 the direction $(0, 1)$, and feature 3 the direction $(0.71, 0.71)$, halfway between them. If only feature 3 is present, with strength 1, the activation is $(0.71, 0.71)$, and both neurons fire. So neuron 1 responds to features 1 and 3, and neuron 2 responds to features 2 and 3. Packing three features into two neurons made both neurons polysemantic.

It also made the features leak into each other. Feature 3's direction has a cosine of 0.71 with each of the others. So if we read feature 1 by projecting onto $(1, 0)$, we get 0.71 when only feature 3 is on.

In plain words, features are the variables the model computes with, neurons are the places where the model's nonlinearity looks, and superposition means there are more variables than places. In the tiny case, $n = 3$, $m = 2$, and $\mathbf{W}$ has rows $(1, 0)$, $(0, 1)$, and $(0.71, 0.71)$. Each column of $\mathbf{W}$ has two nonzero entries, so each neuron is polysemantic.

Nobody has an agreed definition of a feature, and the definition above sidesteps the question. Elhage et al. (2022) list three candidates. A feature could be any function of the input, but then "cat or car" counts as much as "cat". It could be a property a human can name, but that excludes concepts we have no words for. Or it could be a property that a large enough network would reliably give a neuron of its own, which matches the intuition but is hard to test. Most work, including this section, uses the decomposition in item 1 and stays agnostic about the rest.

Superposition forces polysemantic neurons by a counting argument.

Proof. Suppose every neuron is monosemantic, so each of the $m$ columns of $\mathbf{W}$ has at most one nonzero entry. Then $\mathbf{W}$ has at most $m$ nonzero entries, so at most $m$ of its $n$ rows can be nonzero. Since $n \gt m$, some feature has $\mathbf{w}_i = \mathbf{0}$, a contradiction. ∎

The converse fails. A layer with $n \le m$ features can still have polysemantic neurons, if the feature directions are rotated away from the neuron axes. Whether that can happen depends on whether the neuron axes are special at all, which is the next question. In practice, the theorem is why counting neurons never counts features. An interviewer who asks "why are neurons polysemantic?" wants this counting argument first.

Privileged and non-privileged bases

When we say "neuron 4e:55" we read one coordinate of a layer. That makes sense only if the coordinates mean something. §5.1 showed with a small example that in the residual stream, the running vector that every Transformer block reads from and adds its output to, they do not. Take a stream $(3, 1)$ that a sublayer reads with the column $(1, 2)^\top$, so it reads $3 + 2 = 5$. Rotate the stream by a quarter turn to $(-1, 3)$, and rotate the read column to match, $(-2, 1)^\top$. The sublayer still reads $2 + 3 = 5$. Nothing downstream can tell that the coordinates changed.

A ReLU applied before the read would notice. The ReLU, $\max(\cdot, 0)$ applied to each coordinate, leaves $(3, 1)$ unchanged, so the original stream still reads 5. It turns the rotated stream $(-1, 3)$ into $(0, 3)$, which reads 3. An operation that looks at each coordinate on its own is what makes coordinates special.

Two layers of a Transformer behave very differently under this test. Recall that an orthogonal matrix $\mathbf{R}$ satisfies $\mathbf{R}\mathbf{R}^\top = \mathbf{I}$, so it is a rotation or reflection and preserves lengths.

In plain words, the residual stream has no preferred axes. Anything the model does in one basis it can do equally well in a rotated one. An MLP's hidden layer has axes that the ReLU looks at one by one, and the only harmless changes are relabeling the neurons and rescaling each by a positive number. So "neuron $j$" is a meaningful unit and "residual coordinate $j$" is not, at least in principle. Both proofs are short and sit in the derivations below.

Trained models are less symmetric than the theorem suggests. Large Transformers have a few residual coordinates whose values are up to 20 times larger than any other. These outlier features are what make 8-bit quantization of activations hard (Dettmers et al. 2022, §20.2). Elhage, Lasenby, and Olah (2023) asked where they come from, with a clean test.

The test treats the $d$ coordinates of one token's activation as $d$ samples and computes their kurtosis. Kurtosis is the average fourth power of the deviations from the mean, divided by the squared variance. A Gaussian has kurtosis 3, and heavy tails push it higher. If features pointed in random directions, each coordinate would be a sum of many random projections, so it would look Gaussian and the kurtosis would be about 3. The cell runs the test on made-up activations in which 30 of 768 coordinates are 8 times larger than the rest, and then on the same activations after a random rotation.

The stored basis gives a kurtosis of about 37, and the rotated one gives almost exactly 3. The rotation mixes every outlier coordinate into all 768 coordinates, so the heavy tail disappears. A kurtosis well above 3 therefore means that the activations line up with the basis they are stored in. The converse does not hold. If the feature values were themselves Gaussian, basis-aligned features would also give 3.

In a 200-million-parameter model with $d = 1{,}280$, Elhage, Lasenby, and Olah found 20 to 60 outlier coordinates in a typical layer and a kurtosis far above 3. After a fixed random rotation of the same activations, the kurtosis fell to almost exactly 3, as in the cell. So the stream really does favor its own coordinates. Their provisional culprit is the optimizer. Adam gives every parameter its own step size, dividing by the root of that parameter's running average of squared gradients (§0.6). That per-coordinate scaling is the one operation in training that depends on the basis. This is an empirical finding about training, not a property of the architecture.

In practice this decides where to look for meaning. Neurons are worth reading in MLPs, never in the residual stream. Outlier coordinates in the stream are a training artifact to handle in quantization, not a concept. An interviewer will ask whether the residual stream has a privileged basis, and why outlier dimensions appear anyway.

Room for more features than dimensions

Why is $n \gt m$ possible at all? §1.3 proved the geometric fact. In $m$ dimensions, the cosine between two random unit directions has mean 0 and variance exactly $1/m$, so it is typically about $\pm 1/\sqrt m$. At $m = 768$ that is $\pm 0.036$. And there are at least $e^{m\varepsilon^2/4}$ directions whose pairwise cosines are all smaller than $\varepsilon$ in size. At $m = 4{,}096$ and $\varepsilon = 0.1$ that is about 28,000 directions, seven times $m$, and the count grows exponentially with $m$. So a layer can give far more than $m$ features nearly orthogonal directions.

The catch is the leak. Reading feature $i$ back by projecting onto its direction picks up a little of every other active feature. This leak is called interference, and the question is whether the model can live with it.

A linear model cannot do anything clever about interference. The proof is worth seeing, because it shows exactly what a nonlinearity buys. Here is the idea on one feature. Take a feature that is zero 90 percent of the time and otherwise uniform on $[0, 1]$. Its mean is $0.05$, and its variance is $0.1/3 - 0.05^2 \approx 0.0308$. A Gaussian feature with the same mean and variance is never zero. Yet any linear reconstruction has exactly the same expected squared error on both, because that error depends only on means and covariances. The sparsity is invisible to it.

The proof uses the Frobenius norm $\lVert\mathbf{M}\rVert_F$, the square root of the sum of the squares of all entries of a matrix. It also uses the Eckart-Young theorem of §0.10, which says that the best rank-$m$ approximation of a matrix in this norm keeps its $m$ largest singular values.

In plain words, a linear model sees only means and covariances, and a sparse feature has the same mean and variance as some dense feature. So however sparse the features are, a linear model represents at most $m$ of them, the most important ones. That is what principal component analysis does, keeping the $m$ directions of largest variance. To exploit sparsity the model must be nonlinear. An interviewer who asks "what exactly does the ReLU add?" is looking for this argument.

The toy model of superposition

Elhage et al. (2022) built the smallest model that can exploit sparsity. It compresses $n$ features into $m \lt n$ dimensions with a linear map. It decompresses with the transpose of the same map, adds a bias, and applies a ReLU.

In plain words, each feature gets an arrow $\mathbf{W}_i$ in a small space. The hidden vector $\mathbf{h}$ is the sum of the arrows, each scaled by its feature's value. Each feature is read back by projecting onto its own arrow, adding a bias, and clipping at zero. Entry $(i, j)$ of the $n\times n$ matrix $\mathbf{W}\mathbf{W}^\top$ is the dot product $\mathbf{W}_i\cdot\mathbf{W}_j$, so the readout of feature $i$ is

That is the signal, plus interference from every other active feature, plus a bias. The importance $I_i$ stands in for how much the downstream loss cares about feature $i$.

Two details matter. First, the hidden layer has no nonlinearity, so by the section on privileged bases its basis is not privileged. Replacing $\mathbf{W}$ by $\mathbf{W}\mathbf{R}$ for an orthogonal $\mathbf{R} \in \R^{m\times m}$ leaves $\mathbf{W}\mathbf{W}^\top$, and hence the model, unchanged. That is why we always look at $\mathbf{W}\mathbf{W}^\top$ and at angles between rows, never at single hidden coordinates. Second, the bias lets the model set a feature it does not represent to that feature's mean, and, as we will see, subtract small amounts of interference.

The ReLU is what makes sparsity useful, and the smallest example shows why.

The two arrows point in opposite directions, so we call this arrangement an antipodal pair. The cell reproduces the three cases, with and without the ReLU.

The hand example generalizes into one formula that explains most of this section. When features are very sparse, the inputs with two or more active features are rare, and the loss is dominated by inputs with exactly one feature on. On those inputs only positive overlaps cost anything.

Proof. With only $x_j$ nonzero, the readout formula gives $x'_j = \operatorname{ReLU}(\lVert\mathbf{W}_j\rVert^2 x_j + b_j)$ and $x'_i = \operatorname{ReLU}((\mathbf{W}_i\cdot\mathbf{W}_j)x_j + b_i)$ for $i \ne j$. The true value of every inactive feature is 0, so its error is $x'^2_i$. ∎

The input with every feature off costs $\sum_i I_i\operatorname{ReLU}(b_i)^2$, which is zero whenever no bias is positive. So in the sparse limit the loss is a weighted sum of these one-feature-on errors.

In plain words, the first term rewards representing feature $j$, which Elhage et al. (2022) call the feature benefit. The second term charges for interference, but a negative overlap $\mathbf{W}_i\cdot\mathbf{W}_j \lt 0$ costs nothing, because the ReLU clips it to zero. A negative bias $b_i$ also cancels small positive overlaps. A linear model has no ReLU, so it pays for overlaps of both signs. This is why sparse features push a ReLU model toward arrangements whose overlaps are mostly negative, such as the antipodal pair and the pentagon we will meet below.

Reading a toy model's weights

Before training anything, we need ways to read a trained $\mathbf{W}$ that do not depend on the arbitrary hidden basis. The first is $\mathbf{W}\mathbf{W}^\top$ itself. Its diagonal holds the squared lengths $\lVert\mathbf{W}_i\rVert^2$, which say whether a feature is represented at all (about 1 if it is, about 0 if it was dropped). Its off-diagonal entries are the interference weights. The second is a measure of how much of a dimension each feature gets.

Take the antipodal pair again, with rows $1$ and $-1$. Feature 1 has squared length 1. The total squared weight lying along its direction is $1^2$ from itself plus $(-1)^2$ from feature 2, which is 2. Feature 1 owns half of what lies on its axis, so we say it gets half a dimension. A feature alone on its axis would own all of it, one full dimension.

In plain words, the numerator says how strongly feature $i$ is represented. The denominator says how much total weight, feature $i$'s own included, lies along feature $i$'s direction. A feature alone on its axis gets $D_i = 1$. Every represented feature has $\lVert\mathbf{W}_i\rVert^2 \approx 1$, so $\lVert\mathbf{W}\rVert_F^2 = \sum_i\lVert\mathbf{W}_i\rVert^2$ roughly counts the represented features. So $D^\star$ is the number of dimensions divided by the number of features stored. Rotations of the hidden space change neither quantity. Scaling every row by the same factor leaves $D_i$ unchanged but changes $D^\star$.

How do these numbers add up across features? The answer needs one more term. A set of directions is a tight frame if it spreads evenly over the space, so that every unit direction $\mathbf{u}$ receives the same total squared weight, $\sum_j(\mathbf{u}\cdot\mathbf{W}_j)^2 = c$. In matrix form that is $\mathbf{W}^\top\mathbf{W} = c\,\mathbf{I}_m$. An antipodal pair, a regular pentagon in the plane, and a regular tetrahedron in three dimensions are all tight frames.

In plain words, a model has $m$ dimensions to hand out and never hands out more. When features are packed as evenly as possible, each gets an equal share.

import numpy as np

def interference(W):
    raise NotImplementedError

def feature_dims(W):
    raise NotImplementedError

Two features in one dimension, solved exactly

The hand example showed that the antipodal pair is perfect when at most one feature is active. When is that worth the cost of the times both are on? With two features and one hidden dimension we can answer exactly.

Let feature 1 have importance $I_1 = 1$ and feature 2 importance $I_2 = I$. Relabeling the features swaps their roles, so we may take $I \le 1$, which makes feature 2 the less important one. The two features are independent, each 0 with probability $S$ and otherwise uniform on $[0, 1]$, with density $p = 1 - S$.

We need three moments of one feature. With probability $p$ it is a $\operatorname{Uniform}[0, 1]$ variable $U$, with $\mathbb{E}U = \tfrac12$ and $\mathbb{E}U^2 = \tfrac13$, and otherwise it is 0. So

Now compare two candidate solutions.

Dedicated. Give feature 1 the dimension, $\mathbf{W} = (1, 0)^\top$, with $b_1 = 0$. Then $x'_1 = \operatorname{ReLU}(x_1) = x_1$ is always exact. Feature 2 is not represented, so $x'_2 = \operatorname{ReLU}(b_2)$, a constant. The best constant under squared error is the mean, $b_2 = \mathbb{E}[x_2] = p/2$, and its error is the variance. The loss is

With $b_2 = 0$ instead, the loss would be $I\,\mathbb{E}[x_2^2] = Ip/3$, which is worse. The bias is part of the solution.

Antipodal. Take $\mathbf{W} = (1, -1)^\top$ and $\mathbf{b} = \mathbf{0}$. When at most one feature is on, the output is exact, as in the hand example. Both are on with probability $p^2$. Say $x_1 \ge x_2$. Then $h = x_1 - x_2 \ge 0$, so $x'_1 = x_1 - x_2$ and $x'_2 = 0$. Both errors equal $x_2 = \min(x_1, x_2)$, and the weighted squared error is $(1 + I)\min(x_1, x_2)^2$. The case $x_2 \gt x_1$ is the mirror image.

For two independent uniforms, $\mathbb{P}[\min(U_1, U_2) \gt u] = (1 - u)^2$, so the minimum has density $2(1 - u)$ on $[0, 1]$ and

So the loss is

Antipodal wins when $(1 + I)p^2/6 \lt I(p/3 - p^2/4)$. Divide by $p \gt 0$ and multiply by 12 to get $2(1 + I)p \lt 4I - 3Ip$, that is, $(2 + 5I)p \lt 4I$.

In plain words, store the second feature on top of the first, pointing the other way, whenever the features are sparse enough that they are seldom on together. For equal importance the switch is at density $4/7 \approx 0.57$, that is, $S \gt 3/7$. For $I = 0.5$ it is at $p = 4/9$ ($S \gt 5/9$), and for $I = 0.2$ at $p = 4/15$ ($S \gt 11/15$). The less a feature matters, the sparser it must be to earn a share of the dimension.

The cell checks both formulas by computing the expected loss exactly. It splits on which features are on (both, only one, or neither) and integrates each case on a fine grid. Then it asks a numerical optimizer for the best $(\mathbf{W}, \mathbf{b})$ it can find, starting from each candidate.

The formulas match to four decimals. The optimizer adds a caveat. At $S = 0.5$, $I = 1$ it finds a loss of 0.0763, below both candidates, with $\mathbf{W} \approx (0.97, -0.97)^\top$ and small positive biases $\mathbf{b} \approx (0.10, 0.10)$. When both features are on, the antipodal readout comes out too small, and the positive bias lifts it. That costs a little when a feature is off or alone. The two candidates explain the shape of the answer, and the full optimum refines it.

The change from one solution to the other is abrupt. At $I = 0.32$ the best $W_2$ shrinks from about $-0.88$ at density 0.32 to about $-0.74$ at 0.385, and then jumps to exactly 0 by 0.39. Physics calls this a first-order phase change. The optimal loss is the smaller of two smooth curves, so the loss itself is continuous. Its slope jumps where the curves cross, and the optimal weights jump with it.

The widget draws the whole plane of densities and importances. Its shading shows which of three candidates wins in closed form, keeping feature 1, keeping feature 2, or the antipodal pair. The line is the boundary from the theorem. The dots mark where the best solution an optimizer finds drops a feature. At the ring you drag, the side panel compares the losses and weights of the closed-form winner with those of that best solution.

This derivation is a favorite interview question, because it fits on a whiteboard and tests probability, optimization, and judgment at once. The judgment part is saying that it compares candidate solutions, not all solutions.

import numpy as np

def expected_losses(S, I):
    raise NotImplementedError

def boundary_density(I):
    raise NotImplementedError

Watching superposition form

Two features in one dimension fit in a box of algebra. For more features we train the model, and training needs the gradient. The gradient of the toy model is a classic interview problem, because $\mathbf{W}$ is used twice, once to encode and once to decode, and the chain rule has to collect both uses.

import numpy as np

def toy_loss_and_grads(W, b, X, I):
    raise NotImplementedError

The trainer below runs four models at once, with a leading batch dimension on every array. It uses Adam (§0.6) and keeps the best of the four runs, since small toy models often settle in worse arrangements. Our first experiment is the one in Elhage et al. (2022), five features of equal importance in two dimensions at density 0.05.

Now play with it. The widget trains the same model in your browser, four runs at a time, and replays the training of the best one. Watch features appear as the density falls, and see how unequal importances deform the picture.

Next comes the larger experiment of Elhage et al. (2022), twenty features in five dimensions with importances $I_i = 0.7^i$. Feature 0 matters most, and feature 19 matters about $10^{-3}$ as much. We train one model at each of four densities and show $\mathbf{W}\mathbf{W}^\top$.

At density 1 the model behaves like the linear model of the theorem above. It represents exactly the five most important features, orthogonally ($\mathbf{W}\mathbf{W}^\top$ is a $5\times5$ identity block), with $D^\star = 1.0$. It sets the other features to their means through positive biases. As the density falls, the count of represented features grows to 8, 9, and 10, and $D^\star$ drops to 0.62, 0.55, and 0.48. Off-diagonal blocks appear in $\mathbf{W}\mathbf{W}^\top$. The most negative bias goes from 0 to about $-0.13$, as the model pays to filter interference.

Which features share? Computing $D_i$ for these runs with the functions of Exercise 23.5a answers it. At density 0.3, features 0 and 1 still have $D_i = 1$, while features 2 to 7 have $D_i$ between 0.44 and 0.55, mostly antipodal pairs. At density 0.1 only feature 0 keeps a dimension of its own. At 0.03 all ten represented features have $D_i \approx 1/2$, five antipodal pairs. So the features that enter superposition first are the less important ones. That ordering is the two-feature theorem again. A feature of small importance must be sparse to earn a share of a dimension, and the important features keep a dimension of their own as long as they can.

What about many equal features? Elhage et al. (2022) trained $n = 400$ features into $m = 30$ dimensions and plotted $D^\star$ against sparsity. It does not fall smoothly. It sits at 1, then at $1/2$ over a wide range, and the weights show why, since they are antipodal pairs. Beyond that it keeps falling, and the individual $D_i$ cluster at a few special values.

The PyTorch version below sweeps the density for fifty equal features in five dimensions and prints $D^\star$, so you can see the plateau at one half.

In our run, $D^\star$ is 1.02 at density 1 and 0.89 at 0.66. It is 0.505 and 0.490 at densities 0.43 and 0.29, where exactly ten rows (five antipodal pairs) are represented. It then falls through 0.35, 0.28, and lower as more features crowd in.

The run also shows why $D^\star$ is the right measure when features are dense. At density 1 only two rows have norm above 0.5, yet $D^\star \approx 1$. With equal importances and dense data, one five-dimensional subspace of feature space is nearly as good as any other. So the model spreads its capacity over many features with small weights, rather than choosing five of them. Counting rows assumes that the model singles out individual features, which it need not do when they are dense. $\lVert\mathbf{W}\rVert_F^2$ counts the dimensions in use, whichever directions the model chose.

How much interference, and what the bias does

The toy models are small. How large is interference in a layer the size of a real model's? Suppose $n$ features have random unit directions in $\R^m$. Feature $i$ is off, and $k$ other features are on, with values $a_j$. Feature $i$'s readout before the ReLU is then pure interference, $r_i = \sum_{j}a_j\,\hat{\mathbf{W}}_i\cdot\hat{\mathbf{W}}_j$. It is a sum of $k$ small random cosines, each scaled by its feature's value.

Proof. Each cosine $c_j = \hat{\mathbf{W}}_i\cdot\hat{\mathbf{W}}_j$ has mean 0 by symmetry. Its variance is $1/m$, because the $m$ squared coordinates of a uniform unit vector have equal means that sum to 1 (§1.3). The terms $a_jc_j$ are independent, each with mean $\mathbb{E}a_j\,\mathbb{E}c_j = 0$ and variance $\mathbb{E}a_j^2\,\mathbb{E}c_j^2 = s^2/m$, so the variances add to $ks^2/m$. ∎

For features of value 1, $s = 1$. For the toy's $\operatorname{Uniform}[0, 1]$ values, $s^2 = 1/3$. The cell checks both, and then computes the numbers for the real-scale example below.

This is where the bias earns its place. The ReLU deletes interference that comes out negative. Interference that comes out positive passes through, unless the bias is negative enough to cut it off. A negative bias $b_i = -\tau$ turns the readout into $\operatorname{ReLU}(\text{signal} + \text{interference} - \tau)$. That is a threshold detector, which reports feature $i$ only when its projection clears $\tau$. The cost is that true activations below $\tau$ are lost, and those above are shrunk by $\tau$. The model offsets the shrinkage by lengthening $\mathbf{W}_i$, as the pentagon showed.

In practice, $\sqrt{k/m}$ is the number to carry into a design review. It says how many features a layer can keep active at once before readouts drown. An interviewer may hand you $m$, $n$, and a firing rate and ask for it.

Superposition is compressed sensing with a cheap decoder

The toy model solves a compression problem. The hidden vector $\mathbf{h} = \mathbf{x}\mathbf{W}$ is $m$ linear measurements of an $n$-dimensional sparse vector, and the decoder must recover $\mathbf{x}$ from them. Recovering a sparse signal from few linear measurements is the subject of compressed sensing, a major line of applied mathematics since the mid-2000s.

Without sparsity, recovery is impossible. With $m \lt n$ measurements, there is an $(n - m)$-dimensional set of inputs that measure to zero (the null space), and any two inputs that differ by such a vector look identical. Sparsity changes this. If two $k$-sparse vectors had the same measurements, their difference would be a $2k$-sparse vector in the null space. So every $k$-sparse vector is determined by its measurements exactly when every $2k$ rows of $\mathbf{W}$ (columns, in the usual column notation) are linearly independent, which needs $m \ge 2k$.

Finding the vector by trying every possible set of active features is a combinatorial search. The remarkable fact is that a convex program finds it. Among all vectors with the right measurements, pick the one with the smallest sum of absolute values, the $\ell_1$ norm. This program is called basis pursuit.

How does the toy decoder compare with $\ell_1$ minimization? It is exactly one step of the standard iterative algorithm for it. For nonnegative signals, the penalized form of the problem is $\min_{\mathbf{z}\ge 0}\ \tfrac12\lVert\mathbf{z}\mathbf{W} - \mathbf{h}\rVert^2 + \lambda\sum_i z_i$, where $\lambda \gt 0$ trades fit against sparsity.

Proof. At $\mathbf{z}_0 = \mathbf{0}$ the gradient of $\tfrac12\lVert\mathbf{z}\mathbf{W} - \mathbf{h}\rVert^2$ is $(\mathbf{0}\mathbf{W} - \mathbf{h})\mathbf{W}^\top = -\mathbf{h}\mathbf{W}^\top$, so the gradient step gives $\mathbf{h}\mathbf{W}^\top$. The nonnegative shrinkage subtracts $\lambda$ and clips at 0. ∎

So the toy model decodes by correlating once and thresholding once, with no further iterations. The correlation step leaves every bit of interference in place, and only the threshold removes any of it. Iterating would subtract the interference that the current estimate explains, which is how ISTA converges to the $\ell_1$ solution. Stopping after one step is why the toy needs much sparser features than compressed sensing does. The step size is also off. The standard convergence guarantee for ISTA asks for a step $\eta \le 1/\lVert\mathbf{W}\rVert_2^2$, where $\lVert\mathbf{W}\rVert_2$ is the largest singular value. For $n \gt m$ unit rows, $\lVert\mathbf{W}\rVert_2^2 \ge \lVert\mathbf{W}\rVert_F^2/m = n/m \gt 1$, so the toy's unit step is larger than the safe one.

The cell shows the gap. It uses random $24\times64$ measurement matrices and nonnegative $k$-sparse signals. It compares exact recovery by basis pursuit with the best one-step decoder we can give the toy, which correlates once and keeps the $k$ largest entries. That decoder is even told the true $k$, which the toy is not.

With 24 measurements of 64 features, $\ell_1$ minimization recovers every 1- to 4-sparse signal in these trials and 95 percent of the 6-sparse ones. The one-step decoder already fails on a third of the 2-sparse signals, and on every signal from $k = 4$ on. Compressed sensing says the information survives the compression. The toy model says how much of it a single ReLU layer can extract, and the answer is much less.

This also frames §23.6. A sparse autoencoder learns an encoder $\operatorname{ReLU}(\mathbf{x}\mathbf{W}_{\text{enc}} + \mathbf{b}_{\text{enc}})$, trained so that its sparse code reconstructs the activations through a dictionary of directions. Its encoder again correlates once and thresholds once, but with learned weights that need not be the transpose of the dictionary. It does sparse recovery with a trained network in one pass, instead of with an iterative solver for each input. Neuroscience met the same problem for natural images and solved it with sparse coding (Olshausen and Field 1996). An interviewer will ask how superposition relates to compressed sensing and why the toy needs more sparsity. The one-step-of-ISTA theorem is the answer.

import numpy as np

def soft_threshold(u, t):
    raise NotImplementedError

def ista(Phi, y, lam, steps):
    raise NotImplementedError

Correlated features, computation, and memorization

Everything so far assumed independent features. Real features are often correlated (fur, ears, and eyes appear together) or anticorrelated (a token is French or Korean, rarely both). Elhage et al. (2022) modeled both. In a correlated set, one random draw decides whether all of the set's features are on together. In an anticorrelated set, the set is active with probability $1 - S$, and then exactly one of its features, chosen at random, is on. Their findings are empirical but consistent, and the readout formula makes each one easy to understand.

• Correlated features prefer to be orthogonal. They are often on together, so any overlap between them is paid for often. A model in superposition overall can still give a correlated group a nearly orthogonal local basis. So within one context, a layer may look free of superposition.
• When there is no room, correlated features collapse onto their principal component. Two correlated features $a$ and $b$ get one shared direction that encodes $(a + b)/\sqrt 2$. That is what principal component analysis, and so a linear model, would do.
• Anticorrelated features prefer to share a direction with opposite signs. They are never on together. So the negative interference between them, which the ReLU deletes, costs nothing, and an antipodal anticorrelated pair is reconstructed exactly.

The next exercise has you build both distributions and check these findings, in trained weights and in the loss itself.

import numpy as np

def sample_pairs(rng, B, n_pairs, S, mode):
    raise NotImplementedError

def pair_geometry(W, pairs, tol=0.2):
    raise NotImplementedError

In the four-pair weights, each correlated pair shares one direction of length about 0.7, close to $1/\sqrt2$. That is the collapse onto the principal component, and the four shared directions form two antipodal pairs. Plain training on anticorrelated pairs often stops in the orthogonal arrangement, with a small positive loss, instead of the exact antipodal one. The toy's loss has many local minima, which is also why the cells keep the best of several runs.

The toy model so far only stores and retrieves. Can a network compute while in superposition? Elhage et al. (2022) trained a model with a ReLU hidden layer and separate encoder and decoder weights to output $\lvert x_i\rvert$ for sparse inputs in $[-1, 1]$. The task needs the nonlinearity, since $\lvert x\rvert = \operatorname{ReLU}(x) + \operatorname{ReLU}(-x)$ takes two neurons per feature without superposition. With 100 features, 40 neurons, and sparse enough inputs, the model computed absolute values for more features than it had neurons.

Here the hidden ReLU makes the neuron basis privileged, so features tend to line up with neurons. With dense features each neuron carries one feature. Once features are sparse, many neurons carry one primary feature with a large weight plus secondary features with small weights. Such a neuron's strongest activations all come from its primary feature, while its weaker activations are polysemantic, which is what researchers report for many real language-model neurons.

This led the authors to the hypothesis that motivates §23.6. A real network may be "noisily simulating" a much larger and sparser network, computing on far more features than it has neurons. How far computation in superposition extends is much less understood than storage.

Are real models in superposition?

The toy model proves that superposition can happen and shows what it looks like. Whether real networks use it has to be argued from indirect evidence, since we do not know their true features. Four lines of evidence point the same way.

• Polysemantic neurons are common. The neuron that mixes citations, dialogue, HTTP, and Korean, and InceptionV1's cat-and-car neuron, are typical, not exceptions. The image-text model CLIP has multimodal neurons that respond to one concept across photos, drawings, and written text (Goh et al. 2021). The typographic attacks of §30.6, which fool CLIP with a handwritten label, exploit them.
• Sparse probing finds superposition in early layers and dedicated neurons later. A sparse probe is a linear probe allowed to use only $k$ neurons, so varying $k$ measures how many neurons a feature needs. Gurnee et al. (2023) trained such probes on the MLP neurons of seven models, from 70 million to 6.9 billion parameters, for over 100 features in 10 categories. Early layers represented many features as sparse combinations of neurons, which is superposition. Middle layers had seemingly dedicated neurons for higher-level features of the context. On average, representations became sparser with scale.
• Sparse dictionaries recover far more features than neurons. A sparse autoencoder trained on the 512 neurons of a one-layer model's MLP found interpretable features at every dictionary size up to 131,072 (Bricken et al. 2023). Dictionaries of up to 34 million features extract interpretable directions from the middle-layer residual stream of a frontier model, Claude 3 Sonnet (Templeton et al. 2024). §23.6 gives the numbers and the caveats.
• Basis alignment is a training artifact, not a necessity. The privileged-basis experiments above show residual streams with no architectural reason for special coordinates. That is the setting in which features are free to take any direction.

What the evidence does not show is the toy model's exact structure. Its clean polytopes, pentagons and tetrahedra, have not been shown to organize the features of a large language model. Real features are not independent, not uniformly distributed, and not equally sparse, and some are not one-dimensional at all.

The circular day and month features of Engels et al. (2024), which §23.2 met as a counterexample to the linear representation hypothesis, are the clearest case. Each lives on a circle in a two-dimensional subspace and is used for arithmetic that wraps around, such as "two days after Friday". Gurnee et al. (2025) found that Claude 3.5 Haiku tracks the character count of a line, which it needs to decide where to break fixed-width text, on a curved one-dimensional manifold. That manifold can equally be read as a family of discrete features. Multidimensional features break the one-feature-one-direction picture that the toy model assumes, and they are a live research question.

What the toy model leaves out, and how to read a layer

The toy is a model of storage, with independent one-dimensional features and a one-step decoder. Every one of those assumptions fails somewhere in a real Transformer. Features are correlated and hierarchical, and some are multidimensional. Most of what a layer does is computation rather than reconstruction. The decoder is the rest of the network, many layers deep. The definition of a feature is open. And the toy is trained to reconstruct its input, while a language model's layers are trained only for what they contribute to next-token loss. So the importance of a feature is something the network chooses, not a number we set.

What the toy gets right is the logic, and that logic decides how to study a layer. If a layer is in superposition, its natural units are directions, not neurons, and there are more of them than dimensions. The table compares the units an interpreter can choose.

Superposition also changes engineering decisions.

• Pruning. Removing a neuron removes a little of many features rather than one feature.
• Quantization. The few large coordinates that training produces have to be protected (§20.2).
• Steering. Pushing the activations along one feature's direction (§23.7) also nudges every feature that interferes with it.
• Disentanglement. A variational autoencoder trained with a heavier penalty on its latent code tries to give each factor of variation its own latent (§32.4). That is a different goal from finding features in superposition, since there are more features than latents. A vector-quantized code (§32.5) is the extreme case of a sparse dictionary, with exactly one active element at a time.

Read more

• Elhage et al. (2022). Toy Models of Superposition. The toy model, the two-feature phase diagram, uniform polytopes and sticky fractions, correlated features, computation in superposition, and the adversarial link (interactive version at transformer-circuits.pub).
• Olah et al. (2020). Zoom In: An Introduction to Circuits. Polysemantic neurons in InceptionV1 and the first use of the word superposition for features spread across neurons.
• Elhage, Lasenby, and Olah (2023). Privileged Bases in the Transformer Residual Stream. The kurtosis test for basis alignment and the experiments that point at Adam.
• Henighan et al. (2023). Superposition, Memorization, and Double Descent. The toy model on finite data, where memorization means storing data points in superposition.
• Gurnee et al. (2023). Finding Neurons in a Haystack: Case Studies with Sparse Probing. Sparse probes over seven models showing superposition in early layers and dedicated neurons in middle layers.
• Engels et al. (2024). Not All Language Model Features Are One-Dimensionally Linear. Circular features for days and months, and a definition of irreducible multidimensional features.
• Candès, Romberg, and Tao (2006). Robust Uncertainty Principles: Exact Signal Reconstruction from Highly Incomplete Frequency Information. Exact recovery of a sparse signal from a small random set of its Fourier coefficients by $\ell_1$ minimization, one of the papers that founded compressed sensing.
• Donoho (2006). Compressed sensing. The companion founding paper that named the field, showing that a few nonadaptive measurements of a compressible signal allow near-optimal reconstruction by $\ell_1$ minimization.

Watch next

▶

▶