Steering means changing what a model does by editing its activations while it runs. You do not change the prompt and you do not retrain. You take a vector that stands for a concept, add it to the model's hidden state at one layer, and let the rest of the forward pass run on the edited state. This section shows where such vectors come from, the three ways to apply them, what they cost the model, and whether they beat writing a better prompt.

Here is the idea as one experiment. Run a language model on a few prompts that say "I love this", and average its hidden state at a middle layer. Do the same for prompts that say "I hate this", and subtract the second average from the first. Now run the model on a neutral prompt, add three times that difference at the same layer, and let it write. The continuation turns warm. Nothing was trained, the prompt was not changed, and the whole edit was one vector added at one layer.

Steering matters for three reasons. For science, it tests whether a direction you can read from the activations is one the model actually uses. For control, it changes behavior without any training. For safety, the same tool removed refusal from 13 open chat models with a small edit to their weights and no training at all (Arditi et al. 2024). That result changes how anyone should think about releasing a model's weights.

Three ideas from earlier carry the whole section, so here they are in brief.

The residual stream. A Transformer keeps one vector of $d$ numbers for each token position. Every block reads that vector and adds its own output back into it, so the vector at any layer is the sum of everything written so far (§5.1). We write it $\mathbf{x}^{(\ell)}_t \in \R^d$ for position $t$ after $\ell$ blocks. It is a row vector, as in all of the book's Transformer chapters, so a dot product of two rows is $\mathbf{x}\cdot\mathbf{u} = \mathbf{x}\mathbf{u}^\top$.

Directions. A concept direction is a vector $\mathbf{r} \in \R^d$, and $\hat{\mathbf{r}} = \mathbf{r}/\lVert\mathbf{r}\rVert$ is the unit vector that points the same way. Every activation has a coordinate along $\hat{\mathbf{r}}$, the number $\mathbf{x}\cdot\hat{\mathbf{r}}$. Subtracting $(\mathbf{x}\cdot\hat{\mathbf{r}})\hat{\mathbf{r}}$ deletes that coordinate and leaves everything perpendicular to $\hat{\mathbf{r}}$ untouched. This is the orthogonal projection of §0.10, which already used it to remove a concept from a representation.

Concept directions found by probes. A probe is a linear classifier trained to predict a concept from activations. §23.2 found that many concepts can be read along a single direction. The claim that models store concepts as directions is called the linear representation hypothesis. That section also showed that two natural directions do different jobs. A well-fit classifier's weight vector is the best direction to read a concept with, because it tilts away from other features that overlap the concept. The difference between the two class means is the better direction to write along, because it is where the two classes actually differ.

§23.1 ranked interpretability evidence on a ladder, from "a concept can be decoded from the activations" at the bottom to "the model computes with it in this way" at the top. A probe reaches only the bottom rung. Steering asks a causal question. If you move the stream along the direction, does the behavior move? Park et al. (2023) state this as one form of the linear representation hypothesis, the one §23.2 called the embedding version. It says that adding a multiple of the right vector to the stream raises the probability of one concept and leaves alone every concept that can change independently of it.

So steering is a test. When it works, that is evidence the model uses the direction. When it fails, the concept may not be linear at that layer, it may live at another layer or position, or it may be spread over several directions.

Here is the plan. We define the three edits and try them by hand. We build steering vectors and measure how many examples they need. We weigh the dose against the damage, including inputs that move the wrong way. We prove that deleting a direction at every layer is the same as editing the weights, and rebuild the refusal result on a toy model. We clamp a feature of a sparse autoencoder without breaking anything else, and write a new fact into a model's weights. We end with the question that matters in practice, whether any of this beats a prompt.

Three edits to one direction

You can do three things to an activation along a direction. You can push it by a fixed amount, the same for every input. You can delete its coordinate along the direction. Or you can overwrite that coordinate with a value you choose. Here is each one on three numbers, before the definitions.

The cell repeats the arithmetic.

Now the general statements. A steering vector is built from two sets of inputs that differ in one property, and the edit is applied at a chosen layer and set of positions.

This $\mathbf{r}$ is the difference-of-means direction of §23.2, the one that section showed is right for writing.

In plain words, addition pushes every input by the same amount in the same direction, ablation deletes the coordinate, and clamping overwrites it. The difference matters more than it looks. Addition keeps all the variation between inputs along $\hat{\mathbf{r}}$ and shifts it. Ablation and clamping erase that variation, so every input ends up with the same coordinate.

What does each edit do to a number the model reads off the stream? Take any linear readout $z = \mathbf{x}\cdot\mathbf{u}$. One example is a logit difference between two tokens $a$ and $b$, read through the unembedding along $\mathbf{u} = \mathbf{u}_a - \mathbf{u}_b$, the difference of their readout columns (§23.3). Addition changes $z$ by $\alpha\,(\hat{\mathbf{r}}\cdot\mathbf{u})$, the same amount on every input. Ablation changes it by $-(\mathbf{x}\cdot\hat{\mathbf{r}})(\hat{\mathbf{r}}\cdot\mathbf{u})$, which depends on the input. Either way, a readout moves in proportion to its overlap $\hat{\mathbf{r}}\cdot\mathbf{u}$ with the direction, and a readout perpendicular to $\hat{\mathbf{r}}$ does not move at all.

That is the whole story for a linear model. A real model is not linear after the edit. Its later layers read the edited stream through norms, attention, and MLPs, and everything interesting about steering comes from what those layers do with the push.

The choice of edit has one practical consequence that interviewers like to probe. To remove a behavior, ablation is gentler than subtracting the vector. Ablation moves each input only as far as that input expresses the direction. Subtraction pushes inputs that never had the concept to places no real input occupies. The refusal results below put a number on that difference.

Where steering vectors come from

Every recipe builds the direction from activations on two contrasting sets of inputs, one that shows the behavior and one that does not. The recipes differ in how many contrasts they use, where in the model and the prompt they read, and how they turn the activations into one vector. Here are the five main ones, from the quickest to the most automated.

One pair of prompts. The quickest recipe takes a single pair of short prompts that differ in the concept, such as "Love" and "Hate". It runs the model on both, subtracts the two activations at one layer, and adds a multiple of the difference to the first positions of a new prompt. This is activation addition, ActAdd for short (Turner et al. 2023). On GPT-2-XL, the pair "I talk about weddings constantly" minus "I do not talk about weddings constantly" made unrelated prompts drift toward weddings. At the best layer it steered the topic about 90 percent of the time, against a baseline of 2 percent.

Many matched pairs. One pair is noisy, because every other difference between its two prompts ends up in the vector too. Averaging over many pairs helps. Making the two members of each pair differ in a single token helps more. In contrastive activation addition, CAA (Rimsky et al. 2023, whose first author now publishes as Panickssery), each pair is one multiple-choice question. One copy ends with the answer letter that shows the behavior, say "(A)", and the other with the letter for its opposite. Activations are read at that letter, so the two copies share everything else. On Llama 2 Chat, adding or subtracting vectors built this way raised or lowered all seven behaviors tested in open-ended answers, sycophancy and refusal among them. For most of the behaviors, steering moved the model further than a system prompt asking for the behavior did on its own.

A probe and a few attention heads. Inference-time intervention, ITI (Li et al. 2023), looks for the concept in individual attention heads instead of the whole stream. It trained a probe on every head's output to separate true from false answers on TruthfulQA, a benchmark of questions that tempt models into repeating common misconceptions. It kept the 48 heads whose probes were most accurate on held-out data. At every generated token it shifted each kept head along a direction in that head's own space. On Alpaca, an instruction-tuned LLaMA-7B, the share of answers judged both true and informative rose from 32.5 to 65.1 percent.

The ITI paper also tested which direction to shift along, and the answer is the reading-versus-writing distinction of §23.2, measured on a real model.

Principal components of differences. Representation engineering, RepE (Zou et al. 2023), builds pairs of stimuli that differ in the target concept and subtracts each pair's activations. It takes the first principal component of those differences as the direction and calls it a reading vector. For behaviors such as honesty it flips the sign of every other difference before the principal component analysis (PCA). That detail is essential, as the second cell below shows.

Automated pipelines. Recent work has a model write the contrast sets. Persona vectors (Chen et al. 2025) start from a written description of a trait, such as sycophancy. A model writes pairs of system prompts, one that brings the trait out and one that suppresses it. A judge model keeps the responses that clearly do and do not show it, and the vector is the difference of their mean activations. Emotion vectors (Sofroniew et al. 2026) had Claude Sonnet 4.5 write short stories in which a character feels one of 171 emotions. Each emotion's vector is the mean activation over its stories minus the mean over all emotions.

How many examples a steering vector needs

How good is a difference of means as an estimate of the direction a behavior runs through? A simple model of activations answers it. Each activation is the sum of a context part (what the prompt is about), a noise part, and, for positive examples only, a fixed shift $\boldsymbol\delta = \delta\,\mathbf{g}$. The shift has length $\delta$ and points along an unknown unit direction $\mathbf{g}$, and it is the steering vector we want. In an unpaired design the positive and negative examples are unrelated prompts. In a paired design each positive example has a negative twin with the same context, as in CAA.

In plain words, the estimate is right on average, and its error shrinks like $1/\sqrt{n}$, but it grows like $\sqrt{d}$ with the width. In high dimensions the error vector is nearly orthogonal to $\boldsymbol\delta$ and its length is nearly fixed. So the cosine between the estimate and the true direction is close to $\delta/\sqrt{\delta^2 + \mathbb{E}\lVert\mathbf{r} - \boldsymbol\delta\rVert^2}$. Pairing removes the context term, which is usually the large one. That is why one-token contrasts work well, and why a single prompt pair, whose two prompts differ in many ways at once, gives only a rough direction.

The cell checks both formulas in $d = 256$ with $\delta = 2$, $\sigma_c = 1$, and $\sigma_\epsilon = 0.3$, averaging the cosine over 40 repetitions for each $n$.

Now the PCA detail. Principal component analysis first subtracts the mean of its inputs and then finds the direction of largest variance. If every difference points the same way, the mean difference is the signal, and centering deletes it before PCA ever looks.

The difference of means gets a cosine of 0.92. PCA of the raw differences gets 0.02, a direction of pure noise. With alternating signs the mean is near zero, so the signal survives centering as variance, and PCA recovers it at 0.92. Random signs work as well, at 0.91. So PCA of signed differences finds the direction even when you do not trust which member of each pair is positive, since swapping a pair only flips the sign of its difference. The principal component itself has no sign, so a few labeled examples then decide which way it points.

Real contrast sets have two more problems that the theorem leaves out, and the widget shows both. The first is a confound, some other feature that also differs between the sets. If harmful requests also tend to be shorter, for instance, the difference of means points partly along length, and no amount of data removes that bias. The second is label noise. If a fraction $\eta$ of the examples in each set belong to the other set, the expected vector shrinks by a factor $1 - 2\eta$, but its direction does not change.

The widget's activations live in 32 dimensions, and the plot shows two of them, the true direction $\mathbf{g}$ and a nuisance feature $q$. Its toy model shows the behavior when an activation's coordinate along $\mathbf{g}$ passes the dashed line, so it reads $\mathbf{g}$ and nothing else. The ablation view removes each held-out positive's component along the estimate.

One more choice is the unit of the dose. Adding $\alpha = 8$ along a unit vector means nothing until you compare it with the stream it is added to. Papers use three units.

• The spread along the direction. ITI uses multiples of the standard deviation of the activations along it.
• The stream norm. The emotion-vector work uses fractions of the average stream norm at that layer.
• A feature's range. Steering with features of a sparse autoencoder uses multiples of the feature's largest activation over a dataset (Templeton et al. 2024). The AxBench benchmark (Wu et al. 2025) set its dose the same way, as a factor times the largest activation it measured for each concept.

A dose in raw units does not transfer between layers, because the norm of the stream usually grows with depth.

import numpy as np

def steering_vector(acts_pos, acts_neg, ref_acts):
    raise NotImplementedError

def add_steer(X, r_hat, alpha, scale, positions=None):
    raise NotImplementedError

Dose, damage, and steerability

A steering result has three parts.

• The target effect is how far the behavior moves.
• The off-target damage is how far everything else moves. It is measured as accuracy on a capability benchmark, or as the KL divergence of the next-token distribution on unrelated inputs. The KL divergence $D_{\mathrm{KL}}(\mathbf{p}\,\Vert\,\mathbf{q})$ measures how far a distribution $\mathbf{q}$ is from a reference $\mathbf{p}$, and it is zero only when the two are equal (§0.5).
• The reliability is whether every input moves the intended way, or whether a healthy average hides inputs that move the wrong way.

A paper that reports only the first part has not told you whether the method works.

Why a small push is nearly free

Start with two tokens, before the general case.

The same holds for any number of tokens. Suppose adding $\alpha\hat{\mathbf{r}}$ changes the logits $\mathbf{z} \in \R^V$ to $\mathbf{z} + \alpha\mathbf{s}$, where $\mathbf{s} \in \R^V$ is the logit shift per unit of dose, one number for each of the $V$ tokens. This is exact when the logits are linear in the stream. One example is an edit to the final stream with the final norm's scale held fixed at some value $\sigma$. Folding the norm into the unembedding, as in §23.3, turns the readout into a matrix $\mathbf{W}_U' \in \R^{d\times V}$ that centers, applies the gain, and unembeds. Then $\mathbf{s} = \hat{\mathbf{r}}\mathbf{W}_U'/\sigma$. In every such case the damage depends only on the numbers $s_1, \dots, s_V$ and on the model's own distribution.

In plain words, the logits move linearly in the dose, but the damage, measured by KL, starts out quadratic. Doubling a small dose doubles the effect and quadruples the damage. So there is always a range of small doses where the effect is large compared with the damage, and past it the damage wins. The damage is small when the push changes all plausible next tokens by about the same amount, since a constant shift of the logits is invisible to the softmax.

In a deep network $\mathbf{s}$ is not fixed, because the layers after the edit are nonlinear. To first order, $\mathbf{s}$ is the Jacobian of the logits with respect to the stream, applied to $\hat{\mathbf{r}}$. So the theorem describes small doses, and large doses are an empirical question. An interviewer who asks why steering papers sweep the dose wants this answer.

A dose-response curve in a toy

Now a toy of a more realistic size, built like the synthetic streams used throughout this chapter. Activations in $\R^{64}$ are sparse combinations of 128 random unit directions $\mathbf{g}_i$, the planted features. Random directions in high dimensions are nearly orthogonal, so the stream can hold more features than it has dimensions.

Three parts of the toy matter.

• The behavior logit reads $\mathbf{g}_0$. This is what we steer.
• A 10-way capability readout reads ten other features. It stands for everything we do not want to disturb.
• One more feature, $\mathbf{g}_1$, gates the behavior. It acts through a multiplicative interaction of the kind a gated feed-forward layer computes (§5.3). On inputs where $\mathbf{g}_1$ is strongly on, pushing along $\mathbf{g}_0$ lowers the behavior logit.

The cell adds $\alpha\mathbf{g}_0$ for a range of doses and prints the behavior rate, the capability accuracy, the measured KL on the capability readout, and the theorem's prediction $\tfrac12\alpha^2\operatorname{Var}_{\mathbf{p}}(s)$, averaged over inputs.

Inputs that move the wrong way

Why did the behavior stop near 0.88 instead of reaching 1? Because of the inputs on which the gate is on. On those inputs the push lowers the behavior, so a stronger push loses them. Tan et al. (2024) made this measurable for real models. Earlier work had plotted the behavior against the steering strength averaged over a dataset. Tan et al. computed the same curve for every input separately.

The cell computes the slope for every input of the toy.

The mean curve has a healthy slope of 3.06, yet 7.8 percent of inputs move the wrong way. Every input with $\mathbf{x}\cdot\mathbf{g}_1 > 0.6$ is among them, as the gate predicts, since the slope is about $4 - 10(\mathbf{x}\cdot\mathbf{g}_1 - 0.2)$ when the gate is open. An average cannot show this. Here we planted the interaction and know its cause. In a real model you see only the spread, and finding its cause is a research problem.

The widget runs a toy with depth. Sequences of six positions pass through three stages, and the first four positions are the prompt. The first stage is an MLP. The second is a mixing stage that averages earlier positions into later ones, a stand-in for attention. The third, block 3, has a gated unit. The behavior and the capability are read at the last position. You choose where to add the vector, at which positions, and how hard to push.

Make three predictions as you go. Edit after MLP 1 on all positions and find the dose with the highest behavior rate. Is it the dose you would ship? Then switch to "after block 3". Why do all the per-sequence slopes become equal, and why does the behavior reach 100 percent there? Finally edit after mixing on the prompt only, and explain the flat curves before you read the note.

import numpy as np

def choose_steering(target_val, kl_val, target_test, kl_test, alphas, kl_budget):
    raise NotImplementedError

Removing a direction everywhere

Activation addition edits one layer. Arditi et al. (2024) used the stronger edit of the ablation definition. They ablated $\hat{\mathbf{r}}$ from every residual stream activation, at every layer and every position, after each attention sublayer and after each MLP sublayer, so that the model never carries any component along $\hat{\mathbf{r}}$. Doing that at run time needs a hook on every sublayer, a function that sees a module's output and may replace it (§0.9). They then observed that the same effect can be built into the weights.

The idea fits in two sentences. Suppose the stream after the embedding is $\mathbf{e}$ and one block then adds $\mathbf{y}$, so the stream is $\mathbf{e} + \mathbf{y}$. If neither $\mathbf{e}$ nor $\mathbf{y}$ has a component along $\hat{\mathbf{r}}$, neither does their sum. So if every part of the model that writes to the stream is changed so it cannot write along $\hat{\mathbf{r}}$, the stream never contains $\hat{\mathbf{r}}$ at any layer.

To make that precise, split the model into writers and readers (§5.1). The embedding writes a row of the token table $\mathbf{E} \in \R^{V\times d}$ plus a row of the position table. Each sublayer reads the stream and adds its output. Every write has the form $\mathbf{y} = \mathbf{h}\mathbf{W}_{\text{out}} + \mathbf{b}_{\text{out}}$, where $\mathbf{W}_{\text{out}}$ has $d$ columns. Here $\mathbf{h}$ is whatever the sublayer computed internally.

• For attention, $\mathbf{h}$ is the concatenated head outputs before the output matrix $\mathbf{W}_O$.
• For an MLP, it is the hidden activations before the output matrix.
• For the embedding, it is a one-hot token.

The norms, the query, key, and value maps of attention, the first layer of each MLP, and the unembedding only read the stream. In a pre-norm Transformer, each sublayer normalizes a copy of its input and the stream itself is never normalized, so that split is clean.

In plain words, if no writer can write along $\hat{\mathbf{r}}$, the stream never contains it, because the stream is nothing but the sum of the writes. That is the same as erasing it after every write. The edit is permanent, needs no hooks, and changes each writing matrix by a rank-one term.

import numpy as np

def ablate_run(params, tokens, r_hat):
    raise NotImplementedError

def orthogonalize(params, r_hat):
    raise NotImplementedError

The refusal direction

§19.3 described the threat model. A chat model is fine-tuned to refuse harmful requests, and attackers look for inputs that get around the refusal. Its table of defenses noted, without explaining it, that in open-weight chat models refusal is carried by a single direction that can be ablated. We can now explain that finding, first on a model we build ourselves and then at real scale. Everything in this subsection is synthetic. The "category A" inputs are abstract feature vectors with no content, and nothing here targets a real model.

The toy has the structure the finding suggests.

• An embedding writes the input features to the stream.
• The first MLP block contains a detector neuron that fires only when two particular input features are both on. It writes a direction $\mathbf{g}$ into the stream.
• The second block contains a neuron that reads $\mathbf{g}$ and writes into the direction the unembedding uses for a "refuse" logit.

The other neurons are random. The analysis is not told where $\mathbf{g}$ is.

The next cell estimates the direction from 128 inputs of each category. Then it ablates the direction, adds it, and checks the orthogonalized weights against run-time ablation.

The same steps, at the scale of real chat models, are the result of Arditi et al. (2024).

Three further results matter.

The edit was cheap in capability. On Llama-3 8B Instruct, ablation lowered the refusal score on harmful instructions from 0.95 to 0.01. To measure side effects, the authors had the unedited model answer 100 harmless instructions and scored the edited model on those answers by cross-entropy. Ablation raised it only from 0.195 to 0.213. Subtracting the vector instead, an activation addition with a negative sign, removed refusal just as well but raised the cross-entropy to 0.441.

The geometry of the three edits explains the gap. Subtraction moves every input by the same amount, so harmless inputs, which had a small coordinate along $\hat{\mathbf{r}}$ to begin with, are pushed far past where any real input lies. Ablation sets every input's coordinate to zero. That is a large move only for harmful inputs, and it leaves harmless ones nearly where they were. On five standard benchmarks (MMLU for knowledge, ARC for science questions, GSM8K for grade-school math, and WinoGrande and TinyHellaSwag for commonsense reasoning), the orthogonalized models scored within the 99 percent confidence intervals of the originals for all but two models (Qwen 7B and Yi 34B). TruthfulQA accuracy dropped consistently, which the authors relate to earlier findings that removing safety training lowers it.

The direction explains a known attack. Adversarial suffixes are strings found by optimization that, appended to a request, make a model comply (§19.3). In Qwen 1.8B Chat, such a suffix suppressed the refusal direction on harmful instructions down to the level seen for harmless ones. The eight heads that wrote most to the direction shifted their attention from the instruction to the suffix.

Safety fine-tuning of open weights can be removed. This is a finding about the state of safety fine-tuning, not a procedure. If refusal in a released model runs through one direction that a few hundred example prompts identify, then anyone holding the weights can remove it with a rank-one edit to each writing matrix and no training. Safety behavior installed by fine-tuning is shallow in this specific sense. Risk assessments for open-weight releases have to assume it can be removed, along with the fine-tuning attacks that §19.3 already listed.

What would a defense look like? One research direction changes how the model represents harmful content, instead of training a refusal on top that one edit can remove. Such defenses can be attacked in representation space too, and the box below gives the evidence on both sides.

Clamping a dictionary feature

The directions so far came from contrast sets. §23.6 found directions without labels. A sparse autoencoder (SAE) rewrites each activation as a sum of a few dictionary directions, each scaled by a nonnegative activation, plus whatever it fails to capture. In symbols, the latent activations are $\mathbf{f}(\mathbf{x}) = \operatorname{ReLU}(\mathbf{z}(\mathbf{x}))$ with pre-activations $\mathbf{z}(\mathbf{x}) = (\mathbf{x} - \mathbf{b}_{\text{dec}})\mathbf{W}_{\text{enc}} + \mathbf{b}_{\text{enc}}$. The reconstruction is $\tilde{\mathbf{x}} = \mathbf{f}(\mathbf{x})\mathbf{W}_{\text{dec}} + \mathbf{b}_{\text{dec}}$. Row $i$ of $\mathbf{W}_{\text{dec}}$, written $\mathbf{d}_i$, is the direction latent $i$ writes along.

An SAE latent is a natural handle for steering, since it comes with an interpretation and a typical range. The right way to turn it is not obvious, because the SAE does not reconstruct $\mathbf{x}$ exactly.

Why the two forms are equal. The bracket is $\tilde{\mathbf{x}} - f_i(\mathbf{x})\mathbf{d}_i + c\,\mathbf{d}_i$, because $\tilde{\mathbf{x}}$ contains the term $f_i(\mathbf{x})\mathbf{d}_i$ once. Adding $\mathbf{e}(\mathbf{x}) = \mathbf{x} - \tilde{\mathbf{x}}$ cancels $\tilde{\mathbf{x}}$ and leaves $\mathbf{x} + (c - f_i(\mathbf{x}))\mathbf{d}_i$. ∎

In plain words, clamping moves the activation only along the latent's own direction, by exactly the amount that changes its activation from what it was to $c$. Everything the SAE failed to capture stays in place.

The tempting shortcut is to decode the clamped code and feed that to the model. It also replaces $\mathbf{x}$ by $\tilde{\mathbf{x}}$ everywhere else, which deletes the error term. The error is not small. The SAEs trained on Claude 3 Sonnet explained at least 65 percent of the variance (§23.6), so up to about a third was left in the error. Removing that part changes the model's behavior on its own, and a steering experiment run with the shortcut measures a damaged model.

import numpy as np

def clamp_feature(x, sae, i, c):
    raise NotImplementedError

How much does dropping the error term matter? The cell builds a simple SAE whose decoder rows are the true features of a planted stream. A part of each activation is noise that no latent explains. Then it compares the two ways of clamping.

At real scale, feature clamping produced the best-known steering demonstration.

The three run-time edits now share one form, a rank-one change to the stream along a single direction. Activation addition along a latent is $\mathbf{x} + \alpha\mathbf{d}_i$, which ignores where the latent was. Feature clamping is $\mathbf{x} + (c - f_i(\mathbf{x}))\mathbf{d}_i$, which depends on it. Directional clamping, $\mathbf{x} + (c - \mathbf{x}\cdot\hat{\mathbf{r}})\hat{\mathbf{r}}$, reads the coordinate with the same unit vector it writes along. With an SAE, the reader $f_i$ and the writer $\mathbf{d}_i$ differ.

Editing weights instead of activations

Steering vectors and clamps act at run time and disappear when the hook is removed. Weight orthogonalization showed that some edits can be written into the weights. The most studied case of writing new behavior into weights is editing a single fact, such as making a model say that the Eiffel Tower is in Rome.

Two earlier results set this up.

First, §5.3 showed that a feed-forward layer acts as a key-value memory. In row layout it computes $\operatorname{FFN}(\mathbf{x}) = \mathbf{k}\mathbf{W}_2 + \mathbf{b}_2$ with $\mathbf{k} = \phi(\mathbf{x}\mathbf{W}_1 + \mathbf{b}_1) \in \R^{d_{\text{ff}}}$. The vector $\mathbf{k}$ holds every memory's match strength, so the output matrix $\mathbf{W}_2 \in \R^{d_{\text{ff}}\times d}$ is a linear map from that vector to what the layer writes. Meng et al. (2022) call the whole vector $\mathbf{k}$ the key of the token, and we follow them. (It is a different object from the per-memory key vectors of that section's picture.)

Second, §23.4 showed that causal tracing in GPT-2 XL localizes factual recall to middle-layer MLPs at the last token of the subject. Causal tracing restores clean activations into a corrupted run one site at a time and measures how much of the correct answer comes back.

So to change what the model says about a subject, change what one middle-layer $\mathbf{W}_2$ outputs for that subject's key, and leave its outputs for every other key alone.

Write $\mathbf{W} \in \R^{d_{\text{ff}}\times d}$ for the matrix being edited. Stack many keys the layer has seen, one per row, in $\mathbf{K} \in \R^{N\times d_{\text{ff}}}$. Their outputs under the current weights are $\mathbf{K}\mathbf{W}$, and their uncentered second-moment matrix is $\mathbf{C} = \mathbf{K}^\top\mathbf{K} \in \R^{d_{\text{ff}}\times d_{\text{ff}}}$. We want a new matrix $\mathbf{W}'$ that sends a new key $\mathbf{k}_* \in \R^{1\times d_{\text{ff}}}$ exactly to a new value $\mathbf{v}_* \in \R^{1\times d}$. Among all such matrices, we want the one that changes the outputs on the old keys least in squared error.

The example shows what the formula does but not why it is optimal. The reason is a short argument. The objective is quadratic in the change to $\mathbf{W}$, and the constraint is one linear equation per output coordinate. A Lagrange multiplier for each equation solves it, the same method §9.3 used to find the compute-optimal model size.

In plain words, the edit pushes the memory along $\mathbf{C}^{-1}\mathbf{k}_*^\top$, the new key whitened by the statistics of the old keys. Whitening divides each direction by how much the old keys already use it. So the change lands mostly on directions the old keys rarely occupy, as in the two-dimensional example.

Two consequences are easy to miss. First, the hard constraint makes the update unchanged if $\mathbf{C}$ is multiplied by any positive constant. That is why an estimate of $\mathbb{E}[\mathbf{k}^\top\mathbf{k}]$ from a sample of keys can stand in for the sum over every key the layer has ever seen. Second, a key that is unusual under $\mathbf{C}$, with a large $\mathbf{k}_*\mathbf{C}^{-1}\mathbf{k}_*^\top$, is cheap to edit, while a key that looks like many old ones is expensive.

import numpy as np

def rome_update(W, K, k_star, v_star, ridge=0.0):
    raise NotImplementedError

def specificity(W, W_new, K_other):
    raise NotImplementedError

The cell inserts one new pair into a synthetic memory whose keys are correlated and nonnegative, as post-ReLU activations are. It compares the whitened update with the naive edit that satisfies the same constraint with $\mathbf{C} = \mathbf{I}$, which is the smallest change in Frobenius norm.

This update, applied to a real model, is the method of Meng et al. (2022), and its story comes with a warning.

Steering against prompting and fine-tuning

The question that decides practice is not whether steering moves behavior, which it does, but whether it beats the simplest alternatives. For a long time steering methods were compared mostly with each other. AxBench was built to compare them with everything, including a plain instruction in the prompt.

Read the numbers with care. They say that on generation tasks judged by a model, a written instruction is a better steering tool than adding any single learned direction. They do not say steering is useless. Steering has advantages a prompt lacks. It uses no context, it takes a continuous dose, and it can act where a prompt is ignored or overridden. CAA moved most of its behaviors further than a system prompt did, and for 3 of 7 behaviors it added to what fine-tuning achieved. As an instrument of science, steering tests whether a representation is used. As a safety tool it can remove a behavior from the weights entirely, which no prompt does, and that power serves attackers as well as defenders.

Steering is also finding uses beyond run-time control, in monitoring training and in safety evaluations. The box below has two recent examples.

The table puts the methods of this section side by side.

In PyTorch

The real thing uses forward hooks (§0.9). A hook registered with register_forward_hook sees a module's output, and if it returns a value, that value replaces the output. That is all a steering method needs.

The block trains a tiny residual model whose input tokens come in three groups, A, B, and neutral. The correct output for an input is a uniform distribution over the tokens of its own group. The block then reads the stream after block 1 with one hook and builds a difference-of-means direction between group A and group B inputs. With a second hook it steers group B inputs toward A, quoting the dose in units of the typical stream norm. It measures the KL on neutral inputs, whose outputs the edit should leave alone.

Read more

• Arditi et al. (2024). Refusal in Language Models Is Mediated by a Single Direction. The difference-of-means refusal direction in 13 chat models, its selection criteria, ablation, activation addition, weight orthogonalization with the equivalence proof in Appendix E, and the analysis of adversarial suffixes.
• Rimsky (Panickssery) et al. (2023). Steering Llama 2 via Contrastive Activation Addition. Contrastive activation addition with matched multiple-choice pairs, layer sweeps, and comparisons with system prompts and fine-tuning.
• Li et al. (2023). Inference-Time Intervention: Eliciting Truthful Answers from a Language Model. Probe-selected heads, the mass-mean shift, the truthfulness-helpfulness trade-off, and why the most separating direction is not the best one to intervene along.
• Turner et al. (2023). Steering Language Models With Activation Engineering. ActAdd, the first demonstration that a single prompt pair yields a usable steering vector, with layer and coefficient sweeps on GPT-2-XL.
• Zou et al. (2023). Representation Engineering: A Top-Down Approach to AI Transparency. Reading vectors from PCA of contrast differences, and their use for monitoring and control of honesty and other concepts.
• Tan et al. (2024). Analyzing the Generalization and Reliability of Steering Vectors. Propensity curves, per-input steerability, anti-steerable inputs, and out-of-distribution failures across 40 datasets.
• Wu et al. (2025). AxBench: Steering LLMs? Even Simple Baselines Outperform Sparse Autoencoders. The benchmark that compared steering with prompting and fine-tuning, and detection methods with each other, on 500 concepts.
• Templeton et al. (2024). Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet. The feature-steering procedure that keeps the SAE error, and the Golden Gate Bridge feature.
• Meng et al. (2022). Locating and Editing Factual Associations in GPT. Causal tracing, ROME's rank-one update with its derivation in Appendix A, and the CounterFact evaluation. Read it with Hase et al. (2023), Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Models, which shows that tracing does not pick the layer to edit.

Watch next

▶

▶