Chapter 25 · Section 25.5 · 100 min

Continual learning

Continual learning is training one model on a stream of tasks or data, one after another, so that it learns each new thing without losing what it already knew. It matters because the world keeps changing after a model ships. New facts appear, users bring new kinds of work, and a lab wants next month's model without paying for pretraining again. It is also the deepest kind of memory in this chapter, because what the model learns ends up in its weights, not in its context window or in an external store.

Picture an agent that has worked on a codebase for a week. It kept durable progress notes (§25.4). It kept a long-term store of facts that it pages in and out of its window (§25.3). Yet on day seven the model itself is exactly as good at the work as it was on day one, because nothing it did reached its weights.

The obvious fix is to fine-tune the model every night on the day's work. That fix runs into catastrophic forgetting, a large drop in performance on an old task caused by training on a new one. Train a network on task A, then on task B, and its accuracy on A can collapse. §14.4 explained why with one formula. Let $\theta^\star_A$ be the weights that minimize task A's loss. A small move $\Delta\theta$ away from them raises that loss by about $\tfrac12\Delta\theta^\top\mathbf{H}_A\Delta\theta$ . Here $\mathbf{H}_A$ is the Hessian of A's loss, the matrix of its second derivatives, which says how sharply the loss curves in each direction. Training on B follows B's gradient, and that gradient knows nothing about $\mathbf{H}_A$ . So it moves the weights along the directions A depends on as readily as along the ones A ignores.

This section shows how to learn in sequence without that collapse, and when new knowledge should stay out of the weights altogether. We first measure forgetting across a sequence of tasks. Then we build three families of defenses and test them together in one simulation. A penalty holds still the weights that the old task needs. A replay buffer mixes a few stored old examples into training. Isolation gives each task its own parameters. Then we turn to large language models. We see how labs keep a model current, why reinforcement learning forgets less than supervised fine-tuning, how a model can learn during a single input, and where a new piece of knowledge should live. Interviewers ask most often about the derivation of the penalty, the Fisher information, and replay.

Forgetting is measured with a matrix of accuracies

Start with the setting. A model sees tasks one at a time. It may train on each task only while that task is current. After each task it is tested on every task. A task here is anything with its own training set, such as a new set of classes, a new kind of input, or a new month of data.

📐 Continual learning and its three scenarios

A continual learning problem is a sequence of tasks $t = 1, \dots, n_{\text{task}}$ , presented in order. Task $t$ has a training set $\mathcal{D}_t$ of $N_t$ examples. While learning task $t$ , the learner may use $\mathcal{D}_t$ and whatever it kept from earlier tasks, such as its weights and perhaps a small buffer of old examples. It may not use the full earlier datasets. After the whole sequence it must do well on every task it has seen. Van de Ven and Tolias (2019) separate three scenarios by what is known at test time.

Task-incremental. Each test input comes with the identity of its task, so the model may keep a separate output head per task.
Domain-incremental. The task identity is not given and need not be inferred. Every task has the same set of outputs, and only the inputs change (the same ten digits, rotated a different way in each task).
Class-incremental. The task identity is not given and must be inferred, because each task brings new classes and the model must choose among all the classes seen so far.

The scenario matters because the same method can look excellent in one and fail in another. The standard tests are built from MNIST, a dataset of small images of handwritten digits. Split MNIST cuts it into five tasks of two digits each, 0 and 1, then 2 and 3, and so on. In the task-incremental version the test question is "is this the first or the second digit of task 3?" In the class-incremental version it is "which of the ten digits is this?" Permuted MNIST makes every task the full ten-digit problem. Each task shuffles the pixels by its own fixed permutation, so the tasks' inputs have almost nothing in common.

So the first question to ask of any continual learning result is which scenario it was measured in. Our own simulation later in this section shows a milder version of the same pattern.

To measure forgetting we record a whole matrix, not one number. Row $i$ holds the test accuracy on every task after training through task $i$ . The diagonal entry of row $j$ is the accuracy on task $j$ right after it was learned.

📐 The accuracy matrix, ACC, BWT, and FWT

Let $R_{i,j}$ be the test accuracy on task $j$ after training through task $i$ . These numbers fill the accuracy matrix $\mathbf{R} \in \R^{n_{\text{task}}\times n_{\text{task}}}$ , one row per task. Let $\bar b_j$ be the accuracy of the randomly initialized model on task $j$ . Lopez-Paz and Ranzato (2017) define the average accuracy ACC, the backward transfer BWT, and the forward transfer FWT,

\text{ACC} = \frac{1}{n_{\text{task}}}\sum_{j=1}^{n_{\text{task}}} R_{n_{\text{task}},j}, \qquad \text{BWT} = \frac{1}{n_{\text{task}}-1}\sum_{j=1}^{n_{\text{task}}-1}\big(R_{n_{\text{task}},j} - R_{j,j}\big), \qquad \text{FWT} = \frac{1}{n_{\text{task}}-1}\sum_{j=2}^{n_{\text{task}}}\big(R_{j-1,j} - \bar b_j\big).

In plain words, ACC is the final report card, the mean of the last row. BWT asks how much each task's accuracy changed between the moment it was learned (the diagonal) and the end. Negative BWT is forgetting. FWT asks how much the earlier tasks helped on task $j$ before the model trained on it, compared with the untrained model. BWT skips the last task, because nothing came after it. FWT skips the first, because nothing came before it.

✏️ Reading an accuracy matrix by hand

Take three tasks, chance accuracy $\bar b_j = 0.5$ on each, and

\mathbf{R} = \begin{pmatrix} 0.95 & 0.50 & 0.48 \\ 0.70 & 0.93 & 0.52 \\ 0.60 & 0.75 & 0.94 \end{pmatrix}.

ACC is the mean of the last row, $(0.60 + 0.75 + 0.94)/3 = 0.763$ . BWT compares the last row with the diagonal for the first two tasks, $\big((0.60 - 0.95) + (0.75 - 0.93)\big)/2 = (-0.35 - 0.18)/2 = -0.265$ . So the model lost 26.5 points on each old task, on average. FWT uses the entries just above the diagonal, $\big((0.50 - 0.5) + (0.52 - 0.5)\big)/2 = 0.01$ . Earlier tasks barely helped later ones.

An ACC of 0.76 could come from a model that never learned much. It could also come from one that learned everything and then forgot. Only the matrix tells them apart.

In practice, report ACC and BWT together, and show the matrix when you can. An interviewer who hears a single accuracy for a continual learner will ask how much of it is forgetting.

Warm-up

ACC, BWT, and FWT from an accuracy matrix

Write cl_metrics(R, baseline). It returns the three numbers (acc, bwt, fwt) defined above, as Python floats.

R is an $n_{\text{task}}\times n_{\text{task}}$ array or nested list. R[i, j] is the accuracy on task $j$ after training through task $i$ , with zero-based indices.
baseline is the vector $\bar{\mathbf{b}}$ of accuracies of the untrained model.

You may assume $n_{\text{task}} \ge 2$ .

import numpy as np

def cl_metrics(R, baseline):
    R = np.asarray(R, dtype=float)
    acc = float(np.mean(np.diag(R)))      # wrong placeholder
    return acc, 0.0, 0.0

Tests, hints, verified solutions, and AI help, with Pro at $7.99 a month.

Forgetting in two dimensions, by hand

Here is the picture to keep in mind. Each task's loss, drawn over the weights, is a valley. Training on task A brings the weights to the bottom of A's valley. Task B has its own valley somewhere else, and gradient descent on B walks straight downhill toward it. On the way it climbs A's walls in whatever direction B's valley happens to lie. If A's valley is narrow in that direction, the walk is a disaster for A. If it is wide, the walk costs A almost nothing.

There are three ways to protect A. A penalty makes the walk avoid A's narrow directions. Replay keeps visiting A's valley during the walk. Isolation gives each task its own weights, so the walks never collide. We start with the penalty, in a case small enough to check by hand.

✏️ Two quadratic tasks, trained plainly and with a spring

Let the weights be $\theta \in \R^2$ . Task A has loss $\mathcal{L}_A(\theta) = \tfrac12\theta^\top\mathbf{H}_A\theta$ , with $\mathbf{H}_A = \operatorname{diag}(4, 0.25)$ and optimum $\theta^\star_A = \mathbf{0}$ . So A cares a lot about $\theta_1$ and little about $\theta_2$ . Task B has loss $\mathcal{L}_B(\theta) = \tfrac12\lVert\theta - \mathbf{b}\rVert^2$ , with $\mathbf{b} = (2, 2)$ .

Plain training on B, started from $\theta^\star_A$ , converges to $\mathbf{b}$ . Task B's loss falls to 0. Task A's rises to $\tfrac12(4\cdot 2^2 + 0.25\cdot 2^2) = 8.5$ .

Training with a spring. Add to B's loss a spring that pulls each weight back toward A's optimum. Make each spring as stiff as A's curvature in that direction. Write $F_{A,i}$ for the stiffness of weight $i$ , here $F_{A,1} = 4$ and $F_{A,2} = 0.25$ (the next part explains the letter). The spring adds $\tfrac{\lambda}{2}\sum_i F_{A,i}\theta_i^2$ to the loss, and the number $\lambda$ sets its strength. This is elastic weight consolidation, EWC for short (Kirkpatrick et al. 2017). §14.4 introduced it as one remedy for forgetting during fine-tuning. Here we work it by hand, derive where the springs come from, extend it to many tasks, and test it against the other defenses.

With $\lambda = 1$ the objective is $\tfrac12\lVert\theta - \mathbf{b}\rVert^2 + \tfrac12(4\theta_1^2 + 0.25\theta_2^2)$ . Set its gradient to zero, one coordinate at a time. That gives $(\theta_1 - 2) + 4\theta_1 = 0$ and $(\theta_2 - 2) + 0.25\theta_2 = 0$ , so $\theta = (0.4, 1.6)$ . Then $\mathcal{L}_A = \tfrac12(4\cdot 0.16 + 0.25\cdot 2.56) = 0.64$ and $\mathcal{L}_B = \tfrac12(1.6^2 + 0.4^2) = 1.36$ .

The total is 2.0, against 8.5 for plain training. The direction A barely cares about, $\theta_2$ , moved 80 percent of the way to B's optimum. The direction A cares about, $\theta_1$ , moved only 20 percent of the way.

Both losses here are exactly quadratic, and the spring's stiffness is exactly A's curvature. So EWC with $\lambda = 1$ minimizes $\mathcal{L}_A + \mathcal{L}_B$ . That is the loss we would have minimized had both datasets been available at once. The cell confirms the numbers and shows what a larger $\lambda$ does.

Python · cell 1

With $\lambda = 10$ the spring wins. The weights end at $\theta = (0.049, 0.571)$ . Task A is almost untouched ( $\mathcal{L}_A = 0.046$ ), at the price of task B ( $\mathcal{L}_B = 2.92$ ). This is the stability-plasticity trade-off. A stable learner keeps its old skills, a plastic one learns new ones fast, and $\lambda$ is the dial between them. Every method that protects old tasks has a dial like this. An interviewer will ask how you would set it. Tune it on a validation set that contains both old and new tasks.

Where the spring's stiffness comes from

Why should each spring be as stiff as task A's curvature, and what exactly is $\lambda$ ? Bayes rule answers both (§0.4). Here are its pieces. The likelihood $p(\mathcal{D} \mid \theta)$ is how probable the data are under weights $\theta$ . The prior $p(\theta)$ is how plausible the weights were before any data. The posterior $p(\theta \mid \mathcal{D}) \propto p(\mathcal{D} \mid \theta)\,p(\theta)$ is how plausible they are after the data. Its peak, the mode, is the maximum a posteriori (MAP) estimate.

In this language, learning in sequence is easy. After task A, everything A has to say about the weights is in the posterior $p(\theta \mid \mathcal{D}_A)$ . To learn B, multiply that posterior by B's likelihood. The trouble is that the posterior of a network is a distribution over millions of weights, and nobody can store it.

EWC replaces it by the Gaussian that has the same peak and the same curvature at the peak. This is called the Laplace approximation. The negative log-density of a Gaussian is a quadratic bowl, so the old posterior becomes a spring centered on A's solution. The spring's stiffness is the Gaussian's inverse covariance, called its precision. A tight Gaussian has a high precision, which means a stiff spring.

∴ EWC as a Laplace approximation

Assume the two tasks' datasets are conditionally independent given $\theta$ , which means that once the weights are fixed, seeing one dataset tells you nothing more about the other. Let $\theta^\star_A$ be a mode of $p(\theta \mid \mathcal{D}_A)$ . Let the prior be $\mathcal{N}(\mathbf{0}, \lambda_0^{-1}\mathbf{I})$ , with precision $\lambda_0$ . Let $\mathcal{L}_B$ be the mean negative log-likelihood over task B's $N_B$ examples. Task A has $N_A$ examples. Make three approximations.

Replace $\log p(\theta \mid \mathcal{D}_A)$ by its second-order Taylor expansion at $\theta^\star_A$ .
Replace the average per-example Hessian in it by the Fisher information $\mathbf{F}_A$ , defined in the next subsection.
Keep only the diagonal entries $F_{A,i}$ of $\mathbf{F}_A$ .

Then the MAP estimate after B minimizes

\mathcal{L}(\theta) = \mathcal{L}_B(\theta) + \frac{\lambda}{2}\sum_i F_{A,i}\,\big(\theta_i - \theta^\star_{A,i}\big)^2, \qquad \lambda = \frac{N_A}{N_B},

plus a term $\frac{\lambda_0}{2N_B}\lVert\theta - \theta^\star_A\rVert^2$ from the prior, a weak spring of equal stiffness on every weight that pulls toward $\theta^\star_A$ (not toward zero). EWC usually drops it. The derivation below proves the result in four steps.

∂ From Bayes rule to the EWC penalty, step by step

Step 1, Bayes over tasks. Conditional independence means $p(\mathcal{D}_A, \mathcal{D}_B \mid \theta) = p(\mathcal{D}_A \mid \theta)\,p(\mathcal{D}_B \mid \theta)$ . Then

p(\theta \mid \mathcal{D}_A, \mathcal{D}_B) = \frac{p(\mathcal{D}_B \mid \theta)\,p(\mathcal{D}_A \mid \theta)\,p(\theta)}{p(\mathcal{D}_A, \mathcal{D}_B)} = \frac{p(\mathcal{D}_B \mid \theta)\,p(\theta \mid \mathcal{D}_A)\,p(\mathcal{D}_A)}{p(\mathcal{D}_B \mid \mathcal{D}_A)\,p(\mathcal{D}_A)},

using $p(\mathcal{D}_A \mid \theta)p(\theta) = p(\theta \mid \mathcal{D}_A)p(\mathcal{D}_A)$ in the numerator and $p(\mathcal{D}_A, \mathcal{D}_B) = p(\mathcal{D}_B \mid \mathcal{D}_A)p(\mathcal{D}_A)$ in the denominator. Taking logs,

\log p(\theta \mid \mathcal{D}_A, \mathcal{D}_B) = \log p(\mathcal{D}_B \mid \theta) + \log p(\theta \mid \mathcal{D}_A) - \log p(\mathcal{D}_B \mid \mathcal{D}_A).

The last term does not depend on $\theta$ . The EWC paper wrote $\log p(\mathcal{D}_B)$ there. Huszár (2018) pointed out that the two differ, because the tasks are independent only given $\theta$ , but the term is a constant either way. Everything task A contributes now sits in $\log p(\theta \mid \mathcal{D}_A)$ .

Step 2, Laplace. Write $g(\theta) = -\log p(\theta \mid \mathcal{D}_A)$ . At the mode $\theta^\star_A$ its gradient vanishes. Its Hessian $\boldsymbol\Lambda_A = \nabla^2 g(\theta^\star_A)$ is positive semidefinite, meaning $\mathbf{u}^\top\boldsymbol\Lambda_A\mathbf{u} \ge 0$ for every direction $\mathbf{u}$ , because $g$ curves upward in every direction at a minimum. Taylor's theorem to second order gives

g(\theta) \approx g(\theta^\star_A) + \tfrac12(\theta - \theta^\star_A)^\top\boldsymbol\Lambda_A(\theta - \theta^\star_A).

So $p(\theta \mid \mathcal{D}_A)$ is approximately the Gaussian $\mathcal{N}(\theta^\star_A, \boldsymbol\Lambda_A^{-1})$ . Its mean is the mode, and its precision is the curvature.

Step 3, what the precision is made of. Up to a constant, $g$ is $-\sum_{n \in A}\log p(y_n \mid \mathbf{x}_n, \theta) - \log p(\theta)$ , and the prior's negative log-density is $\tfrac{\lambda_0}{2}\lVert\theta\rVert^2$ plus a constant. So

\boldsymbol\Lambda_A = \sum_{n \in A}\nabla^2\big[-\log p(y_n \mid \mathbf{x}_n, \theta)\big]\Big|_{\theta^\star_A} + \lambda_0\mathbf{I} = N_A\,\hat{\mathbf{H}}_A + \lambda_0\mathbf{I},

with $\hat{\mathbf{H}}_A$ the average per-example Hessian. The next subsection proves that the expectation of this Hessian, with labels drawn from the model, is the Fisher information $\mathbf{F}_A$ . The Fisher needs only first derivatives and is always positive semidefinite, so EWC uses $\boldsymbol\Lambda_A \approx N_A\mathbf{F}_A + \lambda_0\mathbf{I}$ .

Step 4, the objective. Substituting into Step 1 and negating, the MAP estimate after B minimizes

-\log p(\mathcal{D}_B \mid \theta) + \tfrac12(\theta - \theta^\star_A)^\top\big(N_A\mathbf{F}_A + \lambda_0\mathbf{I}\big)(\theta - \theta^\star_A).

Divide by $N_B$ , so the first term becomes the mean loss $\mathcal{L}_B$ . Keep only the diagonal of $\mathbf{F}_A$ , because a $P\times P$ matrix for a model with $P$ parameters cannot be stored and its diagonal can. The result is the boxed objective with $\lambda = N_A/N_B$ .

In plain words, EWC trains on B while pretending that A's whole posterior is a Gaussian centered on A's solution. Each weight's spring is as stiff as A's certainty about that weight. Four approximations separate this from exact Bayesian learning. Each one is a reason to tune $\lambda$ on validation data instead of setting it to $N_A/N_B$ .

The Laplace Gaussian is a local picture of a posterior that is far from Gaussian in a deep network.
The Fisher equals the Hessian only on average over labels drawn from the model, as the next subsection proves.
The diagonal ignores every correlation between weights.
Stochastic gradient descent stops at a point that is not exactly a mode.

Huszár also points out that the original paper's strength $\lambda$ for each task plays the role of the dataset size $N_A$ in this derivation. The widget at the end of this part shows, in two dimensions, how much each approximation costs.

The Fisher information is the expected Hessian

The derivation swapped the Hessian of the loss for a matrix called the Fisher information. Think of a classifier that outputs a probability for each label. The score of a label is the gradient of its log-probability with respect to the weights, $\mathbf{s}(y) = \nabla_\theta\log p_\theta(y \mid \mathbf{x})$ . It says which way to move the weights to make that label more likely. The Fisher information $\mathbf{F}(\mathbf{x})$ at input $\mathbf{x}$ is the average outer product of the score with itself, $\mathbb{E}_{y}\big[\mathbf{s}(y)\mathbf{s}(y)^\top\big]$ . The average is over labels drawn from the model's own predicted distribution, not taken from the data. Over a dataset of $N$ inputs, $\mathbf{F} = \frac1N\sum_n\mathbf{F}(\mathbf{x}_n)$ .

The identity that makes the swap legitimate takes three lines to prove. §14.4 proved it inside its derivation of why the KL to the base model is a Fisher quadratic. Here it is on its own, because EWC rests on it.

∴ The Fisher information is the expected Hessian of the negative log-likelihood

Let $p_\theta(y \mid \mathbf{x})$ be a model over a finite set of outputs, twice differentiable in $\theta$ , with score $\mathbf{s}(y)$ . Then, with $y \sim p_\theta(\cdot \mid \mathbf{x})$ ,

\mathbb{E}\big[\mathbf{s}(y)\big] = \mathbf{0}, \qquad \mathbf{F}(\mathbf{x}) = \mathbb{E}\big[-\nabla^2_\theta\log p_\theta(y \mid \mathbf{x})\big],

and $\mathbf{F}$ is positive semidefinite.

Proof. Write $p$ for $p_\theta(y \mid \mathbf{x})$ . Since $\nabla\log p = \nabla p/p$ , the mean score is $\sum_y p\,\nabla p/p = \sum_y\nabla p = \nabla\sum_y p = \nabla 1 = \mathbf{0}$ . Differentiating once more with the quotient rule gives $\nabla^2\log p = \nabla^2 p/p - \mathbf{s}\mathbf{s}^\top$ . The first term has mean $\sum_y\nabla^2 p = \nabla^2 1 = \mathbf{0}$ , so $\mathbb{E}[-\nabla^2\log p] = \mathbb{E}[\mathbf{s}\mathbf{s}^\top]$ . Each $\mathbf{s}\mathbf{s}^\top$ is positive semidefinite, and so is any average of them. $\blacksquare$

§15.6 met the same matrix in another role, as the curvature of the KL divergence between an old policy and a new one. One prediction worked by hand shows the identity at work.

✏️ The Fisher of one confident prediction

Take a logistic regression with one input $x = 1$ and one weight $w$ . It predicts label 1 with probability $p = \sigma(w) = 0.9$ , where $\sigma$ is the sigmoid. The log-probability of label 1 is $\log\sigma(w)$ , and of label 0 is $\log(1 - \sigma(w))$ . Their derivatives with respect to $w$ are the scores, $1 - p = 0.1$ for label 1 and $-p = -0.9$ for label 0.

Draw the label from the model. The mean score is $0.9\cdot 0.1 + 0.1\cdot(-0.9) = 0$ . The mean squared score, the Fisher, is $0.9\cdot 0.01 + 0.1\cdot 0.81 = 0.09$ . The second derivative of $-\log p$ with respect to $w$ is $p(1 - p) = 0.09$ for either label. So the Fisher equals the curvature, and it needed only first derivatives.

Two consequences of the identity matter for EWC. First, the Fisher needs only gradients, one per sampled label, and it is positive semidefinite everywhere. The actual Hessian of a deep network's loss can have negative eigenvalues away from a minimum. A spring with negative stiffness would push the weights away instead of holding them.

Second, the identity is about the expected Hessian, with labels drawn from the model. With the dataset's real labels, the Hessian differs from $\mathbf{F}$ by an extra term. That term averages to zero over model-drawn labels, but it need not be zero on a given dataset. For logistic regression the extra term is exactly zero, as in the example, so there the Fisher equals the Hessian at every $\theta$ .

🔬 The Fisher is the Gauss-Newton part of the Hessian

Take a network whose outputs are the logits $\mathbf{f}_\theta(\mathbf{x})$ of a softmax, the scores before the softmax turns them into probabilities. Write $\mathbf{J}$ for the Jacobian of the logits with respect to $\theta$ , and $\mathbf{p}$ for the softmax output. Applying the chain rule twice to the cross-entropy gives

\nabla^2_\theta\big[-\log p(y_n \mid \mathbf{x}_n)\big] = \underbrace{\mathbf{J}^\top\big(\operatorname{diag}(\mathbf{p}) - \mathbf{p}\mathbf{p}^\top\big)\mathbf{J}}_{\text{the Fisher of this example, free of } y_n} + \sum_c (p_c - \mathbb{1}[c = y_n])\,\nabla^2_\theta f_c .

So the Fisher is exactly the part of the Hessian that ignores the network's own curvature, the part called the generalized Gauss-Newton matrix. The remainder is weighted by the residuals $p_c - \mathbb{1}[c = y_n]$ , which are small where the model fits. For logistic regression the logits are linear in $\theta$ , so $\nabla^2 f = \mathbf{0}$ and the remainder vanishes.

The true Fisher and the empirical Fisher

§14.4 met this difference at a softmax output layer, where the empirical Fisher of a rare class came out 99 times too small. Here we see why it happens in general and measure it in every direction at once. Computing $\mathbf{F}$ needs a label sampled from the model for each input. Many implementations of EWC, and of optimizers built on the Fisher, use the dataset's own labels instead. That gives the empirical Fisher $\tilde{\mathbf{F}} = \frac1N\sum_n\mathbf{s}(y_n)\mathbf{s}(y_n)^\top$ , where $y_n$ is the dataset's label for input $n$ . Kunstner, Hennig, and Balles (2019) showed that this is a different matrix with different behavior. It is close to $\mathbf{F}$ only when the model is well specified, meaning the data really come from a model of this form, and the weights are near a good fit.

Binary logistic regression with many inputs makes the difference concrete. The model predicts $p_n = \sigma(\mathbf{w}^\top\mathbf{x}_n)$ , with $\mathbf{w}$ the weights. The score of label $y_n$ is $(y_n - p_n)\mathbf{x}_n$ , so

\mathbf{F} = \frac1N\sum_n p_n(1-p_n)\,\mathbf{x}_n\mathbf{x}_n^\top, \qquad \tilde{\mathbf{F}} = \frac1N\sum_n (y_n - p_n)^2\,\mathbf{x}_n\mathbf{x}_n^\top .

In plain words, the true Fisher weights each example by how uncertain the model is about it. The empirical Fisher weights it by how wrong the model was. Take the example from before, with $p_n = 0.9$ . The true Fisher gives it weight $0.9\times0.1 = 0.09$ . If its label is $y_n = 1$ , the empirical Fisher gives it $(1 - 0.9)^2 = 0.01$ , nine times less. If its label is a mistake, $y_n = 0$ , the empirical Fisher gives it $0.81$ , nine times more. The two agree on average only when the labels really come from the model, because $\mathbb{E}_{y\sim\text{Bern}(p)}[(y - p)^2] = p(1-p)$ .

On clean, nearly separable data almost every example is like the first case, so the empirical Fisher is too small. The cell measures the gap in every direction at once. For each direction $\mathbf{u}$ it takes the ratio $\mathbf{u}^\top\tilde{\mathbf{F}}\mathbf{u}/\mathbf{u}^\top\mathbf{F}\mathbf{u}$ . It reports the smallest and the largest ratio, which are the generalized eigenvalues of the pair of matrices. It also checks that the true Fisher matches a Hessian computed by finite differences.

Python · cell 2

The true Fisher matches the Hessian to finite-difference precision in all three cases, as the identity says it must for logistic regression. On clean labels the empirical Fisher is ten times smaller than the true one along one direction. An EWC spring built from it would let the weights drift that way ten times more cheaply than they should. With 10 percent of labels flipped, it is 13 percent too small along one direction and 29 percent too large along another. In neither case is it the curvature.

Core

True Fisher, empirical Fisher, and the Hessian of logistic regression

The model is binary logistic regression, with inputs X of shape (n, d), labels y in $\{0, 1\}$ , and weights w of shape (d,). Write fisher_matrices(X, y, w). It returns (F_true, F_emp, H), each of shape (d, d).

F_true is the Fisher information averaged over the inputs, with labels drawn from the model.
F_emp is the empirical Fisher. It averages, over the examples, the outer product of each example's score computed with its own label.
H is the Hessian of the mean negative log-likelihood $\frac1n\sum_i\big[\log(1 + e^{\mathbf{w}^\top\mathbf{x}_i}) - y_i\mathbf{w}^\top\mathbf{x}_i\big]$ , derived by hand.

The tests check H against finite differences. They check that F_true equals H for this model. They also check that F_emp averages to F_true when the labels are sampled from the model.

import numpy as np

def fisher_matrices(X, y, w):
    raise NotImplementedError

Tests, hints, verified solutions, and AI help, with Pro at $7.99 a month.

⚠️ Two ways to get the Fisher wrong

The diagonal Fisher is the mean over examples of the squared per-example gradients, $F_i = \frac1N\sum_n(\partial_i\log p(y_n\mid\mathbf{x}_n))^2$ . Squaring the minibatch-mean gradient instead, $(\frac1N\sum_n\partial_i\log p)^2$ , gives a number near zero at any optimum, because the mean gradient is zero there. The spring then disappears exactly where it is needed. Exercise 14.4c has a test that catches this mistake.

The second mistake is quieter. Using the dataset's labels gives the empirical Fisher, which is not the curvature on well-fit data. It can be several times too small in some directions and too large in others. Sample $y \sim p_\theta(\cdot \mid \mathbf{x})$ instead. For a softmax you can also compute the expectation over all classes exactly, as the simulation below does.

The widget fits task A, a logistic regression on 200 points in two dimensions, and draws how fast A's loss rises around its fitted weights. The shading is the true rise. The dashed ellipses are the spring that the full true Fisher would give, at loss rises of 0.02 and 0.08 nats. The solid ones are the spring EWC actually uses, built from the Fisher you choose. Task B is a round bowl, and the widget trains on it from A's best weights with and without the spring, with $\lambda = 1$ . The sliders set how correlated A's inputs are and how many of A's labels are flipped.

Many tasks need one anchor, not many

With a third task, the tempting extension of EWC adds one spring per old task. Each spring is anchored at the weights where its task finished. The original EWC paper suggested exactly that. Huszár (2018) showed that the Bayesian derivation gives something else. After tasks A and B, the Gaussian picture of the posterior has a new peak, the weights after B. Its precision is A's plus B's. So the next task needs one spring, anchored at the latest weights, with the summed stiffness. Here is the difference with real numbers.

✏️ One anchor against one per task, in one dimension

Three tasks share one weight $\theta$ . Task $j$ has loss $\tfrac12(\theta - m_j)^2$ , with optima $m_1 = 0$ , $m_2 = 3$ , and $m_3 = 6$ , and every curvature equal to 1. The best single weight for all three is their mean, $\theta = 3$ .

One anchor. After task 1, $\theta = 0$ with precision 1. Task 2 minimizes $\tfrac12(\theta - 3)^2 + \tfrac12\theta^2$ , so $\theta = 1.5$ , and the precision becomes $1 + 1 = 2$ . Task 3 minimizes $\tfrac12(\theta - 6)^2 + \tfrac12\cdot2\,(\theta - 1.5)^2$ , so $\theta = (6 + 3)/3 = 3$ . That is the joint optimum.

One spring per task. Task 3 minimizes $\tfrac12(\theta - 6)^2 + \tfrac12\theta^2 + \tfrac12(\theta - 1.5)^2$ , so $\theta = (6 + 0 + 1.5)/3 = 2.5$ . The anchor at 1.5 was already a compromise that included task 1. So task 1 pulls twice, once through its own anchor and once through task 2's. The total of the three task losses is 9.375 instead of the optimal 9.

The single anchor was not a lucky choice of numbers. The same bookkeeping works for any number of tasks in any dimension, and when every loss is exactly quadratic and each spring is its exact curvature, it lands on the joint optimum every time.

∴ The Laplace recursion for many tasks

Suppose that after tasks $1$ to $t$ the posterior is approximated by $\mathcal{N}(\theta^\star_t, \boldsymbol\Lambda_{1:t}^{-1})$ . Then learning task $t+1$ minimizes

-\log p(\mathcal{D}_{t+1} \mid \theta) + \tfrac12(\theta - \theta^\star_t)^\top\boldsymbol\Lambda_{1:t}(\theta - \theta^\star_t),

a single penalty anchored at the latest mode. The precision updates as $\boldsymbol\Lambda_{1:t+1} = \boldsymbol\Lambda_{1:t} + N_{t+1}\mathbf{F}_{t+1}$ , with $\mathbf{F}_{t+1}$ evaluated at the new mode $\theta^\star_{t+1}$ .

Suppose further that the prior is flat and every task's negative log-likelihood is exactly quadratic, $\tfrac12(\theta - \mathbf{m}_j)^\top\mathbf{H}_j(\theta - \mathbf{m}_j)$ up to a constant, with $N_j\mathbf{F}_j = \mathbf{H}_j$ (true for a linear model with Gaussian noise of fixed variance). Then the recursion is exact, and $\theta^\star_t$ is the joint minimizer of all the tasks seen so far.

∂ Proof of the Laplace recursion

Step 1 of the EWC derivation, with $\mathcal{D}_{1:t}$ in the role of $\mathcal{D}_A$ , gives $\log p(\theta \mid \mathcal{D}_{1:t+1}) = \log p(\mathcal{D}_{t+1}\mid\theta) + \log p(\theta \mid \mathcal{D}_{1:t}) + \text{const}$ . Substituting the Gaussian approximation gives the objective. Its Hessian at the new mode is $N_{t+1}$ times task $t+1$ 's average Hessian plus $\boldsymbol\Lambda_{1:t}$ , and replacing that average Hessian by the Fisher gives the update.

For the quadratic case, induct on $t$ . Suppose $\tfrac12(\theta - \theta^\star_t)^\top\boldsymbol\Lambda_{1:t}(\theta - \theta^\star_t)$ equals $\sum_{j\le t}\tfrac12(\theta - \mathbf{m}_j)^\top\mathbf{H}_j(\theta - \mathbf{m}_j)$ up to a constant. This holds at $t = 1$ with $\theta^\star_1 = \mathbf{m}_1$ and $\boldsymbol\Lambda_{1:1} = \mathbf{H}_1$ . Then the objective for task $t+1$ is the sum over $j \le t+1$ up to a constant. That sum is a quadratic with Hessian $\sum_{j\le t+1}\mathbf{H}_j = \boldsymbol\Lambda_{1:t+1}$ , minimized at $\theta^\star_{t+1}$ , and a quadratic is determined up to a constant by its Hessian and its minimizer. $\blacksquare$

Why does the one-spring-per-task form go wrong? Its penalty is $\sum_{j\le t}\tfrac12(\theta - \theta^\star_j)^\top N_j\mathbf{F}_j(\theta - \theta^\star_j)$ . It has the right total precision but the wrong center. Its minimizer is a precision-weighted average of the old modes $\theta^\star_j$ . Each $\theta^\star_j$ already included tasks $1$ to $j-1$ , so early tasks are counted again in every later anchor. Huszár calls this double counting. The single-anchor form is also cheaper. It stores one anchor and one precision vector, instead of a pair per task.

Online EWC (Schwarz et al. 2018) adds one more setting. It keeps the single anchor. Before adding the new task's Fisher, it shrinks the accumulated precision by a factor $\gamma_{\text{ewc}}$ ,

\boldsymbol\Lambda \leftarrow \gamma_{\text{ewc}}\boldsymbol\Lambda + \mathbf{F}_t, \qquad 0 < \gamma_{\text{ewc}} \le 1,

with the dataset size $N_t$ absorbed into $\lambda$ . The paper's penalty also multiplies the stored precision by $\gamma_{\text{ewc}}$ , a constant factor that $\lambda$ absorbs too. With $\gamma_{\text{ewc}} = 1$ this is the Laplace recursion. With $\gamma_{\text{ewc}} < 1$ , old tasks fade gradually instead of being held forever. That gives back the plasticity that an ever-growing precision takes away. It is also a rough correction for staleness. An early task's Fisher was computed at an early mode, and it stops describing the curvature as the weights move.

Interview

Online EWC across three tasks

Build the single-anchor form of online EWC with a diagonal precision. The state is None before any task. After a task it is a dict with keys "anchor" and "precision", both arrays of the parameter shape. Write two functions.

ewc_state_update(state, theta_star, fisher, gamma) is called after each task with that task's final weights and diagonal Fisher. It returns the new state. The anchor is theta_star. The precision is $\gamma_{\text{ewc}}\boldsymbol\Lambda + \mathbf{F}_t$ , or just $\mathbf{F}_t$ after the first task. Copy the arrays, so that later in-place changes to the weights cannot alter the state.
ewc_penalty_grad(theta, state, lam) returns the gradient of $\frac{\lambda}{2}\sum_i\Lambda_i(\theta_i - a_i)^2$ , where $\mathbf{a}$ is the anchor. It returns zeros if state is None.

The setup defines three quadratic tasks and a train_sequence loop that calls your two functions. With $\gamma_{\text{ewc}} = 1$ and $\lambda = 1$ the tests expect the exact Laplace recursion, which lands on the joint minimizer of all three tasks.

import numpy as np

def ewc_state_update(state, theta_star, fisher, gamma):
    return {"anchor": np.zeros_like(theta_star), "precision": np.asarray(fisher, float).copy()}

def ewc_penalty_grad(theta, state, lam):
    return np.zeros_like(theta)

Tests, hints, verified solutions, and AI help, with Pro at $7.99 a month.

In the quadratic tasks of that exercise, plain sequential training ends at task 3's optimum. Its BWT is $-5.09$ , with the negative loss as the score in place of accuracy. The single-anchor recursion with $\lambda = 1$ lands exactly where joint training would, with BWT $-2.27$ . Some backward loss remains even for the exact posterior. These three tasks conflict, so the joint optimum is a compromise. Real networks are not quadratic, which is why the simulation later in this section is less kind to EWC.

🔬 Synaptic intelligence, importance measured along the way

A different way to measure each weight's importance avoids the separate Fisher pass. Synaptic intelligence (Zenke, Poole, and Ganguli 2017) keeps a running total, during training on each task, of how much each weight's movement reduced the loss. That total is $\omega_i = -\sum_{\text{steps}} g_i\,\Delta\theta_i$ , with $g_i$ the gradient and $\Delta\theta_i$ the step, a path integral of the gradient along the training trajectory. It divides $\omega_i$ by the square of the weight's net change over the task, $(\Delta_i)^2$ , plus a small constant $\xi$ . The stiffness of weight $i$ in an EWC-style quadratic penalty is the sum of these ratios over all earlier tasks, and the penalty is anchored at the weights where the last task ended. A weight that moved a little and bought a large drop in loss counts as important. Like EWC it is a diagonal quadratic penalty, so it shares EWC's limits.

Protecting outputs instead of weights

EWC protects the weights. A different idea protects what the network says. Before training on B, record what the old model predicts on B's inputs. Then train on B while asking the network to keep making those predictions. Li and Hoiem (2016) called this Learning without Forgetting (LwF). It is knowledge distillation from the old model into the new one, that is, training one model to match another model's output probabilities (§20.4).

To measure "keep making those predictions" we use the KL divergence (§0.5). For two distributions $p$ and $q$ , $D_{\mathrm{KL}}(p\,\Vert\,q) = \mathbb{E}_{y\sim p}[\log p(y) - \log q(y)]$ says how much more surprised $q$ is than $p$ by samples from $p$ . It is zero only when the two agree. Run the old model $\theta^\star_A$ on B's inputs. Record its predicted distribution $p_{\theta^\star_A}(\cdot \mid \mathbf{x})$ over task A's outputs. Then train on

\mathcal{L}_B(\theta) + \frac{\lambda_o}{N_B}\sum_{n \in B} D_{\mathrm{KL}}\big(p_{\theta^\star_A}(\cdot \mid \mathbf{x}_n)\,\Vert\,p_\theta(\cdot \mid \mathbf{x}_n)\big),

where the sum runs over task B's inputs only and $\lambda_o$ sets the balance between the two terms.

In plain words, keep saying what you used to say, about the inputs you are seeing now. That needs no old data at all, only the old model's outputs on new inputs. §14.4 expanded this KL to second order when it asked why fine-tuning forgets. The same expansion shows exactly what LwF protects.

∴ Output distillation is a full-Fisher penalty, to second order

Write $\Delta\theta = \theta - \theta^\star_A$ and let $\mathbf{F}(\mathbf{x})$ be the Fisher information of the old model at input $\mathbf{x}$ . Then

D_{\mathrm{KL}}\big(p_{\theta^\star_A}(\cdot\mid\mathbf{x})\,\Vert\,p_\theta(\cdot\mid\mathbf{x})\big) = \tfrac12\Delta\theta^\top\mathbf{F}(\mathbf{x})\Delta\theta + O(\lVert\Delta\theta\rVert^3).

So for small moves the LwF term is the quadratic penalty $\tfrac{\lambda_o}{2}\Delta\theta^\top\mathbf{F}_{B}\Delta\theta$ , where $\mathbf{F}_B = \frac{1}{N_B}\sum_{n\in B}\mathbf{F}(\mathbf{x}_n)$ is the full Fisher of the old model averaged over task B's inputs.

Proof. This is the expansion of §14.4, restated for the old model. The KL is $\mathbb{E}_{y\sim p_{\theta^\star_A}}[\log p_{\theta^\star_A}(y) - \log p_\theta(y)]$ . Expand $\log p_\theta$ around $\theta^\star_A$ to second order. The zeroth-order terms cancel. The first-order term is minus the mean score times $\Delta\theta$ , which is zero because the mean score is zero (the Fisher identity above). The second-order term is $\tfrac12\Delta\theta^\top\,\mathbb{E}[-\nabla^2\log p]\,\Delta\theta$ , and by the same identity that expected Hessian is the Fisher. $\blacksquare$

Compare the two penalties. EWC uses the diagonal of the Fisher over task A's inputs. LwF keeps every correlation that EWC throws away. But it measures sensitivity on the new task's inputs, not the old task's. When B's inputs resemble A's, this costs little. When they do not, LwF fails. It pins the function down where task A never looks and leaves it free where task A does look.

Both sides of that trade-off show up in experiments. Van de Ven and Tolias found LwF ahead of every weight penalty on split MNIST, whose tasks share pen strokes. In the domain-incremental scenario it reached 71.5 percent, against 64.0 for EWC. On permuted MNIST, whose tasks' inputs are unrelated, LwF reached 69.8 percent in the task-incremental scenario. That is below the 81.8 percent of training with no protection at all, while EWC reached 94.7. Li and Hoiem saw the same failure when they trained on ImageNet and then on MNIST. Old-task accuracy fell substantially, though less than with plain fine-tuning.

The same penalty runs through post-training. The KL term of RLHF (§16.4) keeps the policy near its reference model on the prompts it trains on. It uses the reverse direction, $D_{\mathrm{KL}}(\pi_\theta\Vert\pi_{\text{ref}})$ , averaged over the policy's own samples. InstructGPT's extra term, which adds the pretraining loss to the RL objective, is replay. Shenfeld et al. (2025) note that practitioners have seen the KL term reduce forgetting. Their result, later in this section, explains why. The best predictor of forgetting they found is LwF's penalty itself, the KL from the base model to the fine-tuned one on the new task's inputs.

Replay with a buffer that stays fair

The most reliable defense against forgetting is also the least clever. Keep some old examples and mix them into training. §14.4 found it the most effective remedy in a real fine-tune. Rolnick et al. (2019) found the same in reinforcement learning, on Atari games and on DMLab, a suite of 3D navigation and puzzle tasks. They replayed past experience and also trained the policy to match the old policy's action probabilities on it. That cut forgetting as well as or better than EWC and other methods that must be told where one task ends and the next begins. A buffer holding 50 million of the 900 million frames seen, kept as a uniform random sample, worked almost as well as one holding half of them.

That last detail raises a real question. The stream is long and the buffer holds $k$ items. Which items should it keep, so that it is always a fair sample of everything seen so far, without knowing how long the stream will be? Keeping the first $k$ ignores everything later. Keeping the last $k$ , a first-in first-out queue, forgets the past, which is the failure we are trying to fix. The answer is reservoir sampling, Algorithm R in Vitter (1985), which he credits to Alan Waterman.

The rule is short. Keep the first $k$ items. When item $i$ arrives, with $i > k$ , keep it with probability $k/i$ , in a slot chosen uniformly at random, and evict what was there. Here it is on the smallest stream that shows something.

The early items had more chances to be evicted, and the late items had less chance to enter. The two effects cancel exactly, for every stream length.

∴ Reservoir sampling is uniform

Keep the first $k$ items. When item $i > k$ arrives, draw $j$ uniformly from $\{1, \dots, i\}$ . If $j \le k$ , put item $i$ in slot $j$ , evicting what was there. Then after $n \ge k$ items, every item seen is in the buffer with probability exactly $k/n$ . In fact every set of $k$ items is equally likely to be the buffer.

Proof of the marginal, by induction on $n$ . At $n = k$ every item is in the buffer, with probability $1 = k/k$ . Suppose each of the first $n$ items is in the buffer with probability $k/n$ . Item $n+1$ enters with probability $k/(n+1)$ . An item already in the buffer is evicted only if item $n+1$ enters and picks its slot. Given that $j \le k$ , the slot $j$ is uniform over the $k$ slots, so the eviction probability is $\frac{k}{n+1}\cdot\frac1k = \frac{1}{n+1}$ , whatever else the buffer holds. So each old item survives with probability $\frac{k}{n}\big(1 - \frac{1}{n+1}\big) = \frac{k}{n}\cdot\frac{n}{n+1} = \frac{k}{n+1}$ , the same as the new item. $\blacksquare$

∂ Every k-subset is equally likely

Suppose after $n$ items every $k$ -subset of $\{1, \dots, n\}$ is the buffer with probability $1/\binom{n}{k}$ , which is true at $n = k$ . Take any $k$ -subset $S'$ of $\{1, \dots, n+1\}$ .

If $n+1 \notin S'$ , the buffer must already have been $S'$ , and item $n+1$ must have been rejected. So $\mathbb{P}(S') = \frac{1}{\binom{n}{k}}\cdot\frac{n+1-k}{n+1}$ . Since $\binom{n+1}{k} = \binom{n}{k}\frac{n+1}{n+1-k}$ , this is $1/\binom{n+1}{k}$ .

If $n+1 \in S'$ , write $S' = T\cup\{n+1\}$ with $|T| = k-1$ . The previous buffer must have been $T\cup\{x\}$ for one of the $n-k+1$ items $x \in \{1,\dots,n\}\setminus T$ , and item $n+1$ must have entered and landed on $x$ 's slot, with probability $\frac{k}{n+1}\cdot\frac{1}{k} = \frac{1}{n+1}$ . So $\mathbb{P}(S') = \frac{n-k+1}{\binom{n}{k}(n+1)} = \frac{1}{\binom{n+1}{k}}$ by the same identity.

The cell runs the four-item stream 40,000 times and counts how often each item ends in the buffer.

Python · cell 3

Reservoir sampling is a classic interview question in its own right, and the follow-up is always the proof by induction above. It needs $O(k)$ memory and one random number per item. In a continual learner, the buffer it keeps is a fair sample of every task seen so far, in proportion to each task's size.

Interview

A replay buffer with reservoir sampling

Write class Reservoir with three methods, __init__(self, k, seed), add(self, item), and sample(self, m).

Keep the stored items in a list attribute buffer, at most k long. Keep the number of items seen so far in n_seen.
Use np.random.default_rng(seed) for all randomness.
sample(m) returns m distinct stored items, chosen uniformly without replacement. It raises ValueError if m exceeds the number stored.

The tests feed many seeded streams and check that every position in the stream is kept with probability $k/n$ .

import numpy as np

class Reservoir:
    def __init__(self, k, seed=0):
        self.k, self.buffer, self.n_seen = k, [], 0
        self.rng = np.random.default_rng(seed)

    def add(self, item):                      # FIFO placeholder: keeps only the latest k

Tests, hints, verified solutions, and AI help, with Pro at $7.99 a month.

Replay as a constraint on the gradient

The buffer can also act as a check on each step, rather than as extra training data. Gradient episodic memory (GEM, Lopez-Paz and Ranzato 2017) computes the gradient $\mathbf{g}_k$ of the loss on each old task's buffered examples. It then asks that the update direction $\tilde{\mathbf{g}}$ satisfy $\langle\tilde{\mathbf{g}}, \mathbf{g}_k\rangle \ge 0$ for every old task. A small step along $-\tilde{\mathbf{g}}$ then does not increase any old loss, to first order. GEM finds the $\tilde{\mathbf{g}}$ closest to the current gradient $\mathbf{g}$ by solving a small quadratic program. Averaged GEM (A-GEM, Chaudhry et al. 2019) keeps a single constraint. It uses the gradient $\mathbf{g}_{\text{ref}}$ on a random batch from the whole buffer, and then the problem has a closed-form answer.

✏️ One A-GEM step in two dimensions

Say the new task's gradient is $\mathbf{g} = (1, -3)$ and the replayed batch's gradient is $\mathbf{g}_{\text{ref}} = (1, 1)$ . Their dot product is $1 - 3 = -2$ , which is negative. So a step along $-\mathbf{g}$ would raise the old loss.

A-GEM removes the part of $\mathbf{g}$ that points against $\mathbf{g}_{\text{ref}}$ . That part is $\frac{-2}{\lVert\mathbf{g}_{\text{ref}}\rVert^2}\mathbf{g}_{\text{ref}} = \frac{-2}{2}(1, 1) = (-1, -1)$ . Subtracting it gives $\tilde{\mathbf{g}} = (1, -3) - (-1, -1) = (2, -2)$ .

Now $\tilde{\mathbf{g}}^\top\mathbf{g}_{\text{ref}} = 2 - 2 = 0$ , so to first order the old loss does not change. And $\tilde{\mathbf{g}}^\top\mathbf{g} = 2 + 6 = 8 > 0$ , so the step still lowers the new loss.

The step in the example is the closest vector to $\mathbf{g}$ that does not point against $\mathbf{g}_{\text{ref}}$ , and that holds in any number of dimensions.

∴ The A-GEM projection

Let $\mathbf{g}, \mathbf{g}_{\text{ref}} \in \R^P$ be the flattened gradients of the current task's loss and of the loss on a replayed batch. The problem

\min_{\tilde{\mathbf{g}} \in \R^P}\ \tfrac12\lVert\tilde{\mathbf{g}} - \mathbf{g}\rVert^2 \quad \text{subject to} \quad \tilde{\mathbf{g}}^\top\mathbf{g}_{\text{ref}} \ge 0

has the unique solution

\tilde{\mathbf{g}} = \begin{cases} \mathbf{g}, & \mathbf{g}^\top\mathbf{g}_{\text{ref}} \ge 0, \\[4pt] \mathbf{g} - \dfrac{\mathbf{g}^\top\mathbf{g}_{\text{ref}}}{\lVert\mathbf{g}_{\text{ref}}\rVert^2}\,\mathbf{g}_{\text{ref}}, & \mathbf{g}^\top\mathbf{g}_{\text{ref}} < 0, \end{cases}

and it is still a descent direction for the current task, $\tilde{\mathbf{g}}^\top\mathbf{g} \ge 0$ .

∂ Proof of the A-GEM projection

The objective is strictly convex and the feasible set is a closed half-space, so a minimizer exists and is unique. The constraint is affine, so the KKT conditions, the optimality conditions for a constrained problem, are necessary and sufficient. With multiplier $\mu \ge 0$ for the constraint written as $-\tilde{\mathbf{g}}^\top\mathbf{g}_{\text{ref}} \le 0$ , the Lagrangian is $\tfrac12\lVert\tilde{\mathbf{g}} - \mathbf{g}\rVert^2 - \mu\,\tilde{\mathbf{g}}^\top\mathbf{g}_{\text{ref}}$ . Stationarity gives $\tilde{\mathbf{g}} = \mathbf{g} + \mu\mathbf{g}_{\text{ref}}$ , and complementary slackness gives $\mu\,\tilde{\mathbf{g}}^\top\mathbf{g}_{\text{ref}} = 0$ .

If $\mu = 0$ , then $\tilde{\mathbf{g}} = \mathbf{g}$ , which is feasible exactly when $\mathbf{g}^\top\mathbf{g}_{\text{ref}} \ge 0$ . If $\mu > 0$ , the constraint is tight, $0 = \tilde{\mathbf{g}}^\top\mathbf{g}_{\text{ref}} = \mathbf{g}^\top\mathbf{g}_{\text{ref}} + \mu\lVert\mathbf{g}_{\text{ref}}\rVert^2$ , so $\mu = -\mathbf{g}^\top\mathbf{g}_{\text{ref}}/\lVert\mathbf{g}_{\text{ref}}\rVert^2$ , which is positive exactly when $\mathbf{g}^\top\mathbf{g}_{\text{ref}} < 0$ . If $\mathbf{g}_{\text{ref}} = \mathbf{0}$ , every vector is feasible and the first case applies.

For the descent property, the first case gives $\lVert\mathbf{g}\rVert^2 \ge 0$ . In the second, $\tilde{\mathbf{g}}^\top\mathbf{g} = \lVert\mathbf{g}\rVert^2 - (\mathbf{g}^\top\mathbf{g}_{\text{ref}})^2/\lVert\mathbf{g}_{\text{ref}}\rVert^2 \ge 0$ by Cauchy-Schwarz, with equality only when $\mathbf{g}$ points exactly against $\mathbf{g}_{\text{ref}}$ . $\blacksquare$

In plain words, if the new task's step would hurt the old tasks on average, remove the part of the step that points against them and keep everything else. The step $-\eta\tilde{\mathbf{g}}$ , with $\eta$ the learning rate, changes the replayed loss by $-\eta\,\tilde{\mathbf{g}}^\top\mathbf{g}_{\text{ref}} \le 0$ to first order. So the old loss can rise only at second order in $\eta$ . The guarantee covers the replayed batch's average loss, not each old example. That is why the buffer has to cover the directions the old task uses.

Interview

The A-GEM projection

Write agem_project(g, g_ref). Its inputs are flattened parameter gradients, 1-D arrays of the same length $P$ . Return the solution $\tilde{\mathbf{g}}$ of the theorem as a new float array, and leave both inputs unchanged.

The setup provides train_two_tasks(project). It fits a linear model exactly to task A and then runs gradient descent on task B. At each step it passes two gradients through project, the gradient on B and the gradient on a 20-example replay buffer from A. If project is None it skips that step. It returns the final losses (loss_A, loss_B) and the loss on B before training.

import numpy as np

def agem_project(g, g_ref):
    g, g_ref = np.asarray(g, float), np.asarray(g_ref, float)
    return g - (g @ g_ref) * g_ref                        # placeholder: always projects, wrong scale

Tests, hints, verified solutions, and AI help, with Pro at $7.99 a month.

In that run, plain gradient descent on B raises task A's loss from zero to about 8.8. A-GEM with a 20-example buffer keeps it at 0.03 and still drives B's loss from 19.2 to under 0.01. The buffer works here because task A's inputs span only five directions, and 20 examples cover them. Suppose instead that the buffer misses some directions the old task uses. Then a model with more parameters than it needs can forget the unbuffered examples while the single averaged constraint stays satisfied.

When the data itself cannot be kept, generative replay (Shin et al. 2017) trains a generative model on past data and replays its samples, labeled by the old model. A language model is already a generator, so it can produce its own replay data. The risk is that it replays its own mistakes, a risk that training on self-generated data takes up in general.

Giving each task its own parameters

The third strategy gives each task weights that no other task touches, so there is nothing to forget.

Progressive networks (Rusu et al. 2016) freeze the network after each task and add a new column of layers for the next one. Lateral connections from every earlier column's layers let the new column reuse old features. Old tasks are untouched by construction. The cost is growth. Each task adds a column plus connections from all earlier columns, so the parameter count grows faster than linearly in the number of tasks.
PackNet (Mallya and Lazebnik 2018) works inside one fixed network. After training a task, it prunes a large fraction of the weights that task uses, briefly retrains the rest, and freezes them. The next task trains only in the freed weights. Each task is stored as a binary mask over one set of weights, and capacity eventually runs out.
Per-task adapters. For language models the natural unit is a LoRA adapter per task on a frozen base. LoRA adds to each weight matrix a trainable low-rank correction $\gamma\mathbf{B}\mathbf{A}$ , where $\mathbf{B}$ and $\mathbf{A}$ are thin matrices and $\gamma$ is a fixed scale (§14.5). The base cannot forget. Each adapter costs a fraction of a percent of the parameters, and one server can hold many.

Orthogonal adapters go one step further (Wang et al. 2023, who call the method O-LoRA). A penalty keeps each new task's update in a subspace orthogonal to the earlier tasks' updates. In our notation the penalty is $\lambda_1\sum_{i<t}\lVert\mathbf{B}_i^\top\mathbf{B}_t\rVert_F^2$ , where the columns of $\mathbf{B}$ span the column space of the update. The paper names that matrix $\mathbf{A}$ .

Progressive networks, PackNet, and per-task adapters need to know which task a test input belongs to. Either the task is given, as in the task-incremental scenario, or a router must infer it. That is the price of zero forgetting, and in the class-incremental scenario it is the whole problem. O-LoRA needs no router. Its penalty keeps each task's update in its own directions, so it adds all the updates into the base weights and serves one model. It still needs to know where each task begins during training, and orthogonal updates reduce interference without guaranteeing zero forgetting. One more cheap remedy needs no routing. Interpolate between the base weights and the fine-tuned weights, as in the merging part of §14.4. That gives one dial that trades new-task accuracy for old-task accuracy.

Five tasks in a small simulation

Now all three defenses at once, on a problem small enough to run in your browser and large enough to behave like the real thing. Ten classes of Gaussian blobs in 16 dimensions are split into five two-class tasks. Each test input must be classified among all ten classes, as in the class-incremental version of split MNIST. The model is a one-hidden-layer MLP with a ten-way softmax output. We train it on the tasks in order, 300 full-batch steps each, and fill in the accuracy matrix.

The first cell defines the data, the model, its gradient, and the true diagonal Fisher. The Fisher is computed exactly, by summing over all ten labels weighted by the model's probabilities, rather than by sampling one label.

Python · cell 4

The training loop supports every method of this section.

EWC and online EWC use the single-anchor recursion. The spring is applied as an exact implicit step after each gradient step, $\theta_i \leftarrow (\theta_i + \eta\lambda\Lambda_i a_i)/(1 + \eta\lambda\Lambda_i)$ , with $\eta$ the learning rate and $\mathbf{a}$ the anchor. This step minimizes the penalty plus $\frac{1}{2\eta}\lVert\theta - \theta_{\text{old}}\rVert^2$ . So it is stable for any $\lambda$ , and it avoids the divergence we met in two dimensions.
Replay keeps a 200-example reservoir. It averages the gradients of the current batch and of a replayed batch of 64.
Per-task heads mask the softmax to the current task's two classes, in training and at test time. This is the task-incremental scenario, so it answers an easier question than the other four.

Python · cell 5

Python · cell 6

Python · cell 7

✏️ What the five-task simulation shows

Read the heatmaps row by row. Naive training is catastrophic in the literal sense. Task 1's accuracy falls from 1.00 to 0.02 the moment task 2 is trained. The softmax learns to push down every logit outside the current two classes. At the end the first two tasks are gone entirely and the third is at 0.42 (ACC 0.438, BWT $-0.694$ ).

EWC with $\lambda = 1000$ holds much more (ACC 0.843, BWT $-0.163$ ), but its oldest task still slides to 0.59. Online EWC with $\gamma_{\text{ewc}} = 0.7$ gives up a little more of the old tasks for plasticity (ACC 0.794). Replay is far better (ACC 0.952, BWT $-0.041$ ), with a buffer of only 200 examples, about 20 per class and a fifth of the training data. Per-task heads barely forget at all (ACC 0.990). But they solve the easier task-incremental problem, since each test input arrives with its task.

Forward transfer is $-0.035$ for every method that faces the ten-way choice. Before training on task $j$ , the model has never been asked to output task $j$ 's classes, so it scores exactly 0 on them. That is slightly below the 3.5 percent that the random initialization happened to get, on average, on tasks 2 to 5. In the class-incremental scenario there is nothing to transfer forward. All of this matches van de Ven and Tolias, whose weight penalties failed on class-incremental split MNIST while replay worked.

The stability-plasticity dial shows up in a sweep over $\lambda$ .

Python · cell 8

As $\lambda$ grows, BWT climbs toward zero, from $-0.384$ at $\lambda = 10$ to $-0.020$ at $\lambda = 10^4$ . But at $\lambda = 10^4$ the network can no longer learn the newest task at all (accuracy 0.000 on it). So ACC peaks at an intermediate value, 0.843 at $\lambda = 1000$ , and falls to 0.614 beyond it. A method that reports only BWT can look perfect by refusing to learn. That is the reason to always report ACC beside it.

The widget runs a smaller version of the same experiment live, with a linear softmax classifier in place of the MLP. Its losses are on a different scale, so its best $\lambda$ is near 20 rather than 1000. There a tuned EWC keeps up with a replay buffer of 100 or more examples, both near 89 percent, so how far replay pulls ahead depends on the model. Choose a method and its setting, then press Train and watch the accuracy matrix fill in task by task.

The same tools in PyTorch

Here is a reference implementation of the two tools a practitioner would reach for. One is an EWC penalty with a diagonal Fisher estimated from labels sampled from the model. The other is a reservoir buffer. Both run on a small MLP. The two tasks share the same four outputs. They live in different regions of input space and use different label rules, so one network can solve both. That is the domain-incremental case.

🔥 In PyTorchfor a machine with PyTorch

Naive training on B drops task A from 0.96 to about 0.66. EWC and replay both keep A above 0.92 while learning B to above 0.92. In a real codebase the Fisher loop would compute per-sample gradients in one batched call (torch.func.vmap over torch.func.grad) instead of one backward pass per example. The anchor and the Fisher would be stored in float32 next to the optimizer state.

Keeping a large model current with continual pretraining

For a frontier model the sequence of tasks is a sequence of data. The model was pretrained on a corpus, and next month there is new data, such as newer web text, a new language, or more code. Retraining on the union from scratch is the gold standard, and it costs the whole pretraining budget again. Continual pretraining asks for the same result at the cost of training on the new data only.

The first obstacle is the learning rate. Pretraining usually raises the rate during a short warmup and then lowers it along a cosine curve (§8.7). So the model ends at a tiny learning rate, and continuing there barely moves it. Raising the rate again, called re-warming, lets it adapt. But a high rate is exactly what causes forgetting.

Gupta et al. (2023) measured both effects. They took a 410M-parameter model pretrained on 300B tokens of the Pile, an open corpus of web text, books, and code, and continued it on SlimPajama, another open corpus, mostly web text. Re-warming first raised the loss on both corpora, and in the longer run it beat a model trained from scratch on the new data alone. Ibrahim et al. (2024) then showed that three simple ingredients together come within a few hundredths of a nat of retraining, or match it.

📌 Continual pretraining that nearly matches retraining

Ibrahim et al. (2024) pretrained models on 300B tokens of the Pile. They continued them on one of two new datasets. One was 300B tokens of SlimPajama, a weak shift from English to English. The other was 200B tokens of German web text from Common Crawl, a strong shift. They used three ingredients.

Re-warm the learning rate linearly, over 1 percent of the new steps, to the original peak ( $3\times10^{-4}$ at 405M parameters).
Re-decay it by cosine to a tenth of the peak.
Replay old data in a fixed fraction of each batch, with the total token budget held fixed.

Each number below is the plain mean of two final validation losses, one on the Pile and one on the new dataset. The losses are in nats per token, the cross-entropy with natural logarithms.

Weak shift, 405M. With 5 percent replay the average was 2.37 (Pile 2.23, SlimPajama 2.51). A model retrained on the 600B-token union reached 2.35. Without replay the average was 2.47 (Pile 2.44).
Weak shift, 10B (peak learning rate $1.2\times10^{-4}$ ). With 5 percent replay, 1.89, against 1.87 for the union and 1.99 without replay.
Strong shift, 405M. Without replay the average was 2.34, the mean of a German loss of 1.11 and a Pile loss that had risen from 2.17 to 3.56. With 25 percent replay it was 1.75 (Pile 2.33, German 1.16). That equals the 1.75 of retraining on the 500B-token union (Pile 2.26, German 1.25).

The stronger the shift, the more replay it needs. About 5 percent was enough for newer text in the same language, and 25 percent for a new language. An interviewer who asks how you would keep a model current wants these three ingredients and these replay fractions.

Reinforcement learning forgets less than supervised fine-tuning

How a model is fine-tuned matters too, not only what it is fine-tuned on. §14.4 met the evidence as a warning about supervised fine-tuning. Here we restate it and then explain it. Shenfeld, Pari, and Agrawal (2025) compared supervised fine-tuning (SFT), which trains on fixed target answers, with on-policy reinforcement learning. On-policy RL trains on the model's own sampled answers, scored by a reward, as GRPO does (§18.4). They trained Qwen 2.5 3B-Instruct on new tasks in math, science questions, and tool use, and trained a 7B robot policy as well. Their RL runs used no explicit KL penalty.

📌 Forgetting follows the KL to the base model

At matched performance on the new task, on-policy RL forgot much less of the base model's abilities than SFT (Shenfeld et al. 2025). Across both methods and many hyperparameters, one number predicted forgetting. It is the forward KL from the base policy $\pi_0$ to the fine-tuned policy $\pi$ , averaged over the new task's inputs, $\mathbb{E}_{x}\big[D_{\mathrm{KL}}(\pi_0(\cdot\mid x)\,\Vert\,\pi(\cdot\mid x))\big]$ , which is LwF's penalty.

On a controlled task built from MNIST, they fit each candidate predictor of forgetting with a quadratic. The forward KL reached $R^2 = 0.96$ and the reverse KL 0.93. The Fisher-weighted squared weight change, the quantity EWC penalizes (with the Fisher taken over the previous task's inputs), reached only 0.58, and the L1 weight change 0.34.

The authors call the principle RL's Razor. Among the many policies that solve a new task, on-policy RL prefers the one closest in KL to where it started. The simplest case shows why. Take one prompt, a base policy $\pi_0$ over a finite set of answers, and a reward $R$ that is 1 for a correct answer and 0 otherwise. Sample from $\pi_0$ and keep only the correct answers. This rejection sampling gives the distribution $q(y) = \pi_0(y)R(y)/Z$ , where $Z = \sum_y\pi_0(y)R(y)$ is the base policy's success rate.

✏️ Four answers, two of them correct

A base policy gives four answers probabilities $\pi_0 = (0.15, 0.05, 0.50, 0.30)$ . Only the first two are correct, so the success rate is $Z = 0.2$ . Rejection sampling keeps the first two in their old ratio, $q = (0.75, 0.25, 0, 0)$ . Its distance from the base is $D_{\mathrm{KL}}(q\,\Vert\,\pi_0) = 0.75\log\frac{0.75}{0.15} + 0.25\log\frac{0.25}{0.05} = \log 5 = 1.61$ nats.

Now take an SFT target that always gives the second answer, perhaps because a person wrote it. It also always succeeds. Its distance is $\log\frac{1}{0.05} = 3.00$ nats, almost twice as far. Even an even split of the two correct answers, $(0.5, 0.5, 0, 0)$ , is farther, at 1.75 nats.

The theorem says that no always-correct policy can be closer than the rejection-sampling one.

∴ Rejection sampling is the KL-closest way to always succeed

Among all distributions $q'$ that put all their mass on correct answers, the rejection-sampling distribution $q$ is the closest to $\pi_0$ in KL, and $D_{\mathrm{KL}}(q \,\Vert\, \pi_0) = -\log Z$ .

Proof. Wherever $R(y) = 1$ , which is the only place $q'$ puts mass, $\pi_0(y) = Z\,q(y)$ . So

D_{\mathrm{KL}}(q' \,\Vert\, \pi_0) = \sum_y q'(y)\log\frac{q'(y)}{\pi_0(y)} = \sum_y q'(y)\log\frac{q'(y)}{Z\,q(y)} = D_{\mathrm{KL}}(q' \,\Vert\, q) - \log Z \;\ge\; -\log Z ,

with equality only at $q' = q$ . $\blacksquare$

In the example, $Z = 0.2$ and $-\log 0.2 = 1.61$ , the distance we computed by hand. The closest always-correct policy keeps the base model's relative preferences among the correct answers. On-policy RL reinforces the model's own successful samples, so it moves toward reweighted versions of what the model already does. Shenfeld et al. prove that, in an idealized setting, policy gradient converges to this KL-closest optimal policy. SFT on fixed target answers has no such pull, and it can end arbitrarily far from the base.

Keep the two kinds of claim apart. The predictor is an empirical finding. The convergence is a theorem for an idealized case, not a guarantee for every RL setup. In practice, the KL to the base model on new-task prompts is a cheap early warning of forgetting, worth tracking at every checkpoint.

Learning during a single test input

Everything so far learns between tasks. A different family learns during a single test input. It is the most direct sense in which a model's weights can serve as memory.

📐 Test-time training

Test-time training (TTT) updates some of a model's parameters before predicting on a test input. It takes a few gradient steps on a loss computed from that input alone, then predicts with the updated parameters. Sun et al. (2020) introduced it with a self-supervised side task, predicting how an image was rotated, that shares a feature extractor with the main task. The variants that matter now are these.

On nearest neighbors. Retrieve training examples close to the test input and fine-tune on them before predicting (Hardt and Sun 2023).
On the demonstrations. For a few-shot task, fine-tune a per-task LoRA adapter on the demonstrations, typically as leave-one-out tasks, before answering (Akyürek et al. 2024).
TTT layers. A sequence layer whose hidden state is itself the weights of a small model (Sun et al. 2024). Each token, or mini-batch of tokens, trains that model by one gradient step on a self-supervised loss.
Surprise-gated neural memory. A memory network written at test time by gradient steps with momentum and decay (Behrouz, Zhong, and Mirrokni 2024).

📌 What test-time training has bought

Hardt and Sun (2023) built a distributed index over the Pile. For each test sequence they fine-tuned GPT-2 on its retrieved neighbors, one gradient step each. Training on 50 neighbors cut bits per byte, a length-normalized form of the loss, by 20 percent. Most of the gain came from the first 20 neighbors. It substantially narrowed the gap between the 117M-parameter GPT-2 and the 1.3B-parameter GPT-Neo, a model more than ten times larger.
Akyürek et al. (2024) trained a LoRA adapter per task on leave-one-out versions of the task's demonstrations. Each version holds out one demonstration as the question and trains on the rest. ARC is a benchmark of grid puzzles, each defined by a few examples. There they also augmented the data with invertible transformations, such as rotations, flips, and color permutations. An 8B model reached 53.0 percent on ARC's public validation set. Ensembled with program synthesis it reached 61.9 percent. On BIG-Bench Hard, a set of hard reasoning tasks, 10-shot accuracy rose from 50.5 to 57.8 percent.

Why should training on neighbors help? A global model must compromise between every region of its training data. A local fit near the test input can follow the structure that holds there. That matters most when the test input comes from a region the training set covers thinly.

The cell builds that situation with ridge regression, least squares with a penalty $\alpha\lVert\mathbf{w}\rVert^2$ on the size of the weights. The training set is 92 percent a common regime and 8 percent a rare, shifted regime with a different linear rule. We compare one global ridge model with ridge models fit at test time. Each of those is fit on one test point's 40 nearest training points. We report each approach's mean squared error (MSE), first on test points from the common regime and then on points from the rare one.

Python · cell 9

That is the trade test-time training makes everywhere. It buys accuracy where the input is unusual. It pays in compute at test time, and in higher variance where the input is ordinary.

Research

Test-time training on nearest neighbors

Write ttt_nn_predict(X_train, y_train, X_test, k, alpha). It returns predictions of shape (n_test,). For each row of X_test, do three things.

Find its k nearest training points by Euclidean distance.
Fit ridge regression with an unpenalized intercept on those points only, minimizing $\lVert\mathbf{y} - b\mathbf{1} - \mathbf{X}\mathbf{w}\rVert^2 + \alpha\lVert\mathbf{w}\rVert^2$ .
Predict that test point with the local fit.

The setup provides make_shift_data(seed), which builds the same data as the cell above. It returns a training set with a rare regime, and two test sets, one from the common regime and one from the rare one. The tests compare your predictions with a reference. They also check that with 40 neighbors, test-time training wins clearly on the shifted set and not on the common one.

import numpy as np

def ttt_nn_predict(X_train, y_train, X_test, k, alpha):
    X_train, X_test = np.asarray(X_train, float), np.asarray(X_test, float)
    w = np.linalg.lstsq(X_train, y_train, rcond=None)[0]      # placeholder: one global model, no intercept
    return X_test @ w

Tests, hints, verified solutions, and AI help, with Pro at $7.99 a month.

A layer whose state is a learned memory

Test-time training can also be built into the architecture, so that a layer learns from every token it reads. The idea is close to in-context learning, a model's ability to pick up a task from examples in its prompt. §6.8 gave two readings of it. One is inference over the hidden task that the examples share. The other, in a linear setting, is a step of gradient descent on the examples, carried out by one layer of linear attention. Here the gradient step becomes explicit.

The starting point is linear attention (§21.2). Instead of keeping every past key and value, it keeps one matrix $\mathbf{S}_t \in \R^{d_k\times d_v}$ , a running sum of key-value outer products. At token $t$ it has a key $\mathbf{k}_t$ , a value $\mathbf{v}_t$ , and a query $\mathbf{q}_t$ . It writes $\mathbf{S}_t = \mathbf{S}_{t-1} + \mathbf{k}_t\mathbf{v}_t^\top$ and reads $\mathbf{S}_t^\top\mathbf{q}_t$ .

§21.6 showed that this write is already a step of learning. It is one gradient step, with step size 1, on the loss $-\mathbf{v}_t^\top\mathbf{S}^\top\mathbf{k}_t$ . The delta rule of the DeltaNet layers reads what is stored at a key and writes only the correction, $\mathbf{S}_t = (\mathbf{I} - \beta_t\mathbf{k}_t\mathbf{k}_t^\top)\mathbf{S}_{t-1} + \beta_t\mathbf{k}_t\mathbf{v}_t^\top$ , and that section proved it is one step with rate $\beta_t$ on the regression loss $\tfrac12\lVert\mathbf{S}^\top\mathbf{k}_t - \mathbf{v}_t\rVert^2$ . So these layers already train a linear model on the sequence as they read it, a model that maps each key to its value. What is new here is making that model, and its training, general.

TTT layers (Sun et al. 2024) make that idea general. The state is the weights $\mathbf{W}_t$ of any small model $f$ . It is a linear map in the variant called TTT-Linear and a two-layer MLP in TTT-MLP. Each token trains it by one step on a reconstruction loss,

\mathbf{W}_t = \mathbf{W}_{t-1} - \eta\,\nabla_{\mathbf{W}}\big\lVert f(\boldsymbol\theta_K\mathbf{x}_t; \mathbf{W}_{t-1}) - \boldsymbol\theta_V\mathbf{x}_t\big\rVert^2,

and the layer's output is $f(\boldsymbol\theta_Q\mathbf{x}_t; \mathbf{W}_t)$ . Here $\mathbf{x}_t$ is the token's input vector and $\eta$ the inner learning rate. The matrices $\boldsymbol\theta_K, \boldsymbol\theta_V, \boldsymbol\theta_Q$ are ordinary weights learned in the outer training loop, the usual training of the whole network. Their names echo keys, values, and queries, because they play those roles.

In plain words, the layer keeps a tiny model as its memory. Each token is a training example for that tiny model, with a projection of the token as the input and another projection as the target. The layer answers by running the tiny model on a third projection.

∴ TTT-Linear contains linear attention and the delta rule

Let $f(\mathbf{k}; \mathbf{W}) = \mathbf{W}\mathbf{k}$ , write $\mathbf{k}_t = \boldsymbol\theta_K\mathbf{x}_t$ and $\mathbf{v}_t = \boldsymbol\theta_V\mathbf{x}_t$ , and drop the layer norm and residual connection that the paper wraps around $f$ . The gradient of the inner loss is $2(\mathbf{W}\mathbf{k}_t - \mathbf{v}_t)\mathbf{k}_t^\top$ .

Batch gradient descent from $\mathbf{W}_0 = \mathbf{0}$ with $\eta = \tfrac12$ , every token's gradient taken at $\mathbf{W}_0$ , gives $\mathbf{W}_t = \sum_{s\le t}\mathbf{v}_s\mathbf{k}_s^\top$ , which is linear attention (Theorem 1 of Sun et al. 2024).
Online gradient descent, one step per token at the current weights, gives $\mathbf{W}_t = \mathbf{W}_{t-1}(\mathbf{I} - 2\eta\,\mathbf{k}_t\mathbf{k}_t^\top) + 2\eta\,\mathbf{v}_t\mathbf{k}_t^\top$ , which is the delta rule with $\mathbf{S} = \mathbf{W}^\top$ and $\beta_t = 2\eta$ .

∂ Proof that TTT-Linear contains linear attention and the delta rule

At $\mathbf{W}_0 = \mathbf{0}$ each gradient is $-2\,\mathbf{v}_t\mathbf{k}_t^\top$ , so the batch steps add up to $2\eta\sum_s\mathbf{v}_s\mathbf{k}_s^\top$ , which is $\sum_s\mathbf{v}_s\mathbf{k}_s^\top$ at $\eta = \tfrac12$ . Reading with a query gives $\mathbf{W}_t\mathbf{q}_t = \sum_s\mathbf{v}_s(\mathbf{k}_s^\top\mathbf{q}_t)$ , unnormalized linear attention. For the online case, substitute the gradient at $\mathbf{W}_{t-1}$ into one step, which gives $\mathbf{W}_t = \mathbf{W}_{t-1} - 2\eta(\mathbf{W}_{t-1}\mathbf{k}_t - \mathbf{v}_t)\mathbf{k}_t^\top$ , and transpose. $\blacksquare$

The TTT paper notes in its related work that DeltaNet is TTT-Linear with an inner mini-batch of one token and no layer norm or residual. For speed, TTT layers use a middle ground. They take mini-batches of 16 tokens, whose gradients are all taken at the weights left by the previous mini-batch. At 125M to 1.3B parameters, both TTT layers kept lowering perplexity, the exponential of the average loss per token, as the context grew to 32K tokens, as a Transformer does. Mamba, the state-space model of §21.4, stopped improving after 16K.

The memory in Behrouz, Zhong, and Mirrokni (2024), an architecture called Titans, goes one step further. It is a deep network $\mathcal{M}$ trained to map each key to its value. Its loss is $\ell(\mathcal{M}_{t-1}; \mathbf{x}_t) = \lVert\mathcal{M}_{t-1}(\mathbf{k}_t) - \mathbf{v}_t\rVert_2^2$ , where $\mathbf{k}_t$ and $\mathbf{v}_t$ are linear projections of the input. It writes with momentum and forgetting,

\mathcal{M}_t = (1 - \alpha_t)\,\mathcal{M}_{t-1} + \mathbf{u}_t, \qquad \mathbf{u}_t = \mu_t\,\mathbf{u}_{t-1} - \eta_t\,\nabla\ell(\mathcal{M}_{t-1}; \mathbf{x}_t),

with a forgetting gate $\alpha_t \in [0, 1]$ , momentum $\mu_t$ , and learning rate $\eta_t$ , all computed from the data. In plain words, the memory moves most when a token's value is badly predicted from its key, which the paper calls surprise. The momentum carries the surprise of recent tokens forward. The gate $\alpha_t$ acts as weight decay that frees capacity. Reading is a forward pass $\mathcal{M}_t(\mathbf{q}_t)$ with no update. The paper reports scaling to contexts beyond 2M tokens on needle-in-a-haystack tests, which hide one fact in a long context and ask for it back.

Moving test-time learning into a cache or the weights

Two recent systems take what was learned from one corpus or one passage and store it somewhere lasting.

📌 Learning that outlives a single query

A trained cache. A KV cache is the stored keys and values that a prompt leaves behind in every attention layer (§6.5). Cartridges (Eyuboglu et al. 2025) train a small KV cache offline for one corpus, such as a codebase or a set of legal documents. Later queries load the trained cache instead of putting the whole corpus in context. The training method that worked is self-study. The model generates synthetic conversations about the corpus, and the cache is trained to make the model behave on them as it would with the whole corpus in context. That matched in-context quality with 38.6 times less memory and 26.4 times higher throughput.
Self-written training data. SEAL (Zweiger et al. 2025) lets a model write its own fine-tuning data, called a self-edit, and learns by RL which self-edits help. The test was answering questions about a passage from SQuAD, a reading-comprehension dataset, after the passage was removed from the context. Qwen2.5-7B answered 33.5 percent after fine-tuning on the passage itself. It answered 47.0 percent after fine-tuning on self-edits from the RL-trained model. When edits were applied one after another, accuracy on earlier passages declined steadily, the forgetting this section began with.

Where new knowledge should live

The chapter has offered three homes for a new piece of knowledge. It can go in the context window (§25.1 and §25.2), in an external store (§25.3), or in the weights (this section). Inside the weights there are two routes. Training, by fine-tuning or continual pretraining, changes every weight a little. A targeted edit changes almost nothing.

§23.7 derived two such edits. One writes a single fact as a rank-one update to one MLP matrix (ROME, Meng et al. 2022). The other writes thousands of facts at once, as one update per layer spread over several layers, each of rank at most the number of edits (MEMIT, Meng et al. 2023). The evidence on putting new facts into weights by either route is sobering.

📌 Weights, context, or store

Retrieval beats fine-tuning for facts. Across knowledge-intensive tasks, retrieval-augmented generation consistently beat unsupervised fine-tuning on the documents, for old and new knowledge alike (Ovadia et al. 2023). Fine-tuning helped more when it saw many paraphrases of each fact.
New facts are learned slowly and raise hallucination. Fine-tuning examples with knowledge the model lacked were learned much more slowly than examples consistent with what it knew. As they were learned, the model's tendency to hallucinate rose linearly with them (Gekhman et al. 2024).
Facts need varied forms to be usable. On controlled biographies, questions could extract a fact only if pretraining had seen it in varied forms, such as paraphrases and shuffled sentences. Otherwise the model could memorize a fact and still answer 0 percent of questions about it (Allen-Zhu and Li 2023).
Context generalizes better. With the data matched, in-context learning generalized more flexibly than fine-tuning, for example to the reverse of a stated relation (Lampinen et al. 2025).
Edits miss their implications. If Jack Depp is the son of Johnny Depp, he is also the sibling of Lily-Rose Depp. On 5,000 edits, each tested on such implied facts, ROME, MEMIT, and a learned editor largely failed to make those changes. Putting the new fact in context scored best (Cohen et al. 2023).
Sequential edits collapse the model. Edited one fact after another, models first forget gradually and then collapse. ROME broke down somewhere between about 100 and 1,000 edits, and MEMIT near 4,000 on GPT-2 XL and before 1,500 on GPT-J (Gupta, Rao, and Anumanchipalli 2024).

The table puts the homes side by side, including the trained cache as a fourth home between context and weights. Each row is a trade-off, not a winner.

home	write cost	read cost	freshness and deletion	generalization from the knowledge	forgetting risk
context window (§25.1, §25.2)	none	grows with length, every call	instant, delete by omission	most flexible (Lampinen et al. 2025)	none, but limited by the window
external store (§25.3)	an embedding and a write	retrieval plus context tokens	instant update and exact deletion	as good as what is retrieved	none in the model
trained cache (Cartridges)	an offline training run per corpus	small, fixed	retrain to update	close to in-context	none in the base
adapter per task or user (§14.5)	a fine-tune	near zero	swap or delete the adapter	narrow without augmentation	none in the base
targeted edit (ROME, MEMIT, §23.7)	a rank-one update per fact, or one update of rank at most the batch size per edited layer	zero	subtract the stored update (exact for the latest edit, approximate once later edits were computed on top of it)	reliable for the edited statement and its paraphrases, weak on implied facts	grows with the number of edits, degradation after hundreds to thousands of sequential edits
full weights (fine-tune, continual pretraining)	expensive, needs replay	zero	slow, deletion needs unlearning	flexible only with varied data (Allen-Zhu and Li 2023)	real, needs replay and evaluation
test-time training	gradient steps per query	high	per query, nothing persists	strong on unusual inputs	none if reset after the query

The practical rules follow from the table.

Facts that change, or that a user may ask to delete, belong in context or in a store.
Skills and broad shifts in the data belong in the weights, put there by continual pretraining with replay, or by RL, which forgets less.
Large static corpora that are queried many times are what trained caches are for.
A targeted edit suits a few urgent corrections that must hold on every call. It does not reach implied facts and does not survive thousands of edits, so a store remains the default for facts.
Anything put into weights needs an evaluation for forgetting, with probes of the base model's abilities at every checkpoint, as §14.4 recommended.

Where these methods break

EWC at scale. A diagonal Fisher over a language model's pretraining distribution needs many per-example gradients from that distribution, and the diagonal ignores the correlations that dominate a deep network's curvature. That is why §14.4 found EWC rarely used at scale. The box below adds up the cost.
Replay needs the old data. Users' data, licensed data, and deleted data may not be replayable. Generative replay replays a model's beliefs, errors included.
Isolation needs the task identity, and it grows. Adapters, columns, and masks are free of forgetting only when something routes each input to the right one.
Test-time training costs compute at every query, and it can overfit a single input.
Facts in weights are slow to learn, leak into other facts, and are hard to delete. That is why §25.6 treats deletion from weights as a separate, hard problem.

✏️ What EWC costs for a 7B model

The Fisher and the anchor are one float32 vector each, $7\times10^9\times4$ bytes, or 28 GB apiece and 56 GB together. Mixed-precision training with Adam already holds about 16 bytes per parameter, 112 GB (§10.1). So EWC adds about 50 percent more state. ZeRO, which splits optimizer state across GPUs (§10.3), divides it across devices like the rest.

Applying the penalty costs a subtract, a multiply, and a multiply-add per parameter per step, about $4\times7\times10^9$ FLOPs. Training spends roughly $6\times7\times10^9$ FLOPs on every token, and a step holds millions of tokens, so the penalty is negligible. The expensive part is estimating the Fisher. That takes one per-example backward pass per sampled label, over a representative sample of the old data, typically thousands of sequences.

🧭 What to remember

Continual learning is measured by the whole accuracy matrix. ACC is the mean of its last row, BWT compares the last row with the diagonal, and FWT compares the entries just above the diagonal with the untrained model. The scenario (task-, domain-, or class-incremental) changes which methods work.
EWC is MAP inference with a Laplace approximation of the old posterior. Its springs are the diagonal Fisher, and $\lambda = N_A/N_B$ in principle but tuned in practice. The Fisher is the expected Hessian of the negative log-likelihood, always positive semidefinite, and equal to the Hessian for logistic regression. The empirical Fisher, built from the dataset's labels, is a different matrix.
For many tasks the Laplace recursion uses one anchor at the latest mode and adds precisions, with $\boldsymbol\Lambda \leftarrow \gamma_{\text{ewc}}\boldsymbol\Lambda + \mathbf{F}_t$ in online EWC. One spring per task double counts early tasks. Distilling the old model's outputs (LwF) is a full-Fisher penalty to second order that needs no old data, but it fails when the inputs shift.
Replay is the most reliable defense, especially class-incrementally. A reservoir keeps each of $n$ items with probability $k/n$ in $O(k)$ memory. A-GEM turns the buffer into a gradient constraint with a closed-form projection. Isolation (columns, masks, adapters) removes forgetting at the price of growth and routing.
Large models are kept current by continual pretraining with re-warming, re-decaying, and replay. With 5 percent replay for a weak shift it came within 0.02 nats of retraining, and with 25 percent for a strong shift it matched it. On-policy RL forgets less than SFT because it stays closer in KL to the base model.
Test-time training writes to weights during a single input, and TTT layers, delta rules, and Titans make that a layer's state. New facts usually belong in context or a store rather than in weights. Next, §25.6 turns memory toward one person, where learning about a user becomes personalization and brings privacy, consent, and security with it.

Kirkpatrick et al. (2017). Overcoming catastrophic forgetting in neural networks. The EWC paper, with its Bayesian motivation and experiments on permuted MNIST and Atari.
Huszár (2018). On Quadratic Penalties in Elastic Weight Consolidation. A four-page note that derives the many-task recursion and shows why summed penalties double count.
Kunstner, Hennig, Balles (2019). Limitations of the Empirical Fisher Approximation for Natural Gradient Descent. Why the empirical Fisher is not the Fisher, with clear counterexamples.
Lopez-Paz and Ranzato (2017). Gradient Episodic Memory for Continual Learning. The accuracy-matrix metrics ACC, BWT, and FWT, and gradient-constrained replay.
van de Ven and Tolias (2019). Three scenarios for continual learning. The task-, domain-, and class-incremental scenarios and a careful comparison of methods across them.
Ibrahim et al. (2024). Simple and Scalable Strategies to Continually Pre-train Large Language Models. Re-warming, re-decaying, and replay that come within a few hundredths of a nat of retraining, or match it, at 405M and 10B parameters.
Shenfeld, Pari, Agrawal (2025). RL's Razor: Why Online Reinforcement Learning Forgets Less. Forgetting predicted by KL to the base policy, and why on-policy RL keeps it small.
Sun et al. (2024). Learning to (Learn at Test Time): RNNs with Expressive Hidden States. TTT layers, whose hidden state is a model trained on the sequence as it is read.

Made for larger screens

Continual learning

Forgetting is measured with a matrix of accuracies

ACC, BWT, and FWT from an accuracy matrix

Forgetting in two dimensions, by hand

Where the spring's stiffness comes from

The Fisher information is the expected Hessian

The true Fisher and the empirical Fisher

True Fisher, empirical Fisher, and the Hessian of logistic regression

Many tasks need one anchor, not many

Online EWC across three tasks

Protecting outputs instead of weights

Replay with a buffer that stays fair

A replay buffer with reservoir sampling

Replay as a constraint on the gradient

The A-GEM projection

Giving each task its own parameters

Five tasks in a small simulation

The same tools in PyTorch

Keeping a large model current with continual pretraining

Reinforcement learning forgets less than supervised fine-tuning

Learning during a single test input

Test-time training on nearest neighbors

A layer whose state is a learned memory

Moving test-time learning into a cache or the weights

Where new knowledge should live

Where these methods break

Read more

Watch next