This section ties the chapter together and then puts it to work. Every layer we have built reads a sequence one token at a time and carries a fixed-size summary of the past, its state, from each token to the next. We call such a layer recurrent. We will see that all these layers are one kind of matrix, and that they differ only in how the state forgets and how a token writes to it. Then we prove what a fixed state cannot do, and see why the strongest long-context open models of 2025 and 2026 mix many recurrent layers with a few attention layers.

The models make the stakes concrete. Qwen3.5-397B-A17B, released by Alibaba in February 2026, has 60 token-mixing layers, the layers that move information between positions (its feed-forward layers act on each token alone). Forty-five of them are recurrent layers called Gated DeltaNet. Their state takes 189 MB per sequence in 32-bit floats, whether the context holds a thousand tokens or 262,144. Moonshot's Kimi Linear (October 2025) uses about three recurrent layers per attention layer, and it reports that, trained with an identical recipe, it beats a model with attention in every layer. NVIDIA's Nemotron 3 Nano has 29 token-mixing layers, and only 6 of them are attention.

Not everyone agrees. In the same month as Kimi Linear, MiniMax published a note on why its new model M2 went back to attention in every layer. Its previous model, MiniMax-01, had used linear attention, a recurrent layer, in seven of every eight layers.

Three questions run through the section. What do all these recurrent layers have in common? What can a layer with a fixed-size state provably not do? And how many attention layers does a model need to make up for it? Here are the answers in plain words, before any symbols.

• Every layer of this chapter multiplies the sequence by a lower-triangular matrix. Any block of that matrix that maps past inputs to future outputs has rank at most the state size. Mamba-2's layer is exactly linear attention with a forgetting factor.
• The layers differ in how the state forgets (never, at a fixed rate, at a rate each token chooses, or at a different rate per channel) and in how a token writes (by adding to what is stored, or by correcting it).
• A fixed state is a lossy memory. It cannot copy or recall more than it holds. If all its decays are positive, it cannot even tell whether it has seen an odd or an even number of ones.
• Attention keeps an exact record of the context, at a cost that grows with the context. A few attention layers give a recurrent model that record, and their cache becomes the only memory in the model that grows.

By the end you will be able to derive Mamba-2 from linear attention, explain the delta rule as one step of learning, state the limits of a fixed state as theorems, and size the memory of a hybrid model at a million tokens. Interviewers at labs ask about all four.

Every linear-time layer is a structured matrix

Start with one channel of a layer, so that the input is a sequence of $T$ numbers $\mathbf{x}\in\R^T$ and the output is another, $\mathbf{y}\in\R^T$. A layer that is linear in this input computes $\mathbf{y} = \mathbf{M}\mathbf{x}$ for some $T\times T$ matrix $\mathbf{M}$, called the sequence-mixing matrix (§21.1). Entry $\mathbf{M}_{ts}$ is how much input $s$ contributes to output $t$. A causal layer may only read the past, so $\mathbf{M}$ is lower triangular. Attention fits the same picture. Its $\mathbf{M}$ is the table of masked softmax weights, which depends on the input itself. (In this chapter $\mathbf{M}$ never means the additive attention mask of §4.4.)

The simplest recurrent layer keeps one number of state. At step $t$ it multiplies the old state by a decay $a_t$, adds the input $x_t$ with weight $u_t$, and reads the state out with weight $c_t$,

§21.1 unrolled this loop and found the matrix

In plain words, input $s$ enters the state with weight $u_s$, is multiplied by every decay between step $s$ and step $t$, and is read out with weight $c_t$. The product $a_{t:s}$ of the decays in between is called a segment product. It leaves out $a_s$, because $a_s$ acts on the state before input $s$ arrives. Here is the matrix of a tiny recurrence, worked by hand.

That rank of 1 is no accident. Everything the first three inputs can tell the last two outputs has to pass through one number, the state $h_3$. Each row of the block is "how this output reads $h_3$" times "what each input left in $h_3$". With a one-number state, all such rows are multiples of one row. §21.1 proved this rank-one property for any scalar recurrence. The rest of this part extends it to a state of $N$ numbers and gives it a name.

A state of $N$ numbers works the same way, with matrices in place of numbers. A time-varying linear state-space model (SSM) has, at each step $t = 1,\dots,T$, a transition matrix $\mathbf{A}_t\in\R^{N\times N}$, an input vector $\mathbf{B}_t\in\R^N$, and an output vector $\mathbf{C}_t\in\R^N$. It runs

The layers of this chapter are special cases. In S4 the three matrices are the same at every step (§21.3). In Mamba $\mathbf{A}_t$ is diagonal, and all three are computed from the current token (§21.4). Unrolling exactly as in the scalar case gives $\mathbf{h}_t = \sum_{s\le t}\mathbf{A}_t\mathbf{A}_{t-1}\cdots\mathbf{A}_{s+1}\mathbf{B}_sx_s$, so

In plain words, input $s$ enters the state through $\mathbf{B}_s$, is carried forward by every transition between $s$ and $t$, and is read out through $\mathbf{C}_t$.

The four-token example had a block of rank 1 because its state had one number. The name for "every such block has rank at most $N$" is semiseparable.

Only $T$ of these submatrices need checking. Write $\mathbf{M}_{k:T,\,1:k}$ for the block of rows $k$ to $T$ and columns 1 to $k$, the diagonal entry $\mathbf{M}_{kk}$ included. We call it the cut block at $k$. It holds everything that crosses from the past (inputs up to $k$) to the future (outputs from $k$ on). Every submatrix in the lower triangle sits inside one of the $T$ cut blocks. So $\mathbf{M}$ is $N$-semiseparable exactly when every cut block has rank at most $N$ (the derivation below has the one-line argument).

In plain words, everything the past tells the future must pass through the $N$ numbers of the state at the cut, so the block that maps one to the other has rank at most $N$.

The converse holds too. Every $N$-semiseparable matrix can be written in this form for some $\mathbf{A}_t$, $\mathbf{B}_t$, $\mathbf{C}_t$. It can be stored in $O(NT)$ numbers and multiplied by a vector in $O(NT)$ time (Dao and Gu 2024, Proposition 3.4, citing the structured-matrix literature). So the layers that can run as a recurrence with $N$ numbers of state are exactly the layers whose matrix is $N$-semiseparable. Softmax attention is not among them for any fixed $N$, because its cut blocks have whatever rank the content gives them.

This answers a common interview question, whether softmax attention can be rewritten as an RNN with a fixed state. It cannot, because a state of $N$ numbers caps the rank of every cut block at $N$, and attention's cut blocks are not capped. The first cell repeats the worked example. The second compares a diagonal SSM with $N = 4$ against causal softmax attention on $T = 64$ tokens. The SSM is built the Mamba-1 way, with its own decay for each of the four state entries.

The SSM's matrix has full rank 64, so it is not a low-rank matrix. Yet no cut block has rank above 4. The attention matrix's middle cut block, 32 rows by 33 columns, has rank 32, the most it could have. That difference is the chapter in one number. A recurrence squeezes the past into $N$ numbers at every cut, and attention does not.

Mamba-2 is masked linear attention

Two earlier layers meet here. The first is causal linear attention (§21.2). It drops the softmax and scores each earlier token by a plain dot product of the query with that token's key. Run as a recurrence, it keeps a matrix state $\mathbf{S}_t = \mathbf{S}_{t-1} + \phi(\mathbf{k}_t)\mathbf{v}_t^\top$ and reads it out as $\mathbf{o}_t = \mathbf{S}_t^\top\phi(\mathbf{q}_t)$. Here $\phi$ is a feature map, a fixed function applied to queries and keys, and the state's rows are indexed by key features and its columns by value channels. Its weakness was that the state never forgets.

The second is Mamba (§21.4), whose state forgets at a rate each token chooses. Each channel has its own state vector in $\R^N$ with a diagonal transition, so each state entry has its own decay. The decays and the vectors $\mathbf{B}_t$, $\mathbf{C}_t$ are computed from the token.

Mamba-2 (Dao and Gu 2024) makes two changes to Mamba, and together they turn it into linear attention with a forgetting factor. First, the transition becomes one number times the identity, $\mathbf{A}_t = a_t\mathbf{I}$, so all $N$ entries of a state forget at the same rate. Second, a group of $P$ channels, called a head, shares that number and the same $\mathbf{B}_t$ and $\mathbf{C}_t$. Stack the $P$ channels' state vectors side by side, and the state of a head becomes an $N\times P$ matrix.

The state's rows index the $N$ state dimensions and its columns the $P$ channels. That is the chapter's convention of rows for keys and columns for values. Now unroll the recurrence, and attention appears.

Proof. Each entry of $\mathbf{S}_t$ follows a scalar recurrence with decay $a_t$, so the closed form of §21.1, applied entry by entry, gives $\mathbf{S}_t = \sum_{s=1}^{t}a_{t:s}\,\mathbf{B}_s\mathbf{x}_s^\top$. (The induction step is $a_t\sum_{s\le t-1}a_{t-1:s}\mathbf{B}_s\mathbf{x}_s^\top + \mathbf{B}_t\mathbf{x}_t^\top = \sum_{s\le t}a_{t:s}\mathbf{B}_s\mathbf{x}_s^\top$, using $a_ta_{t-1:s} = a_{t:s}$ and $a_{t:t} = 1$.) Then

which is row $t$ of $\mathbf{M}\mathbf{X}$, since $(\mathbf{C}\mathbf{B}^\top)_{ts} = \mathbf{C}_t^\top\mathbf{B}_s$. ∎

In plain words, token $t$ scores every earlier token $s$ by the dot product of its "query" $\mathbf{C}_t$ with that token's "key" $\mathbf{B}_s$. It multiplies the score by how much the state has decayed between $s$ and $t$, and adds up the earlier inputs with those weights. There is no softmax and no normalization. The decay takes the place of a positional encoding. It is how the layer knows how far back $s$ is.

The four-token example above is this theorem with $N = P = 1$. Its matrix was the decays times $C_tB_s$, entry by entry. The table translates every piece between the two views.

Why give up Mamba-1's diagonal transition for a single number? With $\mathbf{A}_t = \operatorname{diag}(\mathbf{a}_t)$, the matrix is $\mathbf{M}_{ts} = \sum_{n=1}^{N}C_{t,n}\,a_{t:s,n}\,B_{s,n}$, with a different mask for every state dimension $n$. That cannot be written as one mask times one matrix of dot products. The quadratic form would need a $T\times T\times N$ tensor, and no matrix multiplication computes it. With a scalar decay, all $N$ dimensions share one mask, and $\mathbf{C}\mathbf{B}^\top$ is a single matrix product. Matrix products are what a GPU's tensor cores, its matrix-multiply units, do best, at about 15 times the speed of its ordinary arithmetic (§21.5).

The price is that every entry of a head forgets at the same rate. The paper spends the savings on a much larger state. The released models use $N = 128$ and the experiments range from 64 to 256, against Mamba-1's 16. Heads have $P = 64$ or 128 channels, where Mamba-1 ran one recurrence per channel. The cell checks the duality at the size of a real Mamba-2 head, $N = P = 64$. Then it sets every decay to 1 and recovers linear attention.

import numpy as np

def ssd_recurrent(X, a, B, C):
    raise NotImplementedError

def ssd_quadratic(X, a, B, C):
    raise NotImplementedError

The chunked algorithm is a block decomposition of the matrix

Neither form is what Mamba-2 runs in training. The recurrence does $O(TNP)$ work per head, but in small elementwise steps that leave the tensor cores idle. The quadratic form uses matrix products, but its work grows as $T^2$. §21.5 derived the compromise for gated linear attention, the chunkwise algorithm. It cuts the sequence into chunks, handles each chunk with small matrix products, and passes one state from chunk to chunk. Under the dictionary of the duality ($\mathbf{q}\to\mathbf{C}$, $\mathbf{k}\to\mathbf{B}$, $\mathbf{v}\to\mathbf{x}$, $d_k\to N$, $d_v\to P$), SSD is gated linear attention with a scalar decay, so the same algorithm applies. The semiseparable view shows why it is exact, and the block picture of §21.5 is the one to keep in mind.

Cut the sequence into chunks of $T_c$ tokens, and cut $\mathbf{M}$ into a grid of $T_c\times T_c$ blocks. There are two kinds of block.

• A diagonal block, chunk $j$ against itself, is the matrix of the same layer run on that chunk alone from a zero state. It is computed in quadratic form, $(\mathbf{L}_{[j]}\odot\mathbf{C}_{[j]}\mathbf{B}_{[j]}^\top)\mathbf{X}_{[j]}$, with a few small matrix products. Here $[j]$ picks out the rows of chunk $j$.
• An off-diagonal block, the outputs of chunk $j$ against the inputs of an earlier chunk $i$, lies entirely below the diagonal. So the proof of the semiseparable theorem factors it through the state.

If chunk $i$ ends at position $e$ and chunk $j$ starts at position $b$, then for $t$ in chunk $j$ and $s$ in chunk $i$

In plain words, the three factors are three steps.

1. The right factor, applied to chunk $i$'s inputs, gives $\sum_{s\in[i]}a_{e:s}\mathbf{B}_s\mathbf{x}_s^\top$. That is the state chunk $i$ would leave if it started from zero.
2. The center factors chain those chunk states into the true state entering each chunk. This is a scan of length $T/T_c$, the only sequential part.
3. The left factor reads that state for every output of chunk $j$.

Steps 1 and 3 are matrix products of size $T_c\times N$ by $N\times P$. The cell runs the algorithm on twelve tokens in chunks of four, then checks one off-diagonal block against its three factors.

The cost follows from the count in §21.5. Per head, the diagonal blocks take $2TT_c(N + P)$ FLOPs (floating-point operations) in all. The chunk states and their read-out take $4TNP$, plus a short scan. With $N = P = T_c$ that is $O(TN^2)$, the same order as the recurrence, but nearly all of it is matrix multiplication.

In an interview, this is the answer to "why did Mamba-2 restrict the transition to a scalar?" The scalar decay makes every block of the matrix a masked matrix product, so training runs on tensor cores.

In PyTorch, with autograd

This part is for readers who will write or debug these kernels. The production version replaces the loop over chunks with einsum calls and keeps all decays in log space. Every product of decays comes from a cumulative sum inside a masked lower triangle or inside one short chunk. It never comes from a ratio of two long products, the stability rule of §21.5. That rule also makes a reset safe. A reset is $\log a_t = -\infty$, and a difference such as cum[-1] - cum[j] would then be $-\infty - (-\infty)$, which is NaN. A masked sum is $-\infty$, whose exponential is 0. The block checks outputs and gradients against the loop, then two resets, then a 4,096-step sequence in float32.

The official kernels fuse these einsums in Triton, a language for writing GPU kernels, and pick $T_c = 256$ by default. The structure is the one above, four batched matrix products around one short scan.

The Mamba-2 block

A layer ships inside a block, the unit that is stacked to make the model. Mamba-2 redesigned the block around the SSD layer too. The goal was to make it look like an attention block, both to the GPU and to the tools that split a model across GPUs (Chapter 10). Three pieces of the definition need a word first. A depthwise convolution filters each channel with its own short filter and never mixes channels. SiLU is the smooth gate $\operatorname{SiLU}(z) = z\,\sigma(z)$, with $\sigma$ the logistic sigmoid. RMSNorm divides a vector by the root mean square of its entries and multiplies by a learned scale (§5.2).

Three features of the block matter. The first two are changes from the Mamba-1 block of §21.4. Mamba-1 already had the third, but the Mamba-2 paper was the first to test it against the alternatives.

One projection instead of two in sequence. Mamba-1 computed $\Delta$, $\mathbf{B}$, and $\mathbf{C}$ from the SSM's input with a second projection. Mamba-2 produces everything from one projection of $\mathbf{u}$, the way attention produces $\mathbf{Q}$, $\mathbf{K}$, and $\mathbf{V}$ side by side. That matters for tensor parallelism, which gives each GPU a slice of every weight matrix (§10.4). Each GPU can then hold whole heads, and the GPUs need to sum their partial results only once per block, as in an attention block.

An extra normalization. An RMSNorm before the output projection keeps larger models stable, much as output normalization does in modern linear attention (§21.2). At about 130M parameters, the single projection and the extra norm together lowered perplexity from 11.76 to 11.49 and saved 2.8 million parameters. (Perplexity is the exponential of the average next-token loss, so lower is better.)

Keys and queries shared across heads. All $H$ heads share one $\mathbf{B}$ and one $\mathbf{C}$ (for $G = 1$), while each head has its own $\mathbf{x}$ and its own decay. Mamba-1 shared them the same way, across all of its channels. In the attention dictionary, one key and one query serve many value heads. That is the mirror image of multi-query attention, where one key and one value serve many query heads (§12.5). Dao and Gu call it multi-value attention. At matched parameters and state, it beat the multi-query pattern (shared keys and values) and the multi-key pattern (shared queries and values), with a perplexity of 11.66 against 12.62 and 12.59 at 126.5M parameters. The $G$ groups generalize it, as grouped-query attention generalizes multi-query attention.

The real-scale count makes the shapes concrete. Mamba-2 2.7B has $d = 2560$, 64 layers, $N = 128$, $P = 64$, and $G = 1$. Its embedding matrix is shared with the output layer (a tied embedding).

Packing puts several documents in one training row without padding (§4.4 built the masks that do this for attention). Mathematically, a reset $a_t = 0$ at each document boundary zeroes every entry of $\mathbf{L}$ that connects two documents. That is the case tested in Exercise 21.6a. Kernels pass a document index instead (seq_idx or cu_seqlens in mamba_ssm, cu_seqlens in fla, the two main kernel libraries).

One family tree of update rules

Dozens of recurrent layers have been published since 2020, and they can look unrelated. They are not. Nearly all keep a matrix state, like linear attention, and each step does two things to it. First it forgets some of what is stored. Then it writes the new token in. The layers differ mainly in those two choices.

Here is the template. A matrix state $\mathbf{S}_t\in\R^{d_k\times d_v}$, with rows indexed by keys and columns by values, is read out as $\mathbf{o}_t = \mathbf{S}_t^\top\mathbf{q}_t$ and updated by

Here $\mathbf{G}_t$ is a linear map of the state chosen by the token, and the write is built from the token's key $\mathbf{k}_t\in\R^{d_k}$ and value $\mathbf{v}_t\in\R^{d_v}$. Most papers store the transpose of $\mathbf{S}_t$, so their equations look mirrored. Every equation below is converted to the book's convention.

A tiny case shows how much the choices matter. Store the value 3 at a key, wait 10 steps, and read it back. Linear attention returns 3, because it never forgets. A fixed decay of 0.9 per step returns $3\cdot0.9^{10}\approx1.05$, whether or not anything else was written. A gate that the token sets can hold the value at 3 through 10 unimportant tokens, then clear it when a new topic starts. Here are the layers, from the simplest forgetting to the richest.

No forgetting. Linear attention (§21.2) uses $\mathbf{S}_t = \mathbf{S}_{t-1} + \mathbf{k}_t\mathbf{v}_t^\top$. The state only grows. The crosstalk between stored keys, the error a read picks up from other keys that are not exactly orthogonal to it, only accumulates, and nothing old is ever cleared.

A fixed decay per head. The state shrinks by the same constant factor at every step, $\mathbf{S}_t = \gamma\,\mathbf{S}_{t-1} + \mathbf{k}_t\mathbf{v}_t^\top$. This is RetNet (Sun et al. 2023). Head $i = 0, 1, \dots$ uses $\gamma_i = 1 - 2^{-5-i}$, so the first eight heads have half-lives from 22 to 2,839 steps. (The half-life, $\ln 2/(-\ln\gamma)$, is the number of steps after which a stored value has shrunk to half, §21.1.) It forgets, but on a schedule that ignores the content.

A fixed decay per channel. Each key channel has its own constant decay, $\mathbf{S}_t = \operatorname{diag}(\mathbf{w})\mathbf{S}_{t-1} + \mathbf{k}_t\mathbf{v}_t^\top$, plus a learned bonus weight for the current token. This is RWKV-5. RWKV-6 makes $\mathbf{w}$ depend on the token (Peng et al. 2024).

A data-dependent scalar gate. The token decides how much of the whole state to keep, $\mathbf{S}_t = a_t\mathbf{S}_{t-1} + \mathbf{B}_t\mathbf{x}_t^\top$. This is Mamba-2.

A data-dependent gate per key channel. The token decides how much to keep along each key direction, $\mathbf{S}_t = \operatorname{diag}(\boldsymbol\alpha_t)\mathbf{S}_{t-1} + \mathbf{k}_t\mathbf{v}_t^\top$, with $\boldsymbol\alpha_t\in(0,1)^{d_k}$ computed from the token. This is Gated Linear Attention, or GLA (Yang et al. 2023), the layer whose chunked kernel §21.5 built.

An exponential input gate with a normalizer. The forget gate $f_t$ is a sigmoid, but the write is weighted by an exponential, $i_t = \exp(\tilde i_t)$ of a number $\tilde i_t$ computed from the token. So one token can write far more strongly than any before it. A second state, the normalizer $\mathbf{n}_t$, tracks the total write weight, and the read-out divides by it,

This is the mLSTM, the matrix-memory layer of xLSTM (Beck et al. 2024). The exponential needs a numerical guard, built in the next subsection.

A vector state with two gates. The recurrent layer of Griffin, called the RG-LRU (De et al. 2024), keeps a vector, not a matrix, $\mathbf{h}_t = \mathbf{a}_t\odot\mathbf{h}_{t-1} + \sqrt{1 - \mathbf{a}_t^2}\odot(\mathbf{i}_t\odot\mathbf{x}_t)$. Both the input gate $\mathbf{i}_t$ and the decay $\mathbf{a}_t$ are computed from the token.

Correcting instead of adding. The token first reads what the state stores at its key, then writes only the difference, $\mathbf{S}_t = (\mathbf{I} - \beta_t\mathbf{k}_t\mathbf{k}_t^\top)\mathbf{S}_{t-1} + \beta_t\mathbf{k}_t\mathbf{v}_t^\top$. This is the delta rule, used by DeltaNet, Gated DeltaNet, Kimi Delta Attention, and RWKV-7. The hybrids of Qwen and Moonshot use this branch, NVIDIA's and IBM's use Mamba-2, and the delta rule gets its own part below.

A state that rotates. Mamba-3 (Lahoti et al. 2026) keeps Mamba-2's scalar decay but makes the state complex, so it can rotate as well as shrink. That is the same as a rotary embedding on $\mathbf{B}$ and $\mathbf{C}$ whose angles depend on the data (§5.5). It gives the transition eigenvalues off the positive real axis, which the limits below show matter. At 1.5B parameters it beat Gated DeltaNet by 0.6 points of average downstream accuracy.

Read down the forget column and the table is a story of finer control over forgetting, from never, to a fixed rate, to a rate each token sets for each channel. Read down the write column and there is one break, from adding to correcting. An interviewer who asks you to compare two of these layers wants exactly those two axes. Say how each one forgets, then how each one writes.

Exponential gates need a stabilizer

This part is for readers who will implement an exponentially gated layer. The rest of the section does not depend on it.

The mLSTM's exponential input gate lets one token write with weight $e^{\tilde i_t}$, far more decisively than a sigmoid gate, which is at most 1. That is the reason it exists. It is also an overflow waiting to happen. In float32, $e^{\tilde i}$ is infinite once $\tilde i$ exceeds about 88.7, and a product of many forget gates underflows toward 0. The fix rescales the state by a running maximum of the log weights. It is the same trick as the online softmax of §12.2, which subtracts the largest score seen so far before exponentiating.

The recipe has three steps.

1. Track $m_t = \max(\log f_t + m_{t-1},\ \tilde i_t)$ with $m_0 = 0$.
2. Run the two recurrences with the rescaled gates $f'_t = e^{\log f_t + m_{t-1} - m_t}$ and $i'_t = e^{\tilde i_t - m_t}$. Both are at most 1 by the choice of $m_t$.
3. Divide by $\max(\lvert\mathbf{n}'^\top_t\mathbf{q}_t\rvert,\ e^{-m_t})$ instead of $\max(\lvert\mathbf{n}_t^\top\mathbf{q}_t\rvert,\ 1)$.

This is the stabilized mLSTM of the xLSTM paper's appendix (Beck et al. 2024) and of the xLSTM kernels. In plain words, keep the states in units of the largest weight seen so far, and convert the floor of 1 in the denominator into those units too. Take $\tilde i = 1000$ at one step. The raw weight $e^{1000}$ overflows, but the stored weight is $e^{1000 - m_t} = 1$, and the floor becomes $e^{-1000}$. In the original units the floor of 1 is negligible next to a normalizer of size $e^{1000}$, and in the new units it is just as negligible.

import numpy as np

def mlstm_recurrent(Q, K, V, i_pre, f_pre):
    T, d_v = V.shape
    H = np.zeros((T, d_v))
    S = np.zeros((K.shape[1], d_v)); n = np.zeros(K.shape[1])
    for t in range(T):                       # unstabilized: overflows for large i_pre
        f = 1 / (1 + np.exp(-f_pre[t])); i = np.exp(i_pre[t])

The delta rule corrects instead of adding

Every rule so far writes by adding. That fails as soon as a key comes back with a new value. Say a memory stores the value 3 at some key, and later the same key arrives with the value 5. An adding memory now returns $3 + 5 = 8$ at that key, a value that was never written. A decay cannot fix this without also fading everything else stored.

What we want is to read what is stored at the key (3), and write only the difference (2). Then the memory returns 5, and nothing stored at other, orthogonal keys is touched. This is the delta rule. It is the least-mean-squares rule of Widrow and Hoff 1960, which nudges a linear model toward each new target by a fraction of its error. Schlag, Irie, and Schmidhuber 2021 proposed it for matrix memories written while the model runs, which they called fast weights, as the fix for the crosstalk that §21.2 measured.

The second form of the delta rule reads as a sentence. $\mathbf{S}_{t-1}^\top\mathbf{k}_t$ is the value currently stored at $\mathbf{k}_t$. $\mathbf{v}_t - \mathbf{S}_{t-1}^\top\mathbf{k}_t$ is the memory's error on this key. The update writes a fraction $\beta_t$ of that error. It is one step of learning.

Proof. Drop the subscript $t$ and write $\ell = \tfrac12\sum_{j}\big(\sum_iS_{ij}k_i - v_j\big)^2$. Then $\partial\ell/\partial S_{ij} = k_i\big(\sum_{i'}S_{i'j}k_{i'} - v_j\big) = k_i\,(\mathbf{S}^\top\mathbf{k} - \mathbf{v})_j$, so $\nabla\ell(\mathbf{S}) = \mathbf{k}(\mathbf{S}^\top\mathbf{k} - \mathbf{v})^\top$. A step of size $\beta$ gives $\mathbf{S} - \beta\mathbf{k}\mathbf{k}^\top\mathbf{S} + \beta\mathbf{k}\mathbf{v}^\top$, because $\mathbf{k}(\mathbf{S}^\top\mathbf{k})^\top = \mathbf{k}\mathbf{k}^\top\mathbf{S}$, which is the delta rule. Replacing $\mathbf{S}_{t-1}$ by $\alpha_t\mathbf{S}_{t-1}$ gives the gated rule.

For item 1, $\mathbf{S}_t^\top\mathbf{k} = \mathbf{S}_{t-1}^\top(\mathbf{I} - \beta\mathbf{k}\mathbf{k}^\top)\mathbf{k} + \beta\mathbf{v}\,(\mathbf{k}^\top\mathbf{k}) = (1 - \beta)\mathbf{S}_{t-1}^\top\mathbf{k} + \beta\mathbf{v}$, using $\mathbf{k}^\top\mathbf{k} = 1$. For item 2, $(\mathbf{I} - \beta\mathbf{k}\mathbf{k}^\top)\mathbf{u} = \mathbf{u}$ because $\mathbf{k}^\top\mathbf{u} = 0$. For item 3, $(\mathbf{I} - \beta\mathbf{k}\mathbf{k}^\top)\mathbf{k} = (1 - \beta)\mathbf{k}$ and every $\mathbf{u}\perp\mathbf{k}$ is fixed. The matrix is symmetric, so its spectral norm is its largest absolute eigenvalue, $\max(1, \lvert1 - \beta\rvert)$, which is at most 1 exactly when $0\le\beta\le2$. ∎

In plain words, each token trains the memory a little, by gradient descent on "when I see this key, produce this value" (the SGD step of §0.6), and the gate $\alpha_t$ is weight decay on the memory. The normalization of $\mathbf{k}$ is not cosmetic. A key of norm $r$ gives the eigenvalue $1 - \beta r^2$, so a key of norm 3 with $\beta = 1$ multiplies the stored component by $-8$ at every write, and the state explodes.

The same lens explains linear attention. Its write $\mathbf{k}\mathbf{v}^\top$ is a gradient step of size 1 on the loss $-\langle\mathbf{S}^\top\mathbf{k}, \mathbf{v}\rangle$, which rewards matching the value but never penalizes what is already stored, so repeated keys pile up. In an interview, the one-line answer to "why is DeltaNet better than linear attention at recall?" is that its write minimizes a squared error instead of maximizing an inner product.

The cell runs four write rules on a stream of 200 writes that keeps rewriting 8 orthonormal keys with fresh values. It measures how far each memory's answer is from the latest value of every key.

The four memories behave very differently.

• The additive memory's error grows without bound (0.55, 2.38, 6.15), because every old value of a key is still in the state.
• The decaying memory settles near 1.0. It forgets keys written long ago as surely as it forgets stale values.
• The delta rule is exact, error 0.00 throughout. With orthonormal keys, each write replaces one value and leaves the other seven alone (items 1 and 2 of the theorem).
• The gated delta rule at $\alpha = 0.95$ sits at 0.29. It pays for forgetting it did not need on this stream.

The gate earns its place on other streams. When most tokens are distractors, every one of them that DeltaNet writes disturbs the stored keys it is not orthogonal to. A selective gate keeps the state unchanged through those tokens. That is why the strongest layers have both a gate and a correction. The widget below runs six update rules on such a stream and two others, with $d_k = 16$, $d_v = 8$, a decay of 0.95, and $\beta = 1$. In the distractor stream, an option lets the layers with input-dependent gates (Mamba-2, GLA, and Gated DeltaNet) skip the distractors, a stand-in for a learned gate that recognizes them. Click a rule's name to draw it bold, and press Play to replay the writes.

import numpy as np

def delta_step(S, k, v, beta):
    return S + np.outer(k, v)          # placeholder: additive linear attention

def gated_delta_step(S, k, v, alpha, beta):
    return alpha * S + np.outer(k, v)  # placeholder: decay plus additive write

The Gated DeltaNet block

A model ships a block, not a rule. The Gated DeltaNet block is what Qwen3-Next and Qwen3.5 put in three of every four layers. Its skeleton is the Mamba-2 block's. It has one wide input projection, a short convolution, a decay computed the way Mamba-2 computes it, a gated normalization (an RMSNorm whose output is multiplied by a SiLU gate), and an output projection. The difference is that it has separate queries and keys in place of the shared $\mathbf{B}$ and $\mathbf{C}$.

Each piece has a reason.

• The convolution gives every query, key, and value a view of the last few tokens. The fla implementation warns that it is crucial to performance.
• The L2 normalization is item 3 of the delta-rule theorem made concrete, since a key of norm $r$ gives the eigenvalue $1 - \beta r^2$.
• The gated RMSNorm plays the role of Mamba-2's extra norm.
• Step 5 is Mamba-2's multi-value pattern, one query and key serving several value heads, which fla calls grouped value attention. Qwen3.5-397B-A17B uses $n_{qk} = 16$ and $n_v = 64$, and Qwen3-Next-80B-A3B uses 16 and 32.

Unlike Mamba-2, the block has no skip term $D\mathbf{x}_t$.

The cell counts one Qwen3.5 layer from its published configuration, $d = 4096$ and $d_k = d_v = 128$. It compares the layer with one of the model's gated attention layers, which have 32 query heads and 2 KV heads of width 256.

So when an interviewer asks whether linear-time layers make a model smaller, the answer is no. They make its per-sequence memory constant, which is what limits serving at long context.

Training the delta rule in chunks

This part is for readers who will write or debug these kernels. The rest of the section does not depend on it.

The delta rule's transition is neither a scalar nor a diagonal, so the chunked algorithm of §21.5 does not apply as it stands. Take one chunk of $r$ tokens, numbered $1,\dots,r$ inside the chunk, that enters with state $\mathbf{S}_0$. Unrolling the rule gives $\mathbf{S}_i = \mathbf{P}_i\mathbf{S}_0 + \mathbf{H}_i$. Here $\mathbf{P}_i = (\mathbf{I} - \beta_i\mathbf{k}_i\mathbf{k}_i^\top)\cdots(\mathbf{I} - \beta_1\mathbf{k}_1\mathbf{k}_1^\top)$ is the product of the transitions, and $\mathbf{H}_i$ is the state the chunk would build from zero. Both are dense matrices. Building them token by token is a chain of $r$ dependent rank-one updates that tensor cores cannot help with.

Yang et al. 2024 made DeltaNet trainable at scale with a classical identity from numerical linear algebra. Each factor $\mathbf{I} - \beta\mathbf{k}\mathbf{k}^\top$ is a generalized Householder matrix (at $\beta = 2$ it is the Householder reflection across the plane orthogonal to $\mathbf{k}$), and the WY representation (Bischof and Van Loan 1987) writes a product of such factors as the identity minus a sum of $r$ rank-one terms, all of which can be found at once with one triangular solve.

The outputs inside the chunk come from the same formula. Since $\mathbf{o}_i = \mathbf{S}_i^\top\mathbf{q}_i$ and $\mathbf{S}_i = \mathbf{S}_0 + \sum_{j\le i}\mathbf{k}_j(\mathbf{u}_j - \mathbf{S}_0^\top\mathbf{w}_j)^\top$, stacking the queries as the rows of $\mathbf{Q}$ and the outputs as the rows of $\mathbf{O}$ gives

where $\operatorname{tril}$ keeps the diagonal, because output $i$ reads the state after token $i$'s own write. In plain words, row $j$ of $\mathbf{U} - \mathbf{W}\mathbf{S}_0$ is what token $j$ really writes once the incoming state and the chunk's earlier tokens are accounted for, and a chunk changes the state only along its own $r$ keys.

Every step is a matrix product. Per chunk, $\mathbf{K}\mathbf{K}^\top$ and $\mathbf{Q}\mathbf{K}^\top$ cost $O(r^2d_k)$, the two triangular solves $O(r^2(d_k + d_v))$, and the products with $\mathbf{S}_0$ cost $O(rd_kd_v)$. So $T$ tokens cost $O\big(Tr(d_k + d_v) + Td_kd_v\big)$, the same order as chunked gated linear attention in §21.5, and only the pass of states from chunk to chunk stays sequential. The cell checks the theorem on a chunk of five tokens.

Gated DeltaNet folds its decays into the same representation. Let $\gamma_i = \alpha_1\cdots\alpha_i$ be the decay accumulated inside the chunk, and let $\boldsymbol\Gamma_{ij} = \gamma_i/\gamma_j$ for $j\le i$ (0 above the diagonal) be the decay between two positions. Dividing the state by $\gamma_i$ turns the gated rule into the plain delta rule with values $\mathbf{v}_i/\gamma_i$, so the theorem applies to it. Multiplying back gives

where $\mathbf{A}_\Gamma$ is $\mathbf{A}$ with each key product $\mathbf{k}_i^\top\mathbf{k}_j$ multiplied by $\boldsymbol\Gamma_{ij}$. The state leaving the chunk is $\gamma_r\mathbf{S}_0 + \mathbf{K}^\top\operatorname{diag}(\gamma_r/\boldsymbol\gamma)(\mathbf{U} - \mathbf{W}\mathbf{S}_0)$. In plain words, every interaction between positions $j$ and $i$ is scaled by the decay between them, and the incoming state is scaled by the decay since the chunk began. Yang, Kautz, and Hatamizadeh measured the training throughput as essentially equal to DeltaNet's and slightly below Mamba-2's, the cost of a more expressive transition.

import numpy as np

def chunked_delta_rule(Q, K, V, beta, T_c, S0=None):
    raise NotImplementedError

What a fixed state provably cannot do

Every layer so far keeps a state of fixed size, however long the context. That is what makes it cheap, and it is also a hard limit. §21.1 gave the intuition. A state that can take only so many values cannot tell apart more prompts than that, by the pigeonhole principle. §21.2 measured how many key-value pairs an additive matrix memory holds before crosstalk swamps them.

Here are the theorems, in two families. Copying and recall are limited by how much the state holds. State tracking, following a quantity that every token can change, is limited by the eigenvalues of the transitions, however large the state. These are the results behind the interview question of what an SSM cannot do.

Copying and associative recall

Proof. Let $\sigma(\mathbf{z})$ be the state after reading $\mathbf{z}$. Everything the model does afterward depends on $\mathbf{z}$ only through $\sigma(\mathbf{z})$, so there is one output distribution $p(\cdot\mid\sigma)$ per state value. Then

where the middle inequality holds because, for each fixed $\sigma$, the probabilities $p(\mathbf{z}\mid\sigma)$ of distinct strings sum to at most 1. ∎

In plain words, if the state has fewer possible values than there are strings, some strings must leave the same state. From then on the model cannot tell them apart. For example, a state of 1,000 bits copies a random 1,100-bit string with probability at most $2^{-100}$. The theorem needs bounded precision, since one real number of infinite precision could encode any string. Real models store states in float32 or bf16, so it applies.

A Transformer's KV cache grows with $n$, so the theorem says nothing about it. Jelassi et al. 2024 prove the matching positive result. A two-layer Transformer can copy strings whose length is exponential in its size, while any model of the class above is capped by its state. They also measured the gap in pretrained models, where Transformers copy and retrieve from context far better than state-space models of similar size.

Associative recall obeys the same count. Multi-query associative recall (MQAR), introduced by Arora et al. 2023, is the test. The model reads $n$ key-value pairs $(\kappa_1, \nu_1), \dots, (\kappa_n, \nu_n)$ with distinct keys and with values drawn from a vocabulary of $c$ symbols. Then it receives queries $\kappa_j$ and must answer each with its $\nu_j$. In "A 4 B 3 C 6, then A ? C ?", the answers are 4 and 6. (The Greek letters keep these task keys and values apart from the layer's own $\mathbf{k}_t$ and $\mathbf{v}_t$.)

Attention escapes both bounds because its cache grows with $n$. The bounds themselves are modest. A thousand associations over a 50,000-symbol vocabulary need at least $1{,}000\log_2 50{,}000\approx15{,}600$ bits, about 2 KB.

The bit count is only a ceiling. §21.1 computed that Mamba-2 2.7B's state is 688 million bits in bf16. That is enough in principle to store a 131,072-token context verbatim many times over, and yet such models fail recall tasks long before that. The binding limits are the structure of the state and the timing. Each head is a small matrix ($64\times128$ numbers) written by a fixed, simple rule, and the layer must decide what to keep before it knows the question.

State tracking and parity

The second limit has nothing to do with memory size. Parity asks whether a bit string contains an odd number of ones, and it needs only one bit of state. A small nonlinear RNN computes it exactly. Yet Mamba-1 and Mamba-2 cannot compute it for every length in one forward pass, and the reason is the sign of their decays. Parity is the simplest case of state tracking. Evaluating code, following chess moves, and keeping track of who holds what in a story are harder cases of the same thing.

An eigenvalue that is not a nonnegative real number is either negative or complex. Mamba-1 and Mamba-2's decays $e^{\Delta A}$ lie in $(0, 1)$. DeltaNet with $\beta_t\in(0, 1)$ has eigenvalues $1 - \beta_t\in(0,1)$ and 1. So all of them fall under the theorem. The full proof uses the Jordan form of a matrix and a careful model of rounding. Its heart is a special case we can prove completely.