A model forgets everything the moment its context window ends. This chapter builds the memory that lets an agent work for hours or days and learn from what it did, from the limits of the window itself to long-term stores, durable state, learning in the weights, and the privacy and security risks that come with remembering.

This chapter is about an agent's memory, everything it can draw on when it acts, from the tokens of the current call to what it learned months ago. An agent is a language model that works in a loop, calling tools and reading their results (§24.1). Real agents run for hundreds of turns, across many sessions, and for one person over months. What they remember decides what they cost, how long a task they can finish, and what they can leak about a user or be tricked into believing. The chapter moves outward one tier at a time. It starts with the context window in this section, then decides what enters the window on each call (§25.2), builds stores outside it (§25.3), keeps a task alive for days (§25.4), writes into the weights (§25.5), and ends with the memory of one person (§25.6).

Two earlier results underlie the chapter. First, §24.1 counted the tokens of the agent loop. Every turn re-reads everything before it, so the total grows with the square of the number of turns. §24.6 priced that count with prompt caching. Second, every attention head reads its context through softmax weights that add up to one (§4.1). The first result makes a long context expensive. The second, as this section proves, makes it hard to use.

This section is about the context window, the longest run of tokens a language model can handle in one call, its input plus the output it writes. A model keeps nothing from one call to the next, so everything it can use about the task in front of it must be in those tokens. For an agent the window is therefore its working memory. We ask two questions about it. What does each token in the window cost? And how well can the model use what is there as the window fills up?

Both questions show up in one ordinary failure. A coding agent has worked on a bug for an hour. It has read 40 files, run 60 shell commands, and reached turn 120, and its context now holds about 180,000 tokens. Two things have gone wrong at once. Each turn costs more than the one before, and by now the last ten turns together have cost more than the first thirty. And around turn 90 the agent stopped obeying an instruction the user gave at turn 3, "never edit the generated files", although that instruction is still in the context, word for word.

Both failures have one cause. The model reads only the sequence in front of it, and the agent keeps appending to that sequence. The first half of this section prices a window in dollars, memory, compute, and time. It recounts, in this chapter's notation, the bill of an agent that resends its whole history, and adds the attention compute behind it. The second half proves that attention must spread itself thinner as the window fills, works out how much sharper a model must look to keep its focus, and reviews what long contexts do to real models in published experiments. Throughout, we keep what is proven apart from what is only measured.

What the window holds

Recall how a model generates text (§6.5). It first reads all $T$ tokens of its input in one parallel pass, called the prefill. Then it writes its answer one token at a time, called the decode, and each new token looks back at every token before it. So that it never recomputes the past, the model stores two vectors for every earlier token in every layer, a key and a value. These are what attention uses to find a token and to read it out (§4.2). This store is the KV cache, and it is the model's only link to its own past. Anything that is not among those $T$ tokens, and not learned in the weights, does not exist for the model.

An agent runs this model in a loop (§24.1). It sends a prompt, runs the tool call the model asks for, appends the tool's output to the prompt, and sends the longer prompt again. The record of everything so far is the transcript. It holds the user's messages, the model's outputs and tool calls, and the tool results. As far as the model can tell, the prompt is the agent's entire state.

In plain words, the window is a fixed-size buffer that several kinds of content compete for. Every token placed in it is paid for five times over, and it also takes a share of the model's attention.

Two practical details matter here. First, whether old reasoning tokens stay in the window on later turns depends on the provider and the model, so check the documentation of the model you use. Second, tokens are not characters. English text runs about four characters per token with modern tokenizers (§2.1), so a budget measured in tokens moves when the tokenizer changes. Anthropic's pricing page, for example, notes that Claude 4.7 and later models use a tokenizer that produces about 30% more tokens for the same text.

Here is what the budget might look like for the agent of the opening, at turn 120. The split is illustrative, but its shape is typical of coding agents, where tool outputs dominate.

The instruction that stopped being followed is a few tokens inside the 2,000 of user messages. More than three quarters of the window is tool output, and most of it was read once and never needed again.

What one call costs

We now price the five resources one at a time, with one running example, Llama 3.1 8B reading 128K (131,072) tokens. (Llama 3.1 8B has the same layers and heads as Llama 3 8B, whose window was only 8K tokens.) Each formula comes from an earlier chapter, and here we only put them together. Two units recur. A FLOP is one floating-point operation, and a PFLOP is $10^{15}$ of them. bf16 is the 16-bit number format most models are served in, 2 bytes per number.

Cache bytes. Every layer keeps a key and a value for every token, so the cache grows in proportion to the context. Its size is

with $L$ layers, $n_{kv}$ key-value heads of width $d_k$ in each layer, and $b$ bytes per stored number (2 in bf16). The 2 in front counts keys and values, and §6.5 derives the formula. Llama 3.1 8B has $L = 32$, $n_{kv} = 8$, and $d_k = 128$. So each token costs $2 \cdot 32 \cdot 8 \cdot 128 \cdot 2 = 131{,}072$ bytes, 128 KiB, and 128K tokens take 16 GiB (17.2 GB). That is more than the 16 GB of the model's own weights.

The model has 32 query heads but only 8 key-value heads. That is grouped-query attention, in which several query heads share one key and one value (§12.5). Without it the cache would be four times larger.

Prefill FLOPs. The prefill does two kinds of work. First, each of the $T$ input tokens is multiplied by every weight matrix, at 2 FLOPs per parameter (one multiply and one add). That gives $2NT$, where $N$ is the number of parameters a token is multiplied by, everything except the input embedding table. Second, every token is compared with every earlier token. In one layer with $h$ query heads, the scores $\mathbf{Q}\mathbf{K}^\top$ cost $2T^2 h d_k$ FLOPs and the weighted sum $\mathbf{A}\mathbf{V}$ costs the same (§4.7). A causal mask lets an efficient kernel skip the half of the score matrix above the diagonal. So this attention core costs $2T^2 d$ per layer, with $d = h d_k$ the total query width. In total,

In plain words, the cost of the weights grows linearly with the context and the cost of the attention core quadratically. The core takes over once $2LdT^2 > 2NT$, that is, beyond $T^\star = N/(Ld)$ tokens. Grouped-query attention shrinks the cache but not this term, since every query head still scores every key. For Llama 3.1 8B ($N \approx 7.5$ billion, $d = 4{,}096$), the weights cost 1.97 PFLOPs at 128K and the core 4.50 PFLOPs. The crossover is at $T^\star \approx 57$K tokens.

Time to the first token, TTFT. The prefill is limited by arithmetic. Each weight read from memory is used for all $T$ tokens at once, so the GPU spends its time multiplying, not waiting for memory. So the wait before the first output token is about the FLOPs divided by the rate the hardware actually delivers (§20.1),

with $\pi$ the peak rate and $u$ the fraction of it achieved, the model FLOPs utilization or MFU (§6.7). An H100 SXM GPU peaks at 989 dense bf16 TFLOP/s, and we assume it delivers $u\pi = 400$ TFLOP/s, about 40%. The 8B model's 6.47 PFLOPs then take 16 seconds before the first output token appears. Larger models run on several GPUs, which divides the time by roughly their number.

Time per output token, TPOT. After the prefill, the model writes one token at a time, and each new token's query reads the whole cache. This step is limited by memory bandwidth, not arithmetic (§20.1). At batch size 1 each number read from memory is used for only a few arithmetic operations, so the time goes into reading every weight once and every cached key and value once,

with $M_W \approx 2N$ the bytes of weights read per step in bf16 and $\beta_{\text{mem}}$ the memory bandwidth in bytes per second. An H100 reads 3.35 TB/s. For the 8B model the 15 GB of weights take 4.5 ms to read, and at 128K the 17.2 GB cache adds 5.1 ms. Each token then takes 9.6 ms, so the model streams at half its short-context speed. (The 15 GB leaves out the input embedding table, since each step looks up only one row of it.) The weights are a fixed cost and the cache grows with the context. Once $M_{\text{KV}}(T)$ passes $M_W$, the context dominates every token's latency.

Dollars. Providers charge per token, $T c_{\text{in}}$ for the input. As of October 2026, Anthropic's list price for Claude Sonnet 4.6 is $3 per million input tokens and $15 per million output tokens. Its pricing page states that Claude 4.6 and later models (with one exception) bill their full 1M-token window at the standard per-token rate. At $3 per million, one 128K-token call costs 39 cents of input. Prices change often, so every cell below takes them as parameters.

The widget splits the cost of one call between the model's weights and the context. Slide the context length and watch the attention core and the KV cache take over.

What this changes in practice is the order of magnitude. A long context is not one big prompt. It is gigabytes of cache, seconds of waiting, and a smaller batch on every call. An interviewer who asks why long-context serving is expensive wants three numbers, the cache per token, the crossover length $T^\star$, and the decode slowdown.

def call_cost(n_layers, n_heads, n_kv, d_k, T, bytes_per, N, price_in):
    kv_bytes = n_layers * n_heads * d_k * T            # placeholder: several factors are wrong
    prefill_flops = 2 * N * T
    dollars = T * price_in
    return {"kv_bytes": kv_bytes, "prefill_flops": prefill_flops, "dollars": dollars}

The bill of an agent loop

A single call is affordable. The trouble is that an agent makes hundreds of them, and each one resends everything before it. §24.1 counted this bill. We repeat the count briefly in this chapter's notation, which the next sections use, and add two things that section did not need, the turn at which the window is full and the attention compute behind the billed tokens. Start with a loop small enough to add up by hand.

Now the general case. Let the fixed prefix be $T_0$ tokens. Let the transcript grow by $m$ tokens before each call, the model's previous output and tool call plus the tool result or user message that answers it. Then the call at turn $i$ reads $T_0 + im$ input tokens, for $i = 1, \dots, n$. (§24.1 writes $P_0$, $s$, and $T$ for $T_0$, $m$, and $n$, and lets turn $i$ read only the $i - 1$ turns before it, so its total is smaller by $m$ tokens per call. This chapter uses the convention here throughout.)

In plain words, the prefix is paid once per turn, which is linear. The transcript is paid once per turn for every turn that came before, which is quadratic. The five-turn example checks out, $5 \cdot 1{,}000 + 500 \cdot 5 \cdot 6/2 = 5{,}000 + 7{,}500 = 12{,}500$.

The tokens are what the provider bills. The compute behind them grows even faster if nothing is reused between calls. The call at turn $i$ prefills $T_0 + im$ tokens from scratch, and its attention core costs $2Ld(T_0 + im)^2$ FLOPs. Summed over turns, the billed tokens grow like $n^2$, but the attention compute grows like $2Ld\,m^2n^3/3$, the cube of the number of turns. For the 100-turn loop above on Llama 3.1 8B ($L = 32$, $d = 4{,}096$), the attention core comes to 410 PFLOPs. That is about 2.5 times the 167 PFLOPs of the weights ($2N$ times 11.1M tokens).

Serving systems avoid most of this compute. They keep the cache of a shared prefix between calls, so each call prefills only its new tokens (prefix caching). That is also why providers can sell cached input tokens at a discount, the subject of §25.2. The cell below checks the closed form against a turn-by-turn sum and plots both curves.

The widget runs the same loop one call at a time. Each bar is the input of one call, so the bill is the total area of the bars. Press Run the loop and watch the bill speed up as the bars grow.

The standard interview question gives you a prefix, a growth per turn, and a number of turns, and asks for the bill. Write $nT_0 + m\,n(n+1)/2$, point out that the transcript term dominates, and name the two levers, caching the prefix and shrinking the transcript.

def loop_tokens(T0, m, n):
    return T0 + m * n * n // 2            # placeholder

def loop_tokens_sim(T0, m, n):
    raise NotImplementedError

def turns_until_full(T0, m, T_max, reserve=0):
    raise NotImplementedError

Attention weights are a fixed budget

Now the second failure, the instruction from turn 3 that stopped being followed. Nothing was removed from the context, so the problem is not storage. The problem is how the model reads.

Recall how one head of attention reads the context (§4.1 and §4.3). The head compares the current token's query vector with the key vector of each of the $n$ tokens it can see. Each comparison gives a score, called a logit. The softmax turns the logits $s_1, \dots, s_n$ into weights

which are positive and add up to 1. The head's output is the weighted average of the tokens' value vectors. Because the weights always add up to 1, they are a fixed budget. Every token added to the context takes a share of it, whether or not it is relevant.

Call the one token the head should find the needle, as in a needle in a haystack, and every other token a distractor. Here is how the budget splits between them in the simplest case.

§4.7 met this effect as softmax dilution, for one relevant key among competitors that all score the same. The next theorem says the same thing for any logits at all.

In plain words, no key can hold more than $e^\delta$ times its fair share $1/n$. A head can focus only as far as the spread of its logits allows. Once $n$ exceeds $2e^\delta$, even its favorite key gets less than half the weight.

Are the logits of a real model bounded independently of the length? For a model with fixed weights, yes. Every attention layer of a modern Transformer normalizes its input first. So the queries and keys have bounded length however long the context is, and so do their dot products. The derivation below makes this precise. Veličković et al. prove a version for Transformers whose inputs come from a finite vocabulary (their Theorem 2.2). For any temperature $\theta > 0$ and any $\epsilon > 0$, a long enough input makes every attention weight of a head smaller than $\epsilon$. A model has finitely many heads, so a long enough input does this in all of them at once.

This is a theorem about the limit, and the worst-case constant $e^{2B}$ can be astronomically large. So it is fair to ask whether the bound bites at real lengths. One measurement suggests that it does, because real spreads are modest.

What matters in practice is a comparison of two gaps. One is the gap a trained head actually produces between the needle and the rest. The other is the gap the context length demands, which the next box states exactly.

In plain words, whatever the temperature, the effective gap $\Delta/\tau$ must grow by about one unit for every factor of $e \approx 2.7$ in context, and by 2.3 units for every factor of 10. §4.7 derived the same identity. Only a sharpening that itself grows with the context can keep up, which is where the next part of the section goes.

What does a small weight do to the head's output? The output is the needle's value scaled down by its weight, plus a blend of the distractors' values. If the distractor values are unrelated to the needle's and average to zero, the error is about $1 - a^\star$ in units of the value's length. The derivation below makes this exact.

In plain words, the head's output shrinks toward the average of the distractor values, which here is zero. The needle's value is still in the output, but scaled down by $a^\star$ and blurred. The layers above receive a signal that says almost nothing.

Real distractors do not all have the same logit. Suppose their logits are random, with mean 0 and standard deviation 1, roughly as the scores of unrelated keys are at initialization (§4.3). Each distractor adds $e^{s_j}$ to the denominator. For a standard normal $s$ the average of $e^{s}$ is $e^{1/2} \approx 1.65$. That follows from completing the square in the Gaussian integral, $\int e^{s}e^{-s^2/2}\,ds = e^{1/2}\int e^{-(s-1)^2/2}\,ds$. So a crowd of random distractors acts like 1.65 times as many equal ones, and the needle's weight should be about $1/(1 + 1.65(n-1)e^{-\Delta})$.

The cell checks this with a needle at gap 5 and random distractors. It measures the relative readout error $\lVert\mathbf{o} - \mathbf{v}^\star\rVert/\lVert\mathbf{v}^\star\rVert$. It also tries one fix, multiplying every logit by $0.3\log n$, which the next part of the section explains.

With plain softmax the measured weight matches the prediction, from 0.92 at $n = 10$ to 0.0009 at $10^5$. The relative readout error climbs from 0.09 to 0.999, so the output has lost the needle's value entirely. With the fix, the needle keeps a weight of about 0.7 to 0.8 out to $10^4$ keys. Then it too slips, to 0.46 at $10^5$, for a reason we will see in a moment.

The widget extends the dilution picture of §4.7 in three ways. You can add distractors that resemble the needle, each with a logit 80% of the needle's gap. You can compare plain softmax with a fixed temperature of 0.5 and with the length-aware fix. And it plots the readout error next to the weight, since the error is what the layers above receive.

Predict three things before you slide or press Grow the context. Set the gap to 5, no similar distractors, and plain softmax, and guess the context length at which the needle's weight crosses one half. Then add 5 distractors whose logits are 80% of the needle's, and guess whether the crossing moves by a little or a lot. Finally switch to Scalable-Softmax with $\sigma_s = 0.3$, drag the number of keys to $10^6$, and ask why the curve flattens instead of falling, and what happens if you lower $\sigma_s$ below $1/\Delta = 0.2$. The answers follow the widget.

In practice this is why a long transcript loses its early instructions without deleting them. Their tokens are still there, but each now holds a tiny share of every head's weight. An interviewer will ask you to prove the $1/n$ bound and to say what gap keeps one key at weight one half.

Sharpening attention as the context grows

If the gap a head needs grows like $\log n$, the natural fix is to make the logits grow like $\log n$ too. §4.7 met one version that is applied only at inference. Once a query sees more keys than the training length $T_{\text{train}}$, the Qwen models multiply its scores by $\log n/\log T_{\text{train}}$. The Qwen report names this LogN scaling (Bai et al. 2023), and the formula is in Qwen's released model code.

Nakanishi (2025) arrived at the same idea from data. Nakanishi trained a 162M-parameter Transformer (12 layers, 12 heads, sequence length 1,024) whose logits were multiplied by a learned factor. The factor combined a scale and an offset for each layer and head with one learnable number $p_n$ for every context size $n$, shared by all layers and heads. After training, the learned $p_n$ closely followed $a_1\log n + a_2$ for fitted constants $a_1$ and $a_2$. That led Nakanishi to build the $\log n$ into the softmax itself.

In plain words, plain softmax asks for a gap that grows with the context, while SSMax asks for a fixed gap of a bit more than $1/\sigma_s$ at every length. The catch is in the second statement. The guarantee asks the top key to beat the runner-up, not the average distractor, by more than $1/\sigma_s$. A few close rivals do no lasting harm, since their share still shrinks as $n$ grows. A crowd of close rivals whose number grows with $n$ does.

That catch explains the slip in the simulation above. There the distractor logits were standard normal, and SSMax multiplied them by $\beta = \sigma_s\log n$, a factor that keeps growing. The average of $e^{\beta s}$ for a standard normal $s$ is $e^{\beta^2/2}$, by the same completing of the square as before. So the $n$ distractors together weigh about $n e^{\beta^2/2}$, against the needle's $e^{\beta\Delta}$. Writing both as powers of $n$, the needle's weight is about

With $\sigma_s = 0.3$ and $\Delta = 5$ the exponent starts at $-0.5$, but its last term grows with $\log n$. It turns positive once $\log n > 2(\sigma_s\Delta - 1)/\sigma_s^2 \approx 11.1$, near $n = 67{,}000$, and from there the weight falls below one half. The formula gives 0.69 at $10^4$ and 0.45 at $10^5$, close to the 0.71 and 0.46 the cell measured. In plain words, the upper tail of a Gaussian crowd always holds keys within $1/\sigma_s$ of the needle, and their number grows with $n$.

Nakanishi reports three results. Models trained with SSMax lowered their pretraining loss faster than the standard model. They kept a much lower test loss well beyond the training length. And they retrieved planted facts reasonably accurately at contexts up to about ten times the training length. The deeper box below gives the conditions of those tests. An interviewer who asks about length generalization of attention expects this idea, sharpen the logits by $\log n$, with its condition $\sigma_s\Delta > 1$ and its runner-up catch.

The other option is to measure how spread out a row of attention already is, and to sharpen it only as much as it needs. The measure of spread is the entropy of the weights, $H(\mathbf{a}) = -\sum_j a_j\log a_j$, computed with natural logarithms and so measured in nats (§0.5). It is 0 when one key has all the weight and $\log n$ when the weight is uniform over $n$ keys. Veličković et al. (2025) proposed a rule that maps the entropy of a row to a sharpening factor.

All of these methods rescale logits so that focus survives length. None of them is free.

• Sharper attention amplifies a wrong match as faithfully as a right one. This matters most when distractors resemble the target, the subject of the next part.
• A learned or fixed factor tuned at one length may be wrong at another, and retrofitting it into a pretrained model only partly works.
• None of them changes the price per token. A sharper head over a 1M-token window still pays for 1M tokens of cache, compute, and money.

import numpy as np

def causal_attention(Q, K, V, ssmax=None):
    S = Q @ K.T / np.sqrt(Q.shape[-1])
    A = np.exp(S) / np.exp(S).sum(-1, keepdims=True)      # placeholder: no mask, no SSMax, unstable
    return A @ V, A

def adaptive_attention(Q, K, V):

Similar distractors hurt more

So far every distractor was alike. Real contexts are not like that. The tokens most likely to steal attention from the instruction at turn 3 are not random filler. They are other text that resembles it, such as other instructions and other mentions of the same files. The softmax arithmetic says exactly how much worse they are.

Model a head whose query and keys are unit vectors and whose logits are $s_j = \beta\cos(\mathbf{q}, \mathbf{k}_j)$. The cosine measures how closely key $j$ points along the query. The number $\beta$ is an inverse temperature, a multiplier on every logit that sets how sharp the head is. The needle's key matches the query, so its cosine is 1 and its logit is $\beta$. Measure each distractor's share of the denominator in units of the needle's share, $e^{s_j - \beta} = e^{-\beta(1 - \cos_j)}$.

For a random unrelated key in $d$ dimensions, the cosine with the query is approximately normal with mean 0 and variance $1/d$ (§1.3). So its average share is $e^{-\beta}\,\mathbb{E}[e^{\beta\cos}] \approx e^{-\beta}e^{\beta^2/(2d)}$, by the same Gaussian average as before. A near-duplicate at cosine $c$ has share $e^{-\beta(1-c)}$. The ratio of the two is how many random tokens one near-duplicate is worth.

The measured average of $e^{\beta\cos}$ over random keys, 2.145, is a little below the Gaussian approximation, 2.184. The exact average for a uniformly random direction is 2.165, and the rest of the gap is sampling noise. The approximation runs high because a cosine cannot exceed 1, so its true distribution has lighter tails than the normal. With the measured average, one near-duplicate at cosine 0.9 is worth about 3,800 random keys. With 2,000 random keys the needle keeps a weight of about 0.84. Five near-duplicates at cosine 0.9 cut it to 0.33, as much damage as about 19,000 more random keys would do.

Experiments on real models agree in direction. Shi et al. (2023) built GSM-IC, grade-school math problems with one irrelevant sentence added, and found that accuracy fell sharply. The most damaging sentences were generally on the problem's topic and reused the names of the problem's people. Whether the sentence's numbers were in the problem's range mattered less. Chroma's context rot report (Hong, Troynikov, and Huber 2025) found that a single distractor already lowered accuracy compared with the needle alone, and that four lowered it further.

§12.7 described two long-context benchmarks that bear on this. RULER, a suite of synthetic long-context tasks, includes retrieval tasks with distractor needles of the same form as the real one. NoLiMa shows the opposite face of similarity. Its planted facts share no words with their questions, and accuracy collapses with length. In the softmax picture, a head can no longer produce a large logit gap by matching words.

The softmax arithmetic does not prove what happens inside a trained model, which has many heads and layers. But it predicts the direction and the order of magnitude, and it tells you what to do. Keep near-duplicates out of the window. Stale versions of a file, superseded instructions, and repeated tool outputs are the most expensive tokens an agent can carry. An interviewer who asks why an old copy of a file hurts more than a page of unrelated logs wants this ratio, $e^{-\beta(1-c)}$ against $e^{-\beta}e^{\beta^2/(2d)}$.

import numpy as np

def readout(query, K, V, beta):
    s = K @ query
    a = np.exp(s) / np.exp(s).sum()
    return float(a[0]), float(1 - a[0])           # placeholder: no beta, and the wrong error

def breaking_cosine(cs, k, beta, threshold):

Position matters, and the middle gets lost

Where a fact sits in the window matters as well as what competes with it. Liu et al. (2023) placed the one document that answers a question among 10, 20, or 30 retrieved documents and moved it from first to last. Accuracy was highest with the answer at the start, lowest in the middle, and partly recovered at the end. This U-shaped effect is called lost in the middle. §7.6 gave its headline numbers. Here is the whole curve for one model, and then why it has this shape.

For an agent the consequence is direct. The user's original instructions sit near the start of the transcript, which helps. But as the transcript grows they sit ever further from the current position, with a long middle of tool output in between. The most recent turns hold the strong end of the curve. Why the curve has this shape is understood in part, through four mechanisms.

• The causal mask favors early positions. Wu et al. (2025) show that causal masking alone biases attention toward early tokens. In deeper layers, every later token's representation already mixes in the earlier ones, so early positions accumulate influence layer by layer.
• Relative position encodings favor recent positions. Rotary embeddings encode position by rotating queries and keys. For a query and a key with matching content, an upper bound on their score shrinks, with ripples, as their distance grows (§5.5). ALiBi, another scheme, subtracts a penalty that grows linearly with distance. So many heads favor recent tokens. The decay is a ceiling, not a law, and trained heads can still attend far back. The induction heads of §4.7, which find an earlier copy of the current token and read what followed it, do exactly that. Wu et al. analyze how the two pressures trade off across layers. In their experiments a full U-shape also needed training data that put the relevant information at both ends, so the data plays a part too.
• Attention sinks. Many heads park spare weight on the first few tokens (§12.6). That raises attention to the very start without making the start more useful.
• Undertrained distances. Long distances between query and key are rare in training, so they are learned poorly (§12.7).

Hsieh et al. (2024) measured a U-shaped bias in the attention weights directly, high at both ends of the input regardless of relevance. Subtracting an estimate of it improved retrieval-augmented generation by up to 15 percentage points.

Two practical rules follow. First, put what matters at the edges of the context, the system prompt for standing rules and the end for the current task. When a transcript grows long, restate critical constraints near the end, one of the context operations of §25.2. Second, measure on your own model, since the curve varies so much across models. An interviewer will ask what lost in the middle is, why it happens, and whether it holds for every model.

Length alone hurts, and context rot

Input length lowers accuracy even when nothing else is wrong. Suppose the model retrieves the relevant facts perfectly, nothing in the context resembles them, and they sit right next to the question. Two careful experiments show that a longer input still hurts.

The masking experiment matters for the theory. Masked tokens get exactly zero attention weight, so there is no dispersion at all. Yet accuracy still fell, by at least 7.9 points for both open models at 30K masked tokens. So the dispersion theorem is a real pressure but not the whole story. Distance itself, through positional encodings and undertrained long-range positions, degrades what the model does with information it attends to perfectly.

Du et al. propose a cheap fix that follows from this. Ask the model first to recite the evidence it needs, which places that evidence right next to the question, and only then to solve the problem. They report gains of up to 4 points for GPT-4o on RULER, over an already strong baseline.

The fair summary separates what is proven from what is measured.

A needle-in-a-haystack test hides one fact in a long filler text and asks for it back. A model that passes one at 1M tokens has shown one thing, that a head can produce a large logit gap for a fact that shares words with its question. It has not shown that it can follow a constraint stated 400K tokens ago in a transcript full of similar constraints. Nor has it shown that it can combine ten facts spread across the window. Test the task you have, at the length you will use, with distractors like the ones your agent will really carry. An interviewer who asks whether a 1M-token window means a model can use 1M tokens wants this distinction between what fits and what works.

In PyTorch

Multiplying every logit in row $i$ by $\sigma_s\log(i+1)$ is the same as multiplying query $i$ by that factor. So Scalable-Softmax needs no custom kernel. Scale the queries, then call PyTorch's fused attention with a causal mask. The block checks this against an explicit implementation. Then it measures one needle's weight against length for both softmaxes. The plain weight falls from 0.87 at 16 keys to 0.0014 at 65,536, while SSMax with $\sigma_s = 0.3$ stays between 0.56 and 0.82.

Where to keep what an agent needs

The window is one tier of a memory hierarchy, and the rest of this chapter is about the others. Each alternative trades something different.

Read more

• Veličković, Perivolaropoulos, Barbero, and Pascanu (2025). Softmax is not Enough (for Sharp Size Generalisation). The dispersion lemma and its Transformer version, measured logit spreads in Gemma, and an adaptive temperature as a partial fix. ICML 2025.
• Nakanishi (2025). Scalable-Softmax Is Superior for Attention. The empirical log-n law for learned logit multipliers, the SSMax definition, and its bounds.
• Liu et al. (2023). Lost in the Middle: How Language Models Use Long Contexts. The U-shaped curve, with full tables per model and position in the appendix. TACL 2024.
• Levy, Jacoby, and Goldberg (2024). Same Task, More Tokens: the Impact of Input Length on the Reasoning Performance of Large Language Models. The same questions padded to different lengths, isolating length from difficulty.
• Du et al. (2025). Context Length Alone Hurts LLM Performance Despite Perfect Retrieval. The whitespace and masking experiments, and the recite-then-solve mitigation.
• Hong, Troynikov, and Huber (2025). Context Rot: How Increasing Input Tokens Impacts LLM Performance. Eighteen models, distractors, haystack structure, and LongMemEval with focused and full prompts.
• Wu, Wang, Jegelka, and Jadbabaie (2025). On the Emergence of Position Bias in Transformers. Why causal masks favor early tokens and relative positions favor recent ones. With Hsieh et al. (2024), Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization.
• Shi et al. (2023). Large Language Models Can Be Easily Distracted by Irrelevant Context. GSM-IC, the cleanest demonstration that one irrelevant sentence can break reasoning.

Watch next

▶

▶