This section is about deciding what an agent's model reads on each call. An agent calls its model once per turn, and the simplest agent sends its whole history every time. That history grows without limit. The window the model can read is fixed, and every token in it is paid for on every call. So before each call something must decide which tokens go in, in what form, and in what order. This section builds the tools for that decision, prices each one, and shows which ones production agents use.

§25.1 priced a long context in dollars, memory, and time. It also showed that a model uses a long context less reliably, because its attention spreads thinner as the window fills. Its running example states the problem in numbers.

The bill grows with the square of the number of turns, because every turn re-reads all the turns before it. Yet the task never needs all 210,000 tokens of prefix and transcript at once. A directory listing from turn 4 was useful at turn 5 and almost never again. A constraint the user stated at turn 3 matters at every turn. A good context keeps the constraint and drops the listing.

Here is the plan. We name the operations that make such choices. We derive when each one pays for itself, and we implement the ones interviewers ask about. Then we test them together on a simulated agent that runs for 300 turns.

The context is a cache over the history

Two words need to be kept apart. The history is everything the agent has seen or produced so far, every message, tool result, and thought. The context is the sequence of tokens actually sent to the model on one call. The simplest loop sets the context equal to the history. Every technique in this section sends a smaller or cheaper context instead.

It helps to think of the context as a cache. In computing, a cache is a small, fast store that holds copies of a few items from a large, slow one. A miss is a request for an item the cache does not hold. Here the large store is the history plus everything the agent could fetch again, such as files, search results, and notes. The context is the small store. Its capacity is the window $T_{\max}$ minus the room the model's answer needs, and every token in it is paid for on every call.

A miss costs one of two things. Either the model makes a mistake, because it never saw what it needed, or the agent spends an extra tool call to fetch the item again. Every policy below decides what the cache holds, and the right one depends on which kind of miss is cheaper for your task.

Selecting, ordering, and clearing are cheap and mechanical. Compaction needs an extra model call. Caching needs only a disciplined layout. Learned compression and eviction need access to the model or the server. Offloading and isolation change how the agent works. We take them roughly in that order. When an interviewer asks how you would manage the context of a long-running agent, they want to hear these operations and the price of each.

A sliding window keeps the newest messages

The simplest policy keeps the newest messages that fit in a budget and drops the rest. It is called a sliding window, because the kept span slides forward as new messages arrive. Two refinements make it usable.

The first is pinning. A pinned message is kept whatever its age. The system prompt sits in the fixed prefix anyway. But the first user message usually states the task and its constraints ("never edit the generated files"), and dropping it is exactly how an agent forgets its instructions. So the policy pays for the pinned messages first, then fills the rest of the budget with the newest unpinned ones.

The same shape appears inside the model, for a different reason. A model reading an endless stream keeps working if it holds the first few tokens plus a recent window (§12.6). Attention heads park the weight they do not need on those first tokens, called attention sinks, and the model's output degrades sharply when those tokens are evicted (Xiao et al. 2023). The resemblance is only an analogy. Sinks are kept because of how softmax heads behave. We pin the first message because of what it says.

The second refinement is contiguity. When the next-oldest message does not fit, the window stops there, even if a smaller, older message would still fit. A window with holes shows the model a conversation that never happened, such as an answer without its question or a tool result without the call that produced it. Chat APIs also require each tool result to follow its tool call, so a hole can make the request invalid outright.

A sliding window caps the context, so each turn costs at most a fixed amount. The total then grows linearly in the number of turns instead of quadratically. Its weakness is that it forgets by age alone. A decision made at turn 20 and still in force at turn 200 vanishes when it scrolls out, unless someone pinned it.

It has a second, hidden cost. A window that drops one old message every turn changes the tokens right after the pinned part of every request. The part on caching shows that this is the worst possible layout for a prompt cache. Interviewers like this point because it is a trap. The policy that looks cheapest can cost the most once caching is on.

def select_messages(tokens, budget, pinned):
    raise NotImplementedError

Clearing old tool results keeps a reference

Look at where an agent's tokens come from. A coding agent reads files of thousands of tokens, runs test suites that print pages of output, and lists directories. Its own messages are a few hundred tokens each. In the example budget of a coding agent at turn 120 in §25.1, tool results were 78% of the input. In the simulation later in this section they are 92% of all tokens. Once the agent has read a tool result and acted on it, the raw result is rarely needed again. If it is, the agent can run the tool again, provided it knows which call produced it. That works for reads whose source has not changed. The output of a command with side effects, or of anything that has changed since, cannot be fetched again, so its findings must be written down.

Tool-result clearing uses this. It keeps the newest few tool results intact and replaces each older one with a short placeholder. The placeholder keeps a reference, the file path, command, URL, or record id that produced the result. The tool call and its result stay in place as a pair, so the conversation still makes sense and the API still accepts it. If a later step needs the content, the agent uses the reference to fetch it again at that moment. Fetching content only when a step needs it is called just-in-time retrieval.

The rule behind clearing is that a compression should be restorable. The team behind the Manus agent states it this way. A web page can leave the context as long as its URL stays, and a document can leave as long as its path stays (Ji 2025). Anthropic's guide to context engineering calls clearing one of the safest, lightest-touch forms of compaction (Anthropic 2025a).

The Claude API offers clearing as a server-side setting, still in beta as of October 2026 (context editing). By default it starts clearing once the prompt exceeds 100,000 input tokens, and it keeps the 3 most recent tool uses. An option called clear_at_least skips a clear unless it removes enough tokens to be worth breaking the prompt cache, a trade we price later in this section.

Clearing loses exactly one thing, the verbatim content of old tool results that the agent never wrote down. That points to the habit that makes clearing safe. When a tool result contains something that will matter later, the agent should restate it in its own message or in a notes file. Examples are an error message, a version number, or the one relevant line of a long log. Clearing never touches the agent's messages or its notes. The simulation later in this section shows the effect in numbers. In an interview, the answer to "what does clearing lose" is "only what the agent did not write down, and even that can usually be fetched again through its reference".

def clear_tool_results(msgs, keep, placeholder=30):
    return [dict(m) for m in msgs], sum(m["tokens"] for m in msgs)   # placeholder: clears nothing

Compaction draws a sawtooth

Clearing shrinks tool results but leaves every message the agent wrote, and over hundreds of turns even those fill the window. Compaction is the heavier tool. When the context grows too long, the agent asks a model to summarize the older part of the transcript, then continues from the summary.

What goes into the summary decides what survives. Anthropic describes Claude Code's compaction as keeping architectural decisions, unresolved bugs, and implementation details. It discards redundant tool outputs, and it restarts from the summary plus the five most recently accessed files (Anthropic 2025a).

Provider APIs now compact for you. As of October 2026 the Claude API offers server-side compaction, in beta, which replaces older turns with a summary the model writes. You can ask for it on demand, or let the API run it when the input reaches a token threshold. That threshold defaults to 150,000 input tokens and cannot be set below 50,000.

Now watch the length of a compacting agent's transcript over time.

The height of the tooth is a trade. Tall teeth, from a high threshold, mean few compactions but long, expensive contexts on every turn. Short teeth mean cheap turns but many compactions. There is a best tooth height, and it has a formula.

The formula covers the case with prompt caching too, which this section explains fully later. For now, three prices are enough. A provider that keeps the stored keys and values of a repeated prefix charges $c_{\text{read}}$ for each token it reuses, often a tenth of the fresh price $c_{\text{in}}$. It charges $c_{\text{write}}$ for each token it stores for later, a little more than $c_{\text{in}}$ (§24.6 priced these for agent loops). An output token costs $c_{\text{out}}$.

In plain words, a compacting agent pays a holding cost for every token it carries and a fixed cost every time it compacts, and the best tooth height balances the two. A shop that restocks faces the same choice. Holding stock costs money every day, and every order carries a fixed fee. Inventory theory's answer, the economic order quantity (Harris 1913), is to order $\sqrt{2 \cdot \text{demand} \cdot \text{fee}/\text{holding cost}}$ units at a time. The same square-root law sets how often a long training run should save a checkpoint, balancing the cost of saving against the work a crash would lose (§10.7).

The last case explains why production agents compact late, near the window rather than at about 18,000 tokens. The token bill alone says compact early. The information a summary loses says compact rarely. That loss is hard to measure, so it is usually set by judgment. MemGPT evicts only when the window is full. The Claude API starts tool clearing at 100,000 tokens by default and server-side compaction at 150,000, and it does not allow a compaction trigger below 50,000.

Length also has a cost of its own, and it pulls the other way. §25.1 showed that attention spreads thinner and distractors do more harm as the context grows. So a long context costs accuracy as well as tokens. Use the formula to see the trade, not as the setting.

The next cell checks the formula against a simulated sawtooth and prints the cost per turn at several thresholds.

The thresholds sit a whole number of turns above $T_s$, as the derivation assumes. Per turn, compacting at 11,000 costs about 34,300 equivalents, the optimum about 30,650, and 101,000 about 66,800. The curve is flat near its minimum, since 21,000 costs only 31,000. That flatness is the other face of the square-root law. Missing the optimum by a factor $\rho$ in tooth height raises the variable part of the cost only by the factor $(\rho + 1/\rho)/2$, which is 1.25 at $\rho = 2$. It is why a threshold set by information loss rather than by tokens costs less than you might fear.

The widget runs the same agent with a 200,000-token window. The top plot is the context each call reads. The bottom plot is the long-run cost per turn of every threshold, with a dot at the one you chose.

One warning about summaries of summaries. Each compaction rewrites the previous summary, and repeated rewriting erodes detail. Zhang et al. (2025) call this context collapse. Their remedy is to update a structured record a little at a time instead of rewriting it whole, which §25.3 takes up as procedural memory. In an interview, this is the answer to "what goes wrong if you compact every few turns for a week".

import numpy as np

def compaction_cost(T0, m, n, T_c, T_s, c_in, c_out):
    raise NotImplementedError

def optimal_threshold(T0, m, T_s, c_in, c_out):
    raise NotImplementedError

Prompt caching is the one exact saving

Every operation so far throws something away. Prompt caching throws nothing away and still cuts the input bill by a factor close to $c_{\text{in}}/c_{\text{read}}$, the ratio of the fresh input price to the price of a cached token. That is ten times at the common read price of $0.1\times$, and up to forty times on the models with the lowest read prices. It works on the re-reading itself. Every turn resends the same history, and caching makes re-reading that history cheap.

The mechanism is the one §24.6 explained. Under the causal mask a token's keys and values depend only on that token and the tokens before it (§6.5). So a server can keep the KV cache of a prefix and skip the prefill, the pass that computes those keys and values, the next time the same prefix arrives. Providers pass the saving on as a discount on cached tokens, and §20.6 shows how servers store and share these caches.

§7.7 previewed caching for a fixed set of documents, and §24.6 priced it for an agent loop and listed the rules that keep a cache warm. Here we restate that bill in this chapter's notation, then add what context management needs on top of it. When does a cached prefix pay? How long must it live? And how does each operation of this section interact with it?

The prices behind these three rates differ by provider and change often, so here are the terms this section uses.

Now count the cached loop in this chapter's convention. Turn 1 finds an empty cache and writes its whole prompt of $T_0 + m$ tokens. Each later turn $i$ reads the previous prompt, $T_0 + (i-1)m$ tokens, and writes its $m$ new ones, all within the lifetime. Here are three turns by hand.

Summed over $n$ turns, the same pattern gives the theorem of §24.6, here in this chapter's convention.

In plain words, caching keeps the bill quadratic and divides its leading constant by $c_{\text{in}}/c_{\text{read}}$. Only shrinking what is re-read, as compaction does, changes the exponent. That distinction is a favorite interview probe.

The ratio falls from 0.26 at ten turns to 0.12 at a hundred, and toward its limit of 0.1. Short loops gain less, because the extra write price on every new token is a larger share of their bill.

Caching and compaction multiply

Caching and compaction attack different parts of the bill. Caching lowers the price of each re-read token, and compaction lowers how many tokens are re-read. Case 2 of the compaction theorem combined them and predicted a threshold of about 40,900 tokens. The next cell simulates the cached, compacting loop over 3,000 turns and searches a grid of thresholds. After a compaction, the next call reads only the cached prefix and must write the new summary and its turn.

The simulated optimum is the grid point nearest the formula's 40,917. At the optimum a turn costs about 7,680 equivalents, four times less than the 30,650 at the best threshold without caching. Compacting at the uncached optimum of 17,649 would cost about 9,400 per turn with caching, 22% more. So a threshold tuned without caching is wrong once caching is on. Compacting late, at 100,000, costs about 9,540, only 24% above the optimum, because cached tokens are cheap to carry. Without caching the same late threshold cost more than twice its optimum. The two tools multiply, and production agents use both.

When a cached prefix pays, and how long it must live

A write costs more than plain input, so a prefix that is never reused is a loss. How many reuses make it worth caching?

The proof is the inequality itself, solved for $r$. Both sides scale with $P$, which cancels. The harder question is whether the reuse happens in time. A hit needs the next request to arrive before the entry expires. Every use refreshes the entry, so a request misses exactly when the gap since the previous request exceeds $t_{\text{life}}$.

To put numbers on that, model the gaps between requests as independent and exponential with mean $\bar t_{\text{gap}}$. That is a reasonable model for a person who replies at random moments. Each request after the first then hits with probability $1 - e^{-t_{\text{life}}/\bar t_{\text{gap}}}$. It pays $c_{\text{read}}$ on a hit and $c_{\text{write}}$ on a miss, so the expected price of the prefix per request is

In plain words, a cache is worth its write price only if requests come back faster than it forgets. The 1-hour lifetime is insurance against slow humans and long tool runs. The cell prints the hit probability and the cost of 20 requests for both lifetimes.

Laying out a context so the cache hits

The matching rule, identical tokens from the first one, turns into a layout rule. Put what never changes first, the tool definitions and then the system prompt. Put what changes slowly next, such as a project summary or long documents. Put the conversation last, where it only grows at the end. Then audit everything that could change an early token.

This explains the clear_at_least option of tool clearing. Say the agent clears a result that sits $j$ turns back. Every token after that result now follows different tokens, so its cached keys and values no longer apply. The next call must write all of them again at $c_{\text{write}}$. Clearing one result per turn pays that rewrite on every turn. Waiting until several results can be cleared at once pays it rarely. The simulation below measures the difference. Here is what the settings look like with the Anthropic SDK.

In the widget, each of eight requests of one conversation has its own lane, shaded where the request before it keeps the cache alive. Drag a request out of its band, switch to the 1-hour lifetime, or put a timestamp in the system prompt, and watch which requests still hit.

def bill(requests, ttl, c_in, c_write, c_read, min_tokens=1024):
    total, per = 0.0, []
    for t, blocks in requests:
        n = sum(s for _, s in blocks)
        total += c_in * n                      # placeholder: no caching at all
        per.append((0, 0, n))
    return total, per

Five policies on one simulated agent

Now put the pieces together on one agent and compare them. The setup has four parts.

• The agent runs 300 turns with a 128,000-token window, a fixed prefix of 10,000 tokens, and a pinned 500-token task statement.
• Each turn it writes a message of 100 to 300 tokens and receives one tool result. Tool result sizes are lognormal, meaning their logarithm is normally distributed, so most results are modest and a few are huge. The median is 1,500 tokens, and sizes are truncated at 20,000 as agent harnesses do.
• Forty facts that matter later are planted at random turns. Each lands either in the agent's own message or only in a tool result, with equal odds. Here 18 of the 40 land in the agent's messages.
• The prices are $3 per million input tokens, $15 per million output tokens, $1.25\times$ input for cache writes, and $0.1\times$ for reads.

We compare five policies, each with and without caching.

1. Doing nothing.
2. A sliding window that keeps the newest messages fitting in 80% of the window.
3. Tool-result clearing on every turn, keeping the five newest results.
4. The same clearing in batches, run only once five more results have piled up.
5. Compaction into a 4,000-token summary whenever the context passes 80% of the window.

A fact counts as kept if it is still in the context verbatim, or if it made it into the summary. Whether a summary keeps a fact is the one thing we cannot simulate honestly. We assume each compaction keeps each fact it sees with probability 0.8, and that number is an assumption, not a measurement.

Three lessons carry over to real agents. First, where tool outputs dominate the tokens, clearing them gets most of the savings and loses only what the agent never wrote down. So agents should note important tool findings in their own words. Second, judge every policy together with caching. A policy that edits the prefix often can cost more than one that keeps a longer but stable context. Third, combine policies. Production systems clear first, in batches, and compact only when clearing is no longer enough.

A learned model can compress the prompt

Compaction compresses with a full model call that writes text. Two lines of research compress harder, at the price of running a model of your own.

The first trains the model to squeeze a prompt into a few special tokens, called gist tokens (Mu, Li, and Goodman 2023). During instruction tuning, a handful of new tokens are inserted after the prompt. The attention mask is the table of which positions may attend to which (§4.4). It is changed so that everything after the gist tokens can attend to them but not to the prompt itself. The only route from the prompt to the answer then runs through the gist tokens' keys and values, so training forces the model to store what the answer needs there.

At inference the prompt is run once, the gist tokens' keys and values are kept, and the prompt is thrown away. On LLaMA-7B and FLAN-T5-XXL this gave up to 26 times compression of prompts. That meant up to 40% fewer FLOPs (floating-point operations) and a 4.2% faster wall-clock time, with little loss in output quality.

The second works on the text and needs no change to the model that reads it. A small language model scores each token of the prompt by how surprising it is, its negative log probability given the tokens before it. A token the small model predicts easily carries little information, so the least surprising tokens are dropped. This is the method of LLMLingua (Jiang et al. 2023). It adds a budget controller that compresses the examples in a prompt harder than the instruction and the question. An iterative pass then accounts for dependencies between the tokens it keeps. The small model is also instruction-tuned so that its idea of surprise matches the larger model's. The output is a shorter prompt that any model can read, including a closed one behind an API. The paper reported up to 20 times compression with little loss in performance on four datasets, math word problems (GSM8K), hard reasoning tasks (BBH), real chat conversations (ShareGPT), and summaries of recent arXiv papers.

Here is the gist mask, checked on a toy two-layer attention stack. After a full pass with the mask, the outputs of the later tokens can be recomputed from the gist tokens' cached keys and values alone, with the prompt discarded.

The difference is at the level of float32 rounding, because the mask makes the two computations identical. Whether the gist tokens carry enough of the prompt is what training decides, and that is where the quality loss lives. In practice the choice is about access. Text-level pruning works with any API model, while gist tokens need you to train the model.

The server can compress the KV cache

Everything so far edits the tokens that are sent. A different layer of the stack keeps every token in the request but evicts entries from the KV cache that the server holds while it generates. §12.6 introduced these eviction methods next to query-aware selection, which keeps every entry and reads only some. Here we take the two eviction methods most often asked about, build one in an exercise, and ask what eviction means for an agent.

Recall that generation has two phases (§6.5). The prefill reads the prompt in one parallel pass, and the decode then writes one token at a time, each new token reading the whole cache. A smaller cache saves GPU memory and makes each decode step faster. Only whoever runs the model can do this.

The first idea starts from an observation. In a trained model, a small set of tokens receives most of the attention, and the paper calls them heavy hitters. So keep a fixed budget of cache entries, split between the most recent tokens and the tokens with the largest attention summed over the steps so far, and evict the rest. This is the heavy-hitter oracle, H2O (Zhang et al. 2023). Its accuracy was checked on OPT, LLaMA, and GPT-NeoX models. Its throughput was measured on OPT-6.7B and OPT-30B with a KV cache budget of 20% of the full cache, split between heavy hitters and recent tokens. There it reported up to 29 times the throughput of two inference systems that move the cache off the GPU, DeepSpeed Zero-Inference and Hugging Face Accelerate, and 3 times that of a third, FlexGen. The attention-sink window from earlier in this section is a fixed version of the same idea, with the first few tokens as permanent heavy hitters.

The second idea compresses a long prompt once, before generation starts, with no training. The last few tokens of the prompt form an observation window that usually contains the question, and they show which earlier positions the model will need. SnapKV (Li et al. 2024) sums how strongly the observation window attends to each earlier position. It pools those votes over neighboring positions, so that the neighbors of important tokens survive too. Then it keeps the top positions separately for each attention head.

KV eviction fails in a specific way. It keeps what the model has attended to so far, and an agent's next question may need what nobody has looked at yet. A token evicted at step 100 is gone when the user asks about it at step 500. SnapKV's observation window is a partial answer, since it scores the prompt by what the question at its end attends to. But it only works for questions already in the prompt. So KV eviction suits long single-shot generation better than long-lived agent contexts. There, the safer moves are edits to the text that keep references, so that anything can be fetched again. An interviewer who asks you to compare eviction with clearing wants this point, reversibility.

import numpy as np

def window_mass(A, budget, n_sink=0):
    raise NotImplementedError

def h2o_mass(A, budget, n_recent):
    raise NotImplementedError

On the setup's trace with a budget of 20 keys, a pure window keeps 0.22 of the attention mass, a window with four sinks 0.49, and heavy hitters with 10 protected recent keys 0.91. The sink alone more than doubles what a window delivers, the lesson of §12.6 again, and accumulated attention finds the keys that keep coming back. The trace is built to have heavy hitters, so the size of the win here says more about the trace than about real models. The papers measure it on real attention.

Notes outside the window survive compaction

A scratchpad is the reasoning a model writes out before its answer (§18.1). It lives inside the context, so it is cleared, compacted, or slid out like everything else. The step that makes long work possible is to write notes outside the window, in a file or through a memory tool, and read them back when needed. The notes survive every compaction because no compaction touches them.

Anthropic's guide gives three examples (Anthropic 2025a). Claude Code keeps to-do lists, and custom agents keep a NOTES.md file. The third is an agent that played Pokémon over thousands of game steps. It kept tallies of its progress and objectives in notes, read them after each context reset, and continued where it left off.

Notes are an offload with a write policy. The agent decides what is worth writing, and that decision is the hard part. The simulation showed it, since the facts lost under clearing were exactly the ones the agent never wrote down.

Notes also serve the order operation. §25.1 showed that models use the end of a long context more reliably than its middle. Manus exploits this with a to-do file that the agent rewrites as it works, so the current plan keeps reappearing near the end of the context as the work goes on (Ji 2025). The goal never sinks into the middle of a 50-call transcript, and the agent drifts from it less.

The complement of notes is just-in-time retrieval. Instead of loading every file it might need, the agent keeps lightweight identifiers, such as paths, queries, and URLs, and loads content when a step needs it. Often it loads only part of a large file, with head, tail, or grep. This is how a coding agent works in a repository far larger than its window (§24.5), and it is why tool clearing keeps references.

A notes file is the simplest memory outside the window. When notes grow into thousands of entries that must be searched, updated, and forgotten, they become a memory store, with retrieval scores and write policies of their own (§25.3). When the notes record the state of the task itself, such as a progress log and a list of features with their test status, they become the durable state of an agent that runs for days (§25.4).

Sub-agents keep exploration out of the lead's context

The last operation changes who does the work. Some subtasks need a lot of reading but produce a short answer, such as finding every caller of a function, surveying ten papers, or checking twenty log files. The lead agent can hand such a subtask to a sub-agent. A sub-agent is a separate loop of model calls with a fresh context window and a self-contained brief. It may read tens of thousands of tokens, and it returns a condensed summary, often 1,000 to 2,000 tokens in Anthropic's description (Anthropic 2025a). The lead's context grows by the summary, not by the exploration.

Isolation has three costs. First, the sub-agent does not have the lead's context, so the brief must carry everything it needs. Whatever the brief leaves out, the sub-agent must rediscover or get wrong. Second, parallel sub-agents often read the same material, so total tokens can rise even as each context shrinks. Third, the summary is a compression, with all the risks of compaction.

The measured trade is in §24.7. Anthropic's multi-agent research system used about 15 times the tokens of a chat. It beat a single agent by 90.2% on Anthropic's internal research evaluation, and on a browsing benchmark token usage alone explained 80% of the variance in performance (Anthropic 2025c). Read that way, sub-agents win largely because they let a task spend more tokens than one window holds. The trade is worth making for broad, valuable tasks and not for small ones.

How to split work among agents, coordinate them, and combine their results is the subject of §24.7. The point here is narrower. A sub-agent is a context-management tool, a way to spend tokens in a window that is thrown away afterwards.

Context engineering is one design problem

Each operation is simple on its own. The difficulty is that they interact. Clearing breaks the cache unless it is batched. A sliding window breaks it every turn. Compaction resets it once per tooth. Notes cost writes and reads, and sub-agents trade window space for total tokens.

Even the fixed prefix is a design choice. In the example budget of §25.1, tool definitions were 12,000 of the 15,000 tokens of fixed prefix. Deferring all but the most used tools behind a tool search (§24.6) shrinks $T_0$. Every call pays $T_0$, and so does every compaction. The choice interacts with the cache too, because tools are hashed first. Definitions appended to the conversation when found keep the prefix stable. A tool list rewritten per request invalidates the whole cache whenever it changes.

The cache matters this much because an agent's bill is mostly input. Manus reports that its tasks average about 50 tool calls and that input tokens outnumber output tokens about 100 to 1. For this reason its team calls the KV-cache hit rate the single most important metric for a production agent (Ji 2025). Anthropic's guide gives the whole activity a name.

Treat a context policy like any other component that can fail, and log it. For every call, record the token count of each source (system prompt, tool definitions, transcript, tool results, notes, retrieved material). Record the cache read and write counts the provider returns, every clearing and compaction event with what it removed, and the outcome of the step and of the task. With those logs you can replay a task under a new policy. You can see that a regression came from a summary that dropped the user's constraint, and you can measure the cache hit rate before and after a change to the system prompt. Evaluate policy changes on a fixed task suite with repeated runs, as agent evaluation does for any agent change. Report cost per successful task, not only the success rate.

The table puts the operations side by side.

Read more

• Anthropic Applied AI team (2025a). Effective context engineering for AI agents. September 29, 2025. The definition of context engineering, compaction in Claude Code, tool-result clearing, notes, and sub-agents, from the people building them.
• Anthropic (2025b). Managing context on the Claude Developer Platform, with the prompt caching, context editing, and compaction documentation. Context editing and the memory tool with their measured gains, and the caching terms, ordering, invalidation rules, and clearing and compaction defaults used in this section. Anthropic (2025c), How we built our multi-agent research system, for the token cost of sub-agents.
• Ji (2025). Context Engineering for AI Agents: Lessons from Building Manus. July 18, 2025. Production lessons from one agent company, above all the KV-cache hit rate as the metric to watch and masking tools rather than removing them so the prefix stays stable, plus the file system as restorable memory and a to-do list recited at the end of the context.
• OpenAI. Prompt caching. Minimum lengths, lifetimes, write and read prices, and routing for the other major provider.
• Packer et al. (2023). MemGPT: Towards LLMs as Operating Systems. The context window managed as main memory, with memory-pressure warnings, eviction, and recursive summaries. Read it before §25.3.
• Wang et al. (2023). Recursively Summarizing Enables Long-Term Dialogue Memory in Large Language Models. Summaries built from summaries, the recursion behind compaction.
• Mu, Li, and Goodman (2023). Learning to Compress Prompts with Gist Tokens, and Jiang et al. (2023), LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models. Learned compression inside the model and on the text.
• Zhang et al. (2023). H$_2$O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models, and Li et al. (2024), SnapKV: LLM Knows What You are Looking for Before Generation. KV eviction by accumulated attention and by an observation window.
• Zhang et al. (2025). Agentic Context Engineering: Evolving Contexts for Self-Improving Language Models. Why repeated rewriting collapses a context, and incremental updates that avoid it.

Watch next

▶

▶

▶