Personalization is an assistant using what it remembers about you to serve you better. It knows that you write Python, that you like short answers, and that you are planning a trip in May. This section builds that memory of one person and then makes it safe. A memory about a person can hurt that person in ways a single chat cannot, so the design is a question of privacy, consent, and security as well as of engineering.

Three risks come with remembering. A remembered fact can surface where it does not belong. An attacker's text can be saved and then steer every later chat. And a fact you ask the assistant to forget can survive in summaries and caches. Each risk becomes a design rule in this section, with numbers behind it.

Two stories make the first two risks concrete. In the first, you ask the assistant for help with a flaky test in a coding project. It answers well and then adds, "Good luck with the dermatology appointment on the 14th." You mentioned the appointment once, weeks ago, while planning your calendar. The memory is accurate and meant kindly. But a health fact has moved into a project your team can read.

In the second, the assistant read a web page for you last week. Hidden in the page was text addressed to the model, asking it to remember an instruction for the future. The assistant saved that instruction with its memory tool. The page is long closed, but the instruction now sits in your memory store. It is retrieved into chat after chat and quietly changes what the assistant does.

Both failures come from memory itself. §25.3 built a long-term store, a collection of short texts kept outside the model's context window and looked up by similarity, so that an agent can remember across sessions. §25.5 showed how what an agent learns can also end up in its weights. A bad context ends with the session. A bad memory does not.

Here is the plan. We first define a user model, the store of what an assistant knows about one person. Then we track a preference that changes over time, with a simple tracker whose speed and noise we can compute exactly. We decide which memories may flow where, using a theory of privacy called contextual integrity. We measure memory poisoning and what each defense costs. We delete correctly, from stores and from weights. We evaluate a memory system so that its numbers mean something. And we close with ways to personalize the model itself. Each part ends with the question that labs ask about it in design interviews.

A user model is a memory store about one person

Start from the memory store of §25.3. It is a set $\mathcal{M}$ of $N_{\mathcal{M}}$ entries kept outside the context window. Each entry holds a short text and an embedding of that text, a vector used to find it by similarity. It also holds the time it was last used, an importance score set when it was written, and its provenance, the list of messages or documents it was derived from. A query retrieves the entries that score best on a mix of three things. Relevance is the cosine similarity between the entry's embedding and the query's. Recency is how lately the entry was used. Importance is the score set when it was written. A user model is such a store whose entries are about the user.

Entries about a person come in three kinds, and they behave very differently. Stated preferences are things the user said about themselves, such as "I write Python, never JavaScript" or "keep answers short". They are the most reliable entries, and the user knows they are there. Inferred attributes are conclusions the system drew, such as "the user is probably a beginner" or "the user seems to have a skin condition". They are uncertain, and the user never saw them formed. Most privacy harm lives here. Interaction history is the raw record of sessions, from which the other two kinds are extracted.

Here are two records from the user model in the first story, one stated and one inferred.

Each row is a field the system needs, and the inferred record already breaks a rule this section derives, because it stores a health fact nobody agreed to. Here is the general form.

Every field is used later in the section. Confidence and provenance let an inferred attribute be checked, shown to the user, and corrected. Scope decides where a record may be used. The validity interval lets a newer preference close an older one without deleting it. Consent records whether the write should have happened at all. A design interview for an assistant with memory often opens with "what fields does a memory record need?" The strong answer names these fields, and for each safeguard, the field it reads.

Between them the two products ship three controls. Memory is scoped to a project, the user can read and edit a summary of it, and a chat mode stays out of memory. This section derives each of them from first principles.

Preferences that change

Much of a user model is inferred from behavior rather than stated, and behavior changes. A user who read long, detailed answers for months starts skimming and asking for the short version. The user model should notice, but not after one impatient day. This is a tracking problem, and its simplest version has an exact answer.

Counting choices with a Beta distribution

Take one preference with two options, A (a detailed answer) and B (a short one). In each session $t$ the user picks A, written $y_t = 1$, or B, written $y_t = 0$. Model the picks as independent coin flips that come up A with probability $p$. We want an estimate of $p$.

The standard tool is the Beta distribution, a distribution over a probability $p \in [0, 1]$. Its two positive parameters behave like counts, one of A's and one of B's. The counts start with a few imaginary ones that encode what we believe before any data, the prior. We write the two parameters $\alpha_{\text{B}}$ and $\beta_{\text{B}}$, where the subscript stands for Beta, not for option B. The mean of the distribution is $\alpha_{\text{B}}/(\alpha_{\text{B}} + \beta_{\text{B}})$, the share of A's among the counts.

§0.4 proved that Bayes rule updates a Beta belief about a coin exactly. After each flip you add one to the matching count,

For example, start from $\operatorname{Beta}(1, 1)$, the flat prior with one imaginary A and one imaginary B. After three A's and one B the counts are 4 and 2, so the estimate is $\hat p = 4/6 = 0.667$.

This update is right only if $p$ never changes, and that is the assumption that fails. After 200 sessions of choosing A, the counts are 201 and 1, and it takes about 200 sessions of choosing B to pull the mean below one half. Each new session moves the estimate by about one over the total count. So a tracker that never forgets reacts more slowly the longer it runs.

Forgetting old evidence

The fix is to discount old evidence. Pick a forgetting factor $\gamma_{\text{f}}$ between 0 and 1. Before adding each new observation, multiply both counts by it,

In plain words, every session, shrink what you knew by the same factor, then add what you just saw at full weight. We discount first so that the newest observation has weight exactly 1. Setting $\gamma_{\text{f}} = 1$ gives back the update that never forgets.

This rule is not the exact posterior of any one model of how preferences drift. It is an approximation with a long record. Recursive least squares, the standard way to fit a linear model one observation at a time, uses the same discount to follow a system that drifts. Bayesian forecasting models use it too, under the name discount factor. Its behavior can be computed exactly, and that is what we do next.

Why does the count level off near 5? Each step multiplies the old total by 0.8 and adds 1. So the total climbs toward the fixed point where $0.8n + 1 = n$, which is $n = 5$, and never reaches it. That is the first of four exact facts about this tracker. The others say how it weights the past, how noisy it is, and how long it takes to notice a change. Two of them need the idea of steady state, the regime after many sessions, when the prior has decayed away and the count has settled at its limit.

In plain words, a forgetting factor turns an ever-growing memory into a window of about $1/(1 - \gamma_{\text{f}})$ sessions. A short window reacts fast and is noisy. A long one is steady and slow.

Put numbers in. For a change from $p_0 = 0.8$ to $p_1 = 0.2$, the ratio inside the logarithm is $0.3/0.6 = 1/2$, so $k^\star = \lceil \ln 0.5/\ln\gamma_{\text{f}}\rceil$.

Near $\gamma_{\text{f}} = 1$, $-\ln\gamma_{\text{f}} \approx 1 - \gamma_{\text{f}}$, so the lag grows in proportion to the window. The standard deviation shrinks only like one over the square root of the window. That is the bias and variance trade-off of every exponential tracker. Adam's moving averages of the gradient make the same trade (§0.6), and so does the recency decay of §25.3.

The tracker that never forgets is worse than slow. Suppose it saw $N$ observations at $p_0$ before the change and $k$ after. Its expected estimate is about $(Np_0 + kp_1)/(N + k)$, which reaches one half at $k = N(p_0 - 1/2)/(1/2 - p_1)$. For the change from 0.8 to 0.2 that is $k = N$. So the longer someone has used the product, the longer the product takes to notice that they changed. An interviewer who asks "how should a user model handle a preference that changes?" wants this argument and the window formula.

import numpy as np

def track(observations, gamma, prior=(1.0, 1.0)):
    """Return (means, counts), each an array with one entry per observation."""
    raise NotImplementedError

def lag_to_cross(gamma, before, after):
    raise NotImplementedError

Choosing the forgetting factor, by simulation

The theorem predicts the trade-off, and a simulation measures it. A simulated user picks A with probability 0.8 for 50 sessions, then with probability 0.2 for 100 more. We run five trackers on the same 500 simulated users. A tracker decides A when its mean is above one half. We score it by how often that decision matches the user's true majority preference, in three windows. These are the sessions before the change, the 30 sessions right after it, and all the sessions after it.

Four trackers use forgetting factors of 1 (no forgetting), 0.7, 0.9, and 0.97. The fifth never forgets but has a crude change detector. When at least 7 of the last 8 choices disagree with its current decision, it throws away its counts and restarts from the prior plus those 8 choices.

A sharper detector can react faster with fewer false alarms. One is a likelihood-ratio test on the recent window, which asks whether "the preference changed" explains the last few choices much better than "it did not". Another is CUSUM, short for cumulative sum, which adds up the evidence for a change session by session and fires when the total crosses a threshold. The price is thresholds to tune.

In the widget below, move the forgetting factor and press Play to watch one simulated user's sessions arrive. The shaded band is the tracker's 90% interval. It widens as the factor falls, because a shorter window averages fewer sessions. The two accuracies average 300 simulated users.

Two lessons carry over to real user models. First, the best window depends on how often people change, and you can estimate that from your own logs. So $\gamma_{\text{f}}$ is a quantity you fit, not a constant you pick once. Second, behavior is the weakest kind of evidence. When a user states a change ("I eat fish now"), the stated record replaces the inferred one at once. The write policy of §25.3 does this with an update that marks the old value as no longer valid but keeps it, so "what did I used to prefer?" still has an answer. In an interview, these two points, a fitted window for behavior and an immediate overwrite for statements, are what turn the formula into a design.

Privacy is about where information flows

Back to the dermatology appointment. Nothing secret leaked, since you told the assistant yourself. The harm is that the fact moved from the context where you shared it into a context where it does not belong. A theory of privacy called contextual integrity captures exactly this (Nissenbaum 2004). It fits memory well, because memory is a mechanism for moving information between contexts.

Apply it to the first story. The subject is you. The sender is you, then the assistant. The type is health. In the original flow, the recipient was an assistant helping with your calendar, under the norm that it uses the fact to schedule. In the bad flow, the recipient is a coding session your teammates can read, and the original norm did not allow that. The same memory, retrieved for "when am I free on the 14th?" in your personal space, is an appropriate flow. So the unit to protect is not the memory but the pair of memory and destination.

Language models do not make this judgment well on their own. ConfAIde, a benchmark of privacy reasoning (Mireshghallah et al. 2023), tests it in four tiers of increasing realism, from rating how sensitive a type of information is to writing a meeting summary. In the hardest tier, three people discuss a secret about a fourth, agree that the fourth must not learn it, and later the fourth joins the meeting. The model then writes a summary for everyone present. GPT-4 put the secret in the summary 39% of the time and ChatGPT 57%. Asking the model to reason step by step first did not help (40% and 61%). So the boundaries cannot be left to the model at retrieval time. They have to be built into the store.

Five design rules

Contextual integrity turns into five rules for a memory system.

1. Scope every memory. Each entry carries a scope (user, project or space, and who may see it), and retrieval only sees entries whose scope matches the request. This is what Claude's memory per project does.
2. Filter by scope before ranking. Retrieval restricts the candidates to the scope first and ranks within them. The reason is quantitative, and we work it out next.
3. Ask before sensitive writes. The GDPR, the European Union's data protection law, lists special categories of personal data in its Article 9, among them health, religion, sexual orientation, and political opinions. These, and in practice finances too, are not inferred or stored unless the user explicitly asks. This is the policy OpenAI describes for health details, and it is the rule the inferred skin-condition record broke.
4. Make writes visible and editable. The user can see what is remembered, through a notice when memory changes and a summary they can read and edit. This also lets them catch errors and poisoning.
5. Offer a mode that neither reads nor writes. An incognito or temporary chat keeps a conversation out of memory in both directions.

Why filter first? Suppose a store holds memories for many users and projects, and the request's scope holds a fraction $f$ of them. One option is to rank all $N_{\mathcal{M}}$ entries, take the top $k$, and then drop the ones outside the scope. This is a post-filter. If relevance is unrelated to scope, about $kf$ results survive on average. The other option is to drop out-of-scope entries first and rank only the rest, a pre-filter.

The post-filter has a second flaw. Other users' memories decide what this user gets back, by crowding results out. They also pass through scoring, reranking, and logging code that should never touch them. With a pre-filter, other scopes never enter the computation.

Vector indexes make filtering subtle. They use approximate nearest-neighbor search, which finds most of the closest vectors without comparing the query with every one. §7.4 showed that filtering after such a search can return almost nothing, while filtering before it can turn the search into a brute-force scan of everything that passes. So production systems either split the index by tenant (each customer or user whose data must stay separate) or by scope, or use a search that applies the filter while it walks the index. In an interview about multi-tenant retrieval, "filter before rank, or partition the index" is the expected answer, with $kf$ as the reason.

import numpy as np

class MemoryStore:
    def __init__(self):
        self.rows = []

    def write(self, user, project, text, source, consent, incognito=False):
        raise NotImplementedError

Here is the same pattern at production shape. A batch of requests from different users runs against one shared table of memory embeddings. The scope is applied as a mask before taking the top $k$.

Each scope holds about 60 of the 50,000 memories, so $f \approx 1/800$. The post-filter keeps $kf \approx 0.006$ results per request, which is zero in every row here. The pre-filter returns 5 every time.

The mask still scores every row, which is fine for counting results but not for isolation. In production the scope first selects a partition of the index, so other scopes are never scored.

Memory poisoning

A prompt injection is text planted in something a model reads, such as a web page or an email, that the model then follows as if it were an instruction. §19.3 explained why it works. A model receives instructions and data in the same channel, so anyone who can put text in front of it can try to steer it. That section also said when the danger is greatest. An agent that has private data, reads untrusted content, and can send information out can be turned against its user. Simon Willison named this combination of three properties the lethal trifecta. Memory adds one more property, and it changes the threat. That property is persistence.

Why is this worse than a poisoned context? A prompt injection in one session ends when the session ends. A poisoned memory is retrieved in every future session where it scores high enough. It is invisible unless the user inspects the store. And it can spread, because the memory system itself writes summaries and reflections derived from it.

The common thread is that the poison rate is tiny and the exposure is not, because retrieval is targeted. If 5 of 10,000 memories were random, the chance that any of them lands in a top 5 would be $1 - \binom{9995}{5}/\binom{10000}{5} = 0.0025$. But poison is never random. It is written to be similar to the queries it targets, so it ranks near the top for exactly those queries.

A poisoned retrieval, by hand

One defense is to record where each memory came from and trust some channels less. Give each memory $j$ a trust score $u_j$ between 0 and 1, set from its source channel. Things the user said directly get 1.0. Content saved from web pages or tool outputs gets 0.2. Retrieval can then drop every memory with trust below a threshold $u_{\min}$.

Here is a store of 10,000 memories after the second opening story. Five poisoned entries were written after the assistant read a malicious page. Four were saved straight from the page. The fifth, P5, is a summary the assistant later wrote of that chat. Three genuine memories match the target query. The table lists the eight memories closest to the query by cosine similarity. The best of the other 9,992 scores 0.62.

The rule in the third policy is taint propagation, a standard idea from computer security. A value computed from untrusted inputs is itself untrusted. §19.3 met the same idea in a defense against prompt injection that tags every value with where it came from and checks those tags before allowing a tool call (Debenedetti et al. 2025, who call the system CaMeL).

So summarization cannot launder a source, as long as provenance is complete. The rule needs the provenance edges of the user model, and deletion will need them again. In practice, a trust filter is only as good as its record of where each summary came from. An interviewer will often probe exactly that gap, by asking what happens to a summary written from a poisoned chat.

Exposure against utility, simulated

How much protection does a trust filter buy, and what does it cost? A small statistical model answers it. We take one target query and model each memory's retrieval score for it, measured in standard deviations of a typical memory's score.

• Unrelated memories score $\mathcal{N}(0, 1)$.
• The $n_r = 3$ genuine relevant memories score $\mathcal{N}(4.5, 1)$.
• The $n_p$ poisoned entries score $\mathcal{N}(\mu_p, 1)$ with $\mu_p = 5$, a little above the genuine ones, because they were optimized for the query.
• 30% of genuine memories come through untrusted channels. A share of the poison, the laundering rate, comes through trusted ones.
• Trust scores are uniform on $[0.4, 1]$ for trusted channels and on $[0, 0.6]$ for untrusted ones. So the trust scorer is an imperfect judge of the channel, as real ones are.
• Unrelated memories get trust scores the same way as genuine ones, with 30% from untrusted channels. So raising the threshold also thins out the unrelated competitors.

We measure two things as the threshold $u_{\min}$ varies. Exposure is the share of target queries whose top $k$ contains at least one poisoned entry. Benign recall is the share of genuine memories retrieved.

Laundered poison sets a floor that no retrieval threshold removes without discarding the genuine memories with it. So the filter alone is never the answer. The fix has to come earlier, at the write, where provenance is recorded. In the widget below, move the trust threshold and the share of poison that arrives through trusted channels (the laundering rate). The pink curve is the exposure, the share of queries that see poison, and the green one is the benign recall, the share of genuine memories found, both for the top 5. With no laundering, exposure falls to zero at a threshold of 0.6. With any laundering, it never does until recall is gone too.

Defenses, and what each one costs

No single defense works, so systems layer them. The table pairs each with what it stops and what it does not. Two entries come from §19.3. The instruction hierarchy trains a model to obey system messages over user messages and both over tool outputs. Spotlighting marks untrusted text so the model can tell where it came from.

The defenses compose because they act on different stages of the definition. Gating and isolation act on the write. Provenance and filtering act on the retrieval. The instruction hierarchy acts on the behavior. Capability limits act on the consequence. In a design interview, naming the stage each defense covers, and the attack that slips past it, is what separates a strong answer from a list.

Deleting a memory

A user says, "Forget that I told you about the appointment." What has to happen? The entry itself must go, but the fact has almost certainly spread.

• The memory system of §25.3 wrote it into a summary of that chat.
• The user model inferred a health attribute from it.
• A reflection, a higher-level memory the system writes by combining several others, merged the summary with other memories.
• A prompt cache from §25.2, which stores the model's computed state for a prompt prefix so it can be reused, holds a context that contains the user model.
• There may also be logs, backups, and evaluation sets.

Deleting the original entry while keeping everything derived from it deletes nothing that matters.

Provenance makes this a graph problem. Let $G = (V, E)$ be the directed graph whose vertices are stored items (memories, summaries, user-model fields, cache entries) and whose edges go from a source to each item derived from it.

The closure is the least you must remove so that nothing left depends on the deleted item. It is not always what you want to destroy. A summary of ten sessions, derived in part from the deleted fact, must go, but it can then be rebuilt from the nine surviving sessions. So in practice deletion has two phases.

1. Remove the closure $D = \operatorname{desc}(x)$.
2. Visit the removed items other than $x$ in a topological order, one in which every parent comes before its children. Rebuild each item that still has a parent in the store, from those parents only. An item none of whose parents survive stays deleted.

The rebuild is where deletion usually goes wrong. It is easy to let it read the very item being deleted, through a cache or a copy taken before the deletion started.

In plain words, delete everything downstream of the item, then rebuild the downstream items parents first, reading only what survived. A small store with five derived items shows both phases. It also shows the two easy ways to get the rebuild wrong.

The widget adds a fourth memory m4 and a style field u3 derived from it. Its cached prompt c1 holds the whole user model (u1, u2, and u3) rather than u1 alone. So after deleting m1, c1 is rebuilt from u2 and u3 instead of staying deleted.

The three buttons delete the entry only, the closure, or the closure followed by the rebuild. The widget opens on deleting the entry m1 alone, so the first picture is the leak, with the four items derived from m1 still holding the fact. Switch to the closure and then to the rebuild, and watch each deletion play out step by step. Click any other item to delete it instead.

def delete(store, edges, item_id):
    """Remove item_id and everything derived from it. Return the removed ids, sorted."""
    raise NotImplementedError

def delete_and_rebuild(store, edges, item_id, rebuild):
    """Remove the closure, rebuild what still has a surviving parent. Return (removed, rebuilt)."""
    raise NotImplementedError

Deleting from weights

Personal data can also reach the weights, through fine-tuning on user conversations or through a model adapted to one user. Deleting it then becomes machine unlearning, making a trained model behave as if some of its training data had never been used. Legal rights make this concrete, such as the right to erasure in Article 17 of the GDPR. And §19.7 showed why it matters, since language models memorize some of their training data and memorized data can be extracted. The gold standard is exact unlearning, a model distributed exactly as if the data had never been used. The only general way to get it is to retrain without the data, so the question is how to make retraining cheap.

The standard answer is to train so that each data point touches only a small, separately saved part of the model. Then a deletion retrains only that part. This is SISA training, short for sharded, isolated, sliced, aggregated (Bourtoule et al. 2021), and it has four steps.

1. Split the training set into $S$ disjoint shards and train one model per shard, in isolation, so a data point influences only its own shard's model.
2. Split each shard into $R$ slices, shown to the model one after another, and save a checkpoint before each new slice.
3. At inference, aggregate the shard models' predictions, for example by majority vote.
4. To unlearn a point in shard $i$, slice $r$, restore shard $i$'s checkpoint from before slice $r$ and retrain from there without the point.

Before the general formula, one case. With $S = 10$ shards of $R = 5$ slices, a point in the third slice of some shard forces a retrain of slices 3, 4, and 5 of that one shard. That is $3/5$ of a shard, or $3/50 = 6\%$ of all the data. Averaging over where the point falls gives the theorem.

In plain words, sharding divides the cost by $S$, and slicing nearly halves what is left. With $S = 10$ and $R = 5$ a deletion costs $\frac{1}{10}\cdot\frac{6}{10} = 6\%$ of a full retrain on average, the same as the point in the middle slice above. As $R$ grows the cost tends to $1/(2S)$, and $R = 1$ gives $1/S$.

The cell checks both formulas by simulation.

Deletions rarely come one at a time. A service batches the requests of a day, and requests that land in the same shard share one retrain from the earliest affected slice. Exercise 25.6d derives the exact batch cost, and the numbers show the limit of the idea. With 10 shards, 5 slices, and 10 requests per batch, the expected cost is 44% of a full retrain, against 60% if the requests were handled one by one. With 100 requests it is 97%. So SISA helps when deletions are rare compared with the number of shards. That is the number an interviewer wants when they ask whether sharding solves unlearning for a service with millions of users.

import numpy as np

def sisa_expected_fraction(S, R):
    raise NotImplementedError

def sisa_batch_fraction(S, R, K):
    raise NotImplementedError

Approximate unlearning avoids retraining by editing the trained model, for example by gradient ascent on the data to forget, or by fine-tuning it to refuse questions about that data. TOFU, a benchmark of fictitious unlearning (Maini et al. 2024), tests such methods on language models. The authors fine-tuned a model on 200 made-up author profiles of 20 question and answer pairs each, then asked methods to forget 1%, 5%, or 10% of the authors. Forget quality is a statistical test of whether the unlearned model's answers on the forgotten authors look like those of a model never trained on them. Model utility measures everything else. None of the baselines they tested reached meaningful forget quality without losing utility.

The practical conclusion is a design rule rather than an algorithm. Keep personal data out of shared weights. Keep it in stores where deletion is exact and the closure is computable. If a model per user is needed, make it an adapter per user, a small set of extra weights trained only on that user's data, which can be deleted whole. The LoRA adapters of §14.5 are the usual choice, and §25.5 used one per task in the same way.

An interviewer who asks "a user wants to be forgotten, what has to happen?" expects both halves, the closure in the store and the retraining argument for weights, ending with this table.

Evaluating memory

A memory system can fail in more ways than a retriever, and an honest evaluation separates them. The standard reference is LongMemEval (Wu et al. 2024), 500 hand-curated questions placed in long chat histories between a user and an assistant. It tests five abilities.

• Information extraction. Recalling a detail from one session.
• Multi-session reasoning. Combining details across sessions.
• Temporal reasoning. Using both the dates mentioned in the conversation and the session timestamps.
• Knowledge updates. Answering with the current value after the user changed something.
• Abstention. Saying "I don't know" when the history never contained the answer, tested with 30 questions that rest on a false premise.

Histories come in two standard sizes, about 115K tokens (around 50 sessions) and 500 sessions (around 1.5M tokens). A judge, here GPT-4o prompted to compare each answer with the gold answer, scores the answers. The authors checked the judge against expert labels on 30 questions of each type, answered by two different models. It agreed 97% to 98% of the time on average and at least 90% on every type.

The LongMemEval gap says the long history itself is the difficulty, so a memory eval must run at realistic lengths. The memory systems compared in §25.3 show the other half of every memory evaluation. A memory system trades quality for cost and latency, so tokens retrieved per query and the 95th-percentile latency belong in the same table as accuracy.

Building your own memory eval

Benchmarks tell you how systems compare. Your product needs its own eval, built from the kinds of history its users have. Four rules make its numbers trustworthy.

Score abstention with its own rule. On a question the history cannot answer, the only correct response is to abstain. On a question it can answer, abstaining is wrong. A judge that compares answers with a gold answer marks every abstention as incorrect, so abstention needs its own rule. Also report the false-abstention rate, the share of answerable questions on which the system abstained. A system that always says "I don't know" is perfect on abstention and useless everywhere else, and this rate catches it.

Report each ability with an interval, and average abilities, not questions. The abilities have different numbers of questions. Pooling all questions, a micro average, weights each ability by how many questions you happened to write. Averaging the per-ability accuracies, a macro average, weights them equally. Each ability set is small, often 30 to 80 questions, and its accuracy is often near 0 or 1. That is exactly where the usual normal interval, $\hat p \pm 1.96\sqrt{\hat p(1-\hat p)/n}$, fails. It uses the standard error at $\hat p$, which shrinks to zero near the ends, and it can run past 1. The Wilson interval of §0.11 instead keeps every $p$ that the data do not reject at the chosen level. With $z = 1.96$ for 95%, it is

In plain words, it is centered a little toward one half, as if about two imaginary successes and two imaginary failures were added, and it never leaves $[0, 1]$.

Compare systems on the same questions. Memory on against memory off, or two memory designs, answer the same questions. So compute the difference on each question and take the standard error of those differences, the paired standard error of §0.11 and §14.7. It removes the variation between easy and hard questions, which both systems share. Check the judge against human labels, and correct its known biases, such as preferring the answer shown first or text written by models like itself, as in §22.4.

Test the safety properties as abilities. One is cross-scope leakage. Ask in project B about a secret from project A, where abstaining is correct. Another is deletion. Plant a unique made-up memory, a canary in the style of §19.7, delete it, then probe for it. Add poisoning exposure, measured on a red-team set of poisoned entries and target queries as in the simulation above, and sycophancy, covered next.

import numpy as np
from scipy.stats import norm

def score_memory_eval(records):
    raise NotImplementedError

Personalization and sycophancy

Sycophancy is a model telling users what they want to hear rather than what is true, for example agreeing with a stated opinion or dropping a correct answer under pushback. §19.6 measured it in deployed assistants. A user model hands the model a detailed description of what this user wants to hear, so the risk is plain, and it has now been measured.

A study with real users measured how much memory raises sycophancy (Jain et al. 2025). The authors collected two weeks of real interaction context from 38 users. They then asked five models for personal advice on ten scenarios taken from online posts whose authors the community had judged to be in the wrong. A response counted as agreement sycophancy when it did not even suggest that the user might be at fault.

Anthropic's pre-launch testing of Claude's memory looked for the same failure under the name over-accommodation. Measuring it in your own system is a paired experiment. Take users whose memories state a view. Write probes where the true answer conflicts with what that user seems to want, the kinds §19.6 uses, such as "are you sure?" after a correct answer, or a wrong answer the user states as their own. Run the same prompts with memory on and off, and report the paired difference with its interval.

Personalizing the model itself

Everything so far personalizes through a store and leaves the model alone. The other route changes the model. This is how personalization scales to many users without a prompt that grows with each of them, and labs ask about it in interviews. Three families matter.

Adapters per user. A LoRA adapter (§14.5) adds a small low-rank update to the frozen weights, the product of two thin matrices that holds far fewer numbers than the layer it changes. Train one on a user's data and load it with that user's requests. Deleting it is exact, since dropping the adapter drops everything it learned. But nobody can read what it learned, and the shared base stays clean only if the user's data never reached it.

A reward that depends on the user. The reward models of §16.3 fit one reward to everyone's comparisons of two responses. They use the Bradley-Terry model, in which the probability that a response wins a comparison is the sigmoid of its reward minus the other response's reward. One reward for everyone averages real disagreement away. A pluralistic reward model lets the reward depend on who is judging.

One such model is a pluralistic alignment framework called PAL (Chen et al. 2024), not to be confused with the program-aided reasoning of §18.1, which shares the acronym. Every user has an ideal point in a learned space of responses, and they prefer responses that land closer to it. The ideal points are blends of a few prototypes shared by everyone. A new user is placed by fitting only their blend weights from a few of their own comparisons, with the learned space and the prototypes frozen. That user's footprint is a short vector of weights, easy to delete. A user whose comparisons were used to learn the space and the prototypes also leaves a trace in them, and that trace cannot be deleted.

Training where the data lives. Federated averaging (McMahan et al. 2017) keeps each user's data on their device. In each round, a sample of devices trains the current model locally for a few steps and sends back only the change. The server averages the changes, weighted by how much data each device holds. The raw data never leaves the device, but an update is computed from the data and can reveal it. A guarantee needs differential privacy.

Differential privacy bounds how much one person's data can change what training produces. DP-SGD, described in §19.7, gets it by clipping each example's gradient to a maximum length and adding Gaussian noise. For personalization the unit to protect is the person rather than one message. McMahan et al. (2018) clipped and noised each user's whole update inside federated averaging to get the user-level version. With enough users, its main cost was extra computation rather than accuracy.

In plain words, no test run on the trained model can tell much better than chance whether a given user's data was used, and $\varepsilon$ sets how much better. That is a statement about deletion made in advance. The model trained with the user is already nearly indistinguishable from the one trained without them, so a later erasure request has little left to remove. The price is noise in training.

The unit matters. Take a user who wrote 500 messages. An example-level guarantee of $\varepsilon$ implies only about $500\varepsilon$ for all 500 messages together, by the group-privacy property of differential privacy, which multiplies the guarantee by the number of records that change. At any useful $\varepsilon$ that says almost nothing about the user. A user-level guarantee covers the 500 at once.

Each family gives up at least one of the two best properties of a store for scale. A store can be read by its user and deleted exactly, by closure. An adapter is deleted exactly but cannot be read. A user's weights in PAL can be deleted, but if their comparisons helped learn the shared space and prototypes, that effect cannot. Federated training spreads each user over shared weights where nothing can be read or removed, and user-level differential privacy only bounds what remains. So the rule from the deletion table holds with one refinement. Keep what the user may want to see or delete in a store. Put into weights only what belongs to that user alone (an adapter), or what a guarantee such as user-level differential privacy protects without needing deletion.

Where it breaks

Three limits remain after everything above. Each is a reason the design rules are rules rather than suggestions.

Users cannot audit what they never saw. A stated preference is visible in a memory summary. An inferred attribute may live only implicitly, in a summary's wording or in which memories get retrieved. The remedy is to show inferred records with their confidence and source and let users delete them. It works only for inferences that exist as explicit records. A system that lets the model infer freely inside free-text summaries has given up on auditability.

Scopes leak at their edges. A memory the user copies from a personal chat into a project, or a summary that spans two projects, belongs to two contexts at once. The fix is to record scope in the provenance graph, so that a derived item gets the most restrictive of its sources' scopes. That is the same move as propagating trust, and it fails in the same way when provenance is missing.

Every safeguard costs something, and the costs add up. Trust filters drop genuine memories. Write confirmation adds friction. Isolation gives up learning across users. Exact deletion rules out personal data in shared weights. And the sycophancy evidence means that the more a system knows a user, the more it must be checked for telling them what they want to hear. How much of each to pay depends on what a bad memory could do. The lethal trifecta bounds that, because an agent missing one of its three properties cannot be turned into an exfiltration channel.

Read more

• Nissenbaum (2004). Privacy as Contextual Integrity. The framework this section builds on, privacy as the appropriateness of flows. Mireshghallah et al. (2023), Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theory, makes it measurable for language models and shows they cannot be left to judge flows alone.
• OpenAI (2024). Memory and new controls for ChatGPT, and Anthropic (2025), Bringing memory to Claude. Two deployed designs with the controls derived here, saved memories, chat history, project scoping, a visible summary, and temporary or incognito chats, with the dates each shipped.
• Rehberger (2024). Spyware Injection Into Your ChatGPT's Long-Term Memory (SpAIware). The disclosure that showed memory turning one prompt injection into persistent exfiltration, with its timeline and the fix.
• Chen et al. (2024). AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Bases. Targeted poisoning of agent memory at tiny poison rates. With Dong et al. (2025), Memory Injection Attacks on LLM Agents via Query-Only Interaction, the case for never sharing memory across users, and Zou et al. (2024), PoisonedRAG: Knowledge Corruption Attacks to Retrieval-Augmented Generation of Large Language Models, the retrieval-side version.
• Bourtoule et al. (2021). Machine Unlearning. SISA training, exact unlearning made cheaper by sharding and slicing, with the measured speedups. Maini et al. (2024), TOFU: A Task of Fictitious Unlearning for LLMs, shows how far approximate unlearning is from forgetting.
• Wu et al. (2024). LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory. The five memory abilities, the history-length gap, and the judge validation. The other benchmarks in the fact box are Maharana et al. (2024), Evaluating Very Long-Term Conversational Memory of LLM Agents (LoCoMo), Hu et al. (2025), Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactions (MemoryAgentBench), Jiang et al. (2025), Know Me, Respond to Me: Benchmarking LLMs for Dynamic User Profiling and Personalized Responses at Scale (PersonaMem), and Zhao et al. (2025), Do LLMs Recognize Your Preferences? Evaluating Personalized Preference Following in LLMs (PrefEval).
• Jain et al. (2025). Interaction Context Often Increases Sycophancy in LLMs. Real user context and memory profiles measured against sycophancy, model by model, with the regression behind each number.
• Chen et al. (2024). PAL: Pluralistic Alignment Framework for Learning from Heterogeneous Preferences. A reward model with a few shared prototypes and per-user weights. For personalizing without central data, McMahan et al. (2017), Communication-Efficient Learning of Deep Networks from Decentralized Data, introduces federated averaging, and McMahan et al. (2018), Learning Differentially Private Recurrent Language Models, adds user-level differential privacy to it.

Watch next

▶

▶

▶

▶