This section is about the infrastructure that connects an agent to the tools it uses. Every tool call has to be described to the model, carried to the program that runs it, checked against what the agent is allowed to do, paid for, and retried when the network fails. None of this is the model's job. All of it decides whether an agent works in production, and interviews for agent roles probe it as hard as they probe the model.

Recall the agent loop of §24.1. A language model reads the history of a task and writes text. A parser turns the text into a tool call, such as a search or a database query. A program runs the call and returns an observation, the tool's output as text, which is appended to the history. Then the model writes again. This section lives in the gap between the parser and the program, where a call travels to a tool and its answer comes back.

Two numbers show why that gap matters. The first is integration work. Suppose a company runs 10 agent products, such as a coding assistant, a support bot, and a research agent. Between them they use 40 tools, such as the issue tracker, the customer database, and the calendar. If each product team writes its own connector to each tool, with its own login code, error handling, and description for the model, that is up to $10 \times 40 = 400$ connectors to build, secure, and keep working.

The second number is cost. One of these agents runs 50 turns per session. It starts from a prompt of 5,000 tokens and adds 1,000 tokens per turn. Every turn re-reads everything before it, so a session reads 1,475,000 input tokens. At $3 per million input tokens that is $4.43 a session, and a million sessions a month cost $4.4 million in input alone.

Both problems have engineering answers, and this section builds them from the bytes on the wire up.

• A shared protocol turns the 400 connectors into 50. We read the Model Context Protocol (MCP) message by message and write a server for it.
• A second protocol, Agent2Agent, handles work handed to another company's agent.
• Tool search keeps hundreds of tools from flooding the prompt, and we measure how much it can cost in accuracy.
• A permission engine and information-flow labels decide what an agent may touch and where data may go, even while it reads text written by an attacker.
• A cost and latency model, with prompt caching, parallel calls, safe retries, and budgets, cuts the $4.43 session to about $0.63 and makes the bill predictable.

Why agents need a protocol

A protocol is an agreement on the messages two programs exchange. With one in place, either side can be written without knowing the other's code. Before agent protocols, every agent framework described tools in its own format, and every tool vendor wrote one integration per framework.

Code editors had the same problem once. Every editor needed its own plugin for every programming language, to offer autocomplete and go-to-definition. The Language Server Protocol put one message format between editors and languages. Each editor now implements the client side once, and each language implements the server side once. MCP's specification names that protocol as its inspiration.

The saving is not only in code. Each bespoke connector carries its own credentials, logging, and bugs, so security review, auditing, and upgrades also scale with $NM$. With a protocol they scale with $N + M$, and a fix to one server reaches every host at once. The price is that the protocol becomes a dependency of every agent. How a request looks, how errors are reported, and who may call what are now decided for all agents together. That is why we read it carefully.

A protocol for agents has three jobs.

• The wire format carries calls and results in a form both sides parse the same way.
• Discovery lets a host learn what a server offers, including the arguments each tool takes.
• Authorization says who may do what.

MCP does all three for the link between an agent and its tools. The Agent2Agent protocol, later in this section, does them for the link between two agents. An interviewer who asks "why MCP?" wants the $NM$ against $N + M$ count, and then the price, which is that one design now binds every agent.

JSON-RPC, the wire format underneath

A remote procedure call (RPC) is a function call that runs in another program, often on another machine. The caller sends the function's name and its arguments as a message. The other side runs the function and sends back the result. JSON-RPC 2.0 writes those messages as JSON objects, and its whole specification fits on one page. MCP and the Language Server Protocol use it, and so does one of the Agent2Agent protocol's three bindings. It is worth knowing exactly, because interviewers ask about it and because many protocol bugs are violations of it.

Here is one call of a function add with two arguments, and its answer.

{"jsonrpc": "2.0", "id": 7, "method": "add", "params": {"a": 2, "b": 3}}
{"jsonrpc": "2.0", "id": 7, "result": 5}

The caller chose the id 7, and the answer carries the same 7 back. A message with no id at all, such as {"jsonrpc": "2.0", "method": "log", "params": {"text": "started"}}, is a one-way note that gets no answer. Those three shapes are the whole format.

In plain words, a request says "please run this, and answer with this id". A response says "here is the answer to that id". A notification says "for your information, do not answer". The id is the only thing that ties an answer to its question, because responses may arrive in any order. Losing it, or changing its type from the number 7 to the string "7", breaks the tie.

MCP adds three rules of its own. First, a request id must never be null, and a sender must not reuse an id while its earlier request still waits for an answer. Second, every message travels alone. JSON-RPC also allows batches, several messages sent as one array, and MCP added them in its revision of 2025-03-26, then removed them in 2025-06-18. Third, MCP claims part of the error range that JSON-RPC leaves to implementations. Codes $-32020$ to $-32099$ belong to the MCP specification. Codes $-32000$ to $-32019$ are left to the codes that older implementations already used, and new code should not use them.

The Model Context Protocol

The Model Context Protocol is an open protocol that connects language model applications to the tools and data they need. Anthropic released it in November 2024 (Anthropic 2024), and within a year the major assistants and coding tools supported it. On December 9, 2025, Anthropic gave it to the newly formed Agentic AI Foundation under the Linux Foundation. Block's goose agent and OpenAI's AGENTS.md instruction format went in alongside it (Linux Foundation 2025). The announcement counted more than 10,000 published MCP servers.

Hosts, clients, and servers

Picture a coding assistant connected to GitHub and to a company database. The assistant is the host. Inside it run two clients, one per connection. GitHub and the database each sit behind a server. The model never talks to a server directly. It writes a tool call, and the host decides whether to send it and through which client.

The split into roles is a security design as much as an engineering one. MCP's design principles say a server should not be able to read the whole conversation or see into other servers. The full history stays with the host. Each server receives only what its requests contain, and the host keeps the servers apart. So when we design permissions below, the host is where they live.

The three primitives answer three different questions. Tools are actions, chosen by the model inside the agent loop. Resources are context, chosen by the application, the way retrieval-augmented generation adds retrieved documents to a prompt (§7.6). Prompts are shortcuts for people. Most of this section is about tools, because tools are what make an agent act.

One tool call, message by message

Here is a complete exchange between a host's client and a weather server. MCP names each revision by its release date, and the current one is 2026-07-28. Two terms first. A tool's arguments are described by a JSON Schema, a JSON document that says which fields an object must have and what type each one takes (§24.2 builds tool calling on it). Every request also carries a _meta object with the protocol version and the client's capabilities, the optional features the client supports. After the first message we shorten it to "_meta": {...}.

Those two failures look alike, yet the specification treats them differently, and the difference is its most important rule about tools. Revision 2025-11-25 made it explicit.

The rule exists for the model's sake. A tool execution error is a message to the model, written so the model can fix its call, like the error observations of the agent loop in §24.1. The specification asks clients to pass these to the model. A protocol error is a message to the client software. It usually means a bug in the host, a server whose tool list changed, or a model that invented a tool name.

So if a server reports a schema violation as a protocol error, many hosts show the model a generic failure or nothing, and the model cannot recover. The check itself is the same JSON Schema validation that §24.2 runs on the model's side. The server repeats it because a server must never trust its inputs. This split is the first thing an interviewer checks when you design a tool server.

The widget below plays this exchange. Step through the messages. Then switch the revision to the older handshake, the transport to HTTP, and the ending to a bad argument or an unknown tool.

From a handshake to stateless requests

Up to revision 2025-11-25, an MCP connection began with a handshake, a short exchange that sets up the connection before any real work. It had three messages.

1. The client sent initialize with the protocol version it wanted, its capabilities, and its name.
2. The server answered with the version it would use, its own capabilities, and its name.
3. The client confirmed with the notification notifications/initialized, which by the JSON-RPC rule gets no reply.

Only then could normal requests flow. Over HTTP the server could also hand out a session id that every later request had to carry. The widget's "2025-11-25, handshake" setting shows this sequence.

The handshake made the server remember things about each connection, and that memory is expensive at scale. A busy remote server runs as several identical copies, called replicas, behind a load balancer that spreads requests across them. If a replica must remember a connection's handshake, the load balancer needs sticky sessions, which send every request of a connection to the same replica. The alternative is that the replicas share their memory. Revision 2026-07-28 removed the memory instead, along with the session id. A protocol whose server needs nothing from earlier requests is called stateless.

In plain words, each request now carries everything the server needs to answer it, so any replica can answer any request. State that really must last across calls, such as a shopping cart or a database transaction, becomes an explicit handle. A tool returns an opaque id such as bsk_a1b2c3, and the model passes it back as an ordinary argument on later calls. A tool that receives an unknown or expired handle should say so as a tool execution error, so the model can create a new one.

Statelessness changes one more thing. A server can no longer send its own request to the client in the middle of a call, for example to ask the user a question. Instead, a tool that needs the user's input returns a result of type "input_required" that lists what it needs. The client gathers the answers and sends the original request again, with a new id and the answers attached. The specification calls this the multi round-trip request pattern.

The interview point is the trade. Statelessness makes a server easy to scale and to put behind ordinary web infrastructure. In exchange, every request is larger, and any state a task needs must be named explicitly in the tool's arguments.

Transports

JSON-RPC says what a message is. A transport says how its bytes move from one program to the other. MCP defines two.

The Origin rule closes a real attack. Browsers send an Origin header that names the website a request came from. A server listening on the developer's own machine (localhost) feels private, but any web page the developer visits can make the browser send requests to localhost. With DNS rebinding, an attacker's domain name is made to resolve to 127.0.0.1, so the browser treats the local server as part of the attacker's site. Without the Origin check and a localhost-only listener, any website could call the tools of an MCP server running on a developer's laptop.

The mirrored headers serve a different purpose. A gateway, a proxy that sits in front of many servers, can then route, rate-limit, or block a call by method and tool name without parsing the JSON body. The server rejects any request whose headers and body disagree, with error $-32020$. So an attacker cannot show the gateway one tool name and the server another.

Python cells in this book run in your browser with no network. So the cell below builds a request and frames it exactly as a client would for each transport, without sending it anywhere.

The argument contained a newline, yet the stdio frame is still one line, because JSON writes a newline inside a string as the two characters \n. That is why one-message-per-line framing is safe for JSON.

Writing a server

In practice you write servers with an SDK, a library that turns Python type hints into schemas and handles the protocol for you. Here is a complete server with the official Python SDK (version 2, current in October 2026). Running the file speaks MCP over stdio. Raising ToolError inside a tool becomes a result with isError true, the tool execution error defined above, whose text is your message after the prefix "Error executing tool get_weather". Any other exception also becomes an isError result, but the SDK replaces its message with a generic "Error executing tool get_weather", so a crash never leaks internal details to the client.

Interviews ask you to write the part the SDK hides, and that is the next exercise. It is the core of a real server. Every rule in it comes from the specification, and the tests check the mistakes real servers make.

def make_server(tools):
    def handle(line):
        msg = json.loads(line)                     # placeholder: answers everything with an empty result
        return json.dumps({"jsonrpc": "2.0", "id": msg.get("id"), "result": {}})
    return handle

Notice the small decision in rule 6. Sorting makes tools/list return the same bytes every time, and that matters for cost. The host puts the tool definitions at the very start of the prompt. A prompt cache, which we price later in this section, lets the model provider skip recomputing a prompt's beginning when it has seen exactly that beginning before, and it charges much less for those tokens. A server that returns its tools in a different order on each call therefore defeats caching for every host that uses it. Revision 2026-07-28 says servers should return tools in a deterministic order for this reason.

Agents talking to agents

MCP connects an agent to tools, which are functions with schemas that return quickly. Sometimes the thing on the other side is itself an agent. Your assistant may ask a travel company's booking agent to plan a trip. That takes minutes, needs follow-up questions, and involves tools and reasoning the company does not want to expose.

Wrapped as an MCP tool, such an agent fits badly, for three reasons. A tool call is expected to return one result. The caller would see the agent's internals or nothing. And there is no standard way to say "I need more information" or "this will take an hour".

The Agent2Agent protocol (A2A) is built for this case. Google announced it on April 9, 2025 as a complement to MCP (Google 2025), and it is now developed under the Linux Foundation. Its version 1.0 specification organizes the interaction around a task, a unit of work with a state that both sides can see.

The two protocols are layers, not rivals. The A2A specification says so itself, and the common design is an A2A server whose agent uses MCP inside to reach its tools. The question for a designer, and the one an interviewer asks, is where the boundary of trust and control lies. The table gives the usual answers.

The fourth row is the easy one to get wrong. Inside one system, agents usually need each other's full context. Hiding it behind a protocol boundary is a common cause of the failures between agents measured in §24.7. Across organizations, hiding it is exactly what you want.

Hundreds of tools, and how to find the right one

A protocol makes connecting tools cheap, so agents end up connected to many. Every tool definition is text in the prompt, read by the model on every turn, and the definitions add up quickly. A typical definition, with its name, a few sentences of description, and the schema of its arguments, runs to several hundred tokens. Fifty of them fill tens of thousands of tokens before the user has typed a word.

Two effects are at work. The obvious one is cost and context. If definitions take a tenth of a 200,000-token context window, that tenth is gone before the task starts, and every turn pays for it again. Caching helps, as we will see, but only if the definitions never change. The less obvious effect is accuracy. Models choose worse among many similar tools, and the accuracy gains above came with fewer tools in view, not more.

The fix is the one retrieval-augmented generation uses for documents (§7.6). Index the tool definitions. For each request, retrieve the $k$ most relevant ones into the prompt, by matching words, by comparing embeddings, or by a hybrid of the two (§7.3 and §7.5). The model sees only those, plus a search tool it can call to look for more.

This puts a hard ceiling on accuracy, and the ceiling is a retrieval score. Two scores are standard (§7.5 defines both). Hit rate@$k$ is the fraction of requests for which at least one right item is among the top $k$ retrieved. Recall@$k$ averages, over requests, the fraction of each request's right items found in the top $k$.

The proof is one line. A correct call requires the right tool to be in the prompt, so the event "correct call" lies inside the event "right tool retrieved", and an event is never more likely than an event that contains it. With one right tool per request, a request's recall is 1 exactly when it is a hit and 0 otherwise, so the two scores coincide.

In plain words, the model cannot pick a tool it was never shown. If the model sees only the top 5 and hit rate@5 is 0.77, no prompt engineering lifts accuracy above 77%. If the model may also call the search tool itself, the ceiling becomes the chance that some search, its own or the first, finds the right tool. So measure tool search like a retriever before you blame the model.

Let us measure it. The catalog below has 30 tools from six servers, and each line pairs a tool with one realistic request that needs it. The requests use the words people use ("privately ping Omar", "call off the meeting"), not the words in the descriptions.

We score with lexical search, which compares a request and a tool by the words they share. Each text becomes a vector of word weights by TF-IDF, which counts how often a word appears in a text and gives rare words more weight. The score is the cosine similarity of the two vectors (§7.5). The index is built from the tool text only, as in production, where requests arrive after indexing. A request that shares no word with its own tool's text scores zero against it. We count that as a miss rather than letting a tie at zero pass as a hit.

Lexical search over terse descriptions finds the right tool in its top 5 for under half the requests, 14 of 30 (0.47). An agent built on it could be right at most that often, however good its model. The failures are vocabulary. "Repo" never matches "repository", and "meeting" never matches "calendar event".

Two fixes are standard, and both change the text the search sees, not the model. The first is better descriptions. Anthropic's guide to writing tools for agents asks for descriptions written as if for a new hire on the team, with the context an insider takes for granted spelled out. It also asks for unambiguous parameter names (user_id rather than user) and for tool names grouped by service, such as jira_search and slack_search. The second fix is retrieval by meaning, whose embeddings place "call off the meeting" near "cancel a calendar event". Here is the first fix in its cheapest form, one sentence per server in the words users say.

The note lifted hit rate@5 from 0.47 to 0.77, but 7 requests are still never found. Each of them shares no word at all with its own tool's text, and most of them only just miss. The notes say "repos", "bugs", "READMEs", "discussions", "replies", and "Booking", while the requests say "repo", "bug", "README", "discussion", "reply", and "book". An index that matches whole words treats each pair as two different words. Stemming, cutting each word to a common root before indexing, joins them. The cell below uses the crudest stemmer there is, which strips a final "s", "ing", or "ies" (turned into "y").

Real catalogs are larger and messier than this one, so the habit to take away is the measurement. Keep a labeled set of requests with their right tools. Report the hit rate@$k$ of tool search next to end-to-end accuracy, and treat a gap between them as a retrieval bug. An interviewer who asks why accuracy fell after adding tools wants to hear exactly this split.

Tools as code, skills, and pinning

Tool search still puts definitions into the prompt, and it still sends every intermediate result through the model. Presenting tools as code goes further. It uses the code agents of §24.5, which act by writing programs and running them in a sandbox, an isolated environment that limits what the program can touch.

Each MCP server becomes a directory of small code files, one per tool. The agent lists the directory and reads only the files it needs. Then it writes a program that calls the tools and filters their results before anything reaches the context. A 10,000-row spreadsheet is filtered in the sandbox, and the model sees the five rows it asked for. The 98.7% above came from a workflow that copies a meeting transcript from Google Drive into a Salesforce record. The agent read only the two tool files it needed, and the transcript went from one tool to the other inside the sandbox without passing through the model. The price is a sandbox with all the isolation requirements of §24.5, and a model that writes correct code.

Tool search and tools as code shrink the definitions of what an agent can do. Much of what fills a system prompt is something else, namely instructions for doing particular jobs well. Examples are the house style for a quarterly report, the steps of a release checklist, or how to fill a PDF form. Agent skills defer that kind of text in the same way. Anthropic introduced them on October 16, 2025 (Anthropic 2025c) and published the format as an open standard on December 18, 2025 (Agent Skills specification). A growing number of agent products read it.

A skill and an MCP tool answer different needs, and they combine. A tool adds a capability, something the agent could not otherwise do, such as querying the warehouse or creating an issue. A program behind the protocol serves it. A skill adds a procedure, how to do a job well with capabilities the agent already has. It runs in the agent's own environment, often with a code tool or a shell, and its instructions can name the MCP tools to call and their order. So a skill is to a set of tools roughly what a runbook, a team's written procedure for an operation, is to a set of commands.

The caching rule carries over. The always-loaded descriptions sit in the system prompt, ahead of the conversation, so they must not change by a single byte. Installing, removing, or editing a skill rewrites that beginning and starts the cache over. Opening a skill only appends to the messages and leaves the cached beginning alone. A skill can also carry scripts that run with the agent's permissions, so Anthropic recommends installing skills only from trusted sources and auditing any others like code you install.

Discovery creates a threat of its own. Tool descriptions are text the model reads and obeys, written by whoever controls the server. Three attacks exploit that (Invariant Labs 2025).

• Tool poisoning hides instructions in a description, which the user never sees and the model follows. The published example told the model to read a private key file and pass its contents in an innocent-looking argument.
• A rug pull is a server that behaves well when approved and changes its descriptions later.
• Shadowing is a malicious server whose descriptions instruct the model about another server's tools.

The specification's answer is that hosts should treat tool descriptions as untrusted unless the server is trusted, and must treat tool annotations that way.

Permissions and information flow

An agent acts with someone's authority. When it reads your mail or sends a message, it uses your credentials. So every tool call is a decision about your authority, made by a program that reads text from strangers.

§19.3 studied the attacks. In indirect prompt injection, an attacker hides instructions in content the agent reads, such as a web page or an email, and the model may follow them. The dangerous combination is an agent with three things at once. It has access to private data, it is exposed to untrusted content, and it has a way to communicate externally. Simon Willison named this combination the lethal trifecta. The strongest defenses are architectural, keeping untrusted text away from the decisions about actions. This part builds the controls in three layers. Tokens say who may call a server, rules decide each call, and labels track where data flows.

The classic name for the danger is the confused deputy (Hardy 1988). A deputy is a program that holds authority on someone's behalf. It is confused when a less privileged party tricks it into using that authority for the wrong purpose. An agent reading a hostile web page is a deputy with your authority taking orders from the page's author.

Tokens and scopes

The first layer decides who may call a remote server at all. OAuth is the standard by which a user lets an application act on their behalf without handing over a password. Remote MCP servers use OAuth 2.1, an IETF draft that gathers OAuth 2.0 and its later security fixes into one document. Its vocabulary is dense, so here it is in one place.

Local stdio servers skip OAuth entirely and read credentials from their environment, so the host's choice of environment variables is a permission decision too. MCP's authorization specification and its security guidance warn against three token mistakes, and each is a favorite interview probe.

• Token passthrough. An MCP server accepts an access token from the client and forwards it to a downstream API. The server then cannot tell its callers apart, the downstream API's logs show the wrong identity, and a stolen token can be laundered through the server. The specification forbids it. Servers must accept only tokens issued for themselves and must use their own, separately issued tokens downstream.
• Audience confusion. A token issued for one service is accepted by another. MCP clients must send a resource indicator naming the exact server in both the authorization request and the token request. Servers must check that the token's audience is them.
• Over-broad scopes. A client asks up front for every scope the server lists, say files:* and admin:*. If that token is stolen, it opens everything. The guidance is to start with minimal scopes and ask for more only when an operation needs them. The server answers such a call with 403 and a WWW-Authenticate challenge (error="insufficient_scope") naming the missing scopes. The client then runs the login again for them, which is called step-up authorization.

A permission engine

Scopes say what a token can reach. Inside the host, a second, finer layer decides each call. Real agent runtimes, coding agents in particular, write it as rules over the action and its target. The rules use glob patterns, the wildcards of a shell, as in *.py for every Python file in a folder. Each rule says allow, ask the user, or deny for every action its pattern matches. When several rules match one action, the strictest one wins, and when none matches, the action is denied.

The first request shows the property that makes this engine safe to share. The deny rule won although an allow rule matched too, and it would have won in any order. That holds for every request and every set of rules, and the proof needs nothing but the definition.

In plain words, many teams can contribute rules without worrying that someone's broad allow quietly overrides a security team's deny. Systems where the first matching rule wins, common in firewalls, lack this property, and their rule order becomes a source of bugs. Expect an interviewer to ask why deny-overrides is the safer design, and this proof is the answer.

Risk classes connect this engine to the autonomy levels of §24.1, which set how much an agent may do on its own. At the "approver" level, an agent works by itself and stops for a person to sign off on consequential steps. A useful classification asks two questions of each action. Can it be undone, and does it leave the system?

The approval prompt must show the arguments, not only the tool name. An approval of "send_email" means nothing if the recipient and body are hidden. The specification asks clients to show tool inputs to the user before calling the server.

def decide(rules, request, risk="low", approval_required=("high",)):
    for effect, pattern in rules:                  # placeholder: first match wins, fnmatch-style globbing
        import fnmatch
        if fnmatch.fnmatch(request, pattern):
            return effect
    return "allow"

Information-flow control

Permissions decide whether a tool may be called. They cannot see where the arguments came from. Take an email assistant that may read mail, read files, and send messages. It has all three parts of the lethal trifecta, and every single call it makes may be permitted. The danger is a particular flow, private data reaching an outside recipient because untrusted text asked for it.

Information-flow control, an idea from operating systems security, tracks flows directly. It attaches a label to every piece of data, saying how secret it is and how far it can be trusted. Every computation then carries the labels of its inputs to its output, and each tool checks the labels of its arguments before it acts.

Here is a tiny join. A value computed from a private, trusted file and a public, untrusted email is private (the larger of private and public) and untrusted (the larger of trusted and untrusted). What counts as private or untrusted is a modeling decision. An email from a stranger is untrusted, since anyone can write it. Its contents are no secret from the stranger who wrote them, so we label it public. A file from the user's disk is private and trusted.

The sink policy for an outbound message, send_message(to, body), makes two checks.

• Block the call if the recipient is untrusted, because untrusted text chose where data goes.
• Block it if the body is both private and untrusted, because untrusted text shaped a message that carries private data.

Labels need a program to flow through. In a plain agent loop, the model reads the email, and every token it writes afterwards depends on everything in its context. The only safe label for each later action is then the join over the whole context. After one untrusted read every action is untrusted, and the policy must block everything or nothing.

So a working design splits the work between two models. A privileged model writes the program from the trusted user request alone, before any untrusted data is read. An interpreter runs the program and tracks labels through every operation, while a quarantined model with no tools does the reading. This is the CaMeL system of Debenedetti et al. (2025), whose paper calls the labels capabilities.

§19.3 reported its result on AgentDojo, a benchmark of 97 agent tasks and 629 security test cases (Debenedetti et al. 2024). The design completed 77% of tasks with provable security against the benchmark's injections, against 84% for an undefended system. Its labels are richer than ours, tracking the sources of each value and the set of readers allowed to see it, but the logic is the one in the table.

The widget runs the program of the example box through a tracker, with three checkboxes.

• The first adds the hidden instruction, and with it the lines it steers.
• The second lets the email choose between two addresses the program wrote itself, for a user who asked to send the summary "to my boss, or to the CEO if it is urgent".
• The third breaks the tracker so that fmt forgets its inputs' labels. Only the injected lines call fmt, so it works only with the first box on.

The widget holds two lessons. First, the recipient check alone stops the obvious exfiltration, so the buggy tracker leaks through the other check. It lets out a body that untrusted text shaped and that carries the key, sent to the recipient the user chose. A link in that body, for instance, could be fetched by the boss's mail client and carry the key to the attacker's server.

Second, labels depend on where a value came from, never on what it says. Once the email chooses the recipient, even the user's own send of the summary is blocked, whichever address is picked. That over-blocking is the price of soundness, a guarantee that no unsafe flow ever passes. A real system turns such a block into an approval request rather than a silent failure.

The exercise below implements the tracker. It includes the case of the second checkbox, which trips most first attempts. When untrusted data decides which of two trusted values is used, the result is not trusted, because the choice itself carries information from the untrusted data. Such implicit flows through control flow are the classic hole in taint tracking, the simpler practice of marking untrusted data and following only its direct copies. An interviewer who asks "why not just taint-track the email?" wants this answer.

def labels_of(*xs):
    return "public", "trusted"                     # placeholder: forgets every label

def concat(*xs):
    return Labeled("".join(str(getattr(x, "value", x)) for x in xs))

def fmt(template, *args, **kwargs):
    raw = lambda x: getattr(x, "value", x)

The tracker is one of several architectures. Beurer-Kellner et al. (2025), a group from industry and academia that includes the CaMeL and AgentDojo authors, organize the options into six design patterns. Each gives up some flexibility to stop injected text from steering actions. The paper's advice is that they sit on top of the basics (sandboxing, least privilege, and confirmation for sensitive actions), never instead of them.

What an agent run costs

Now the second bill. §24.1 counted the tokens of an agent loop. Let the initial prompt (system prompt, tool definitions, and task) be $P_0$ tokens, and let each turn add $s$ tokens (the model's output plus the observation). Turn $t$ re-reads everything so far, $P_0 + (t-1)s$ input tokens. So $T$ turns read $TP_0 + sT(T-1)/2$ input tokens in total, a count that grows with the square of $T$. Providers charge per token, so the bill is that count times a price, reduced by prompt caching.

Prompt caching rests on a fact from §6.5. In a Transformer that reads left to right, each token's key and value vectors depend only on that token and the tokens before it. A server keeps these vectors in the KV cache. So two requests that begin with the same tokens share the KV cache of that beginning, and a server that keeps it can skip recomputing it (§20.6 shows how servers store and share such caches). Providers pass the saving on as a discounted price for cached tokens, and they charge a premium for writing tokens into the cache.

In an agent loop, each turn's prompt is the previous turn's prompt plus the new tokens, so caching fits perfectly. Turn $t$ reads from the cache everything turn $t - 1$ sent, and writes only the $s$ new tokens.

Here it is with tiny numbers. Take $P_0 = 100$, $s = 10$, three turns, a read price of 0.1 times, and a write price of 1.25 times. Count in full-price tokens.

• Turn 1 finds an empty cache and writes all 100 tokens, worth $1.25 \times 100 = 125$.
• Turn 2 reads those 100 and writes its 10 new ones, worth $0.1 \times 100 + 1.25 \times 10 = 22.5$.
• Turn 3 reads the 110 that turn 2 sent and writes 10 more, worth $11 + 12.5 = 23.5$.

The cached run costs 171 full-price tokens against 330 uncached. The first turn costs more than it would without a cache, and every later turn costs far less. The general count follows the same pattern.

In plain words, caching charges the re-read history at a tenth of the price, and it charges a quarter extra on each token the first time it appears. The history term still grows like $T^2$, so caching shrinks the constant in front and leaves the exponent alone. Only shrinking what is re-read changes the exponent. That is the subject of §25.2, where compaction and caching are designed together (it writes $c_{\text{read}} = \kappa_{\text{read}}c_{\text{in}}$ and $c_{\text{write}} = \kappa_{\text{write}}c_{\text{in}}$). "Does caching make a long agent run linear in cost?" is a common interview trap, and the answer is no.

The cell below computes the bill turn by turn, exactly as the definition says, and checks it against the closed form.

The next cell shows the shape over longer runs.

At 200 turns the loop costs $62.70 uncached and $6.97 cached, a ratio of 8.99. Both curves are parabolas. The ratio climbs toward 10 slowly, because the write premium is paid on all $n_T$ tokens of the final prompt, a term that grows linearly while the reads grow quadratically. With these numbers caching costs more than it saves only for a single call (ratio 0.80), which pays the write premium and never reads.

The widget draws both bills turn by turn. Lengthen the run, add output tokens, or lower the price of a cache read.

Three rules keep the cache warm in practice, and each one has burned real deployments.

• Keep the prefix byte-identical. A timestamp in the system prompt, tool definitions serialized in a different order, or a tool list that changes when a server reconnects all start the cache over. Since tools come first, a tool change resets everything.
• Keep the cache alive. If a tool call or a human approval takes longer than the 5-minute lifetime, the next turn writes the whole prefix again at 1.25 times instead of reading it at 0.1 times. Long-running agents use the 1-hour lifetime.
• Append, never edit. Changing an early message, to fix a typo or to drop an old tool result, changes the prefix from that point on. That is why context management must be designed with caching in mind (§25.2).

With Anthropic's API, one top-level cache_control setting moves the cache breakpoint to the end of each request automatically, and the response reports what was read and written.

def agent_cost(turns, prices):
    total = 0.0
    for t in turns:                                   # placeholder: charges only the new tokens, at full price
        total += prices["c_in"] * t["new_in"] + prices["c_out"] * t["out"]
    return total / 1e6

How long an agent run takes

Cost is a sum over tokens. Latency, the wall-clock time a user waits, is a sum over waits. Each turn waits for the model to read its input, then for it to write its output, then for the tools it called.

Caching shortens prefill as well as the bill, since a cached prefix is not recomputed. Tools are the other lever. Modern APIs let one turn request $m$ independent tool calls at once. The host can run them one after another or all at the same time. Run in parallel, the turn waits for the slowest call, not for the sum.

How slow is the slowest of several calls? To put a number on it, model each call's latency as exponential with mean $\mu$. That is the waiting time for an event whose chance of happening in the next instant never depends on how long you have already waited, the standard first model of a service time. The probability that a call takes longer than $x$ seconds is $e^{-x/\mu}$.

In plain words, each added parallel call adds less waiting than the one before, on average $\mu/m$ for the $m$-th, since $H_m - H_{m-1} = 1/m$. So the parallel wait grows like $\ln m$, while the sequential wait grows like $m$. Here is what that means for one ordinary turn.

The widget below draws one turn of $m$ such calls, run side by side and one after another. Add calls and compare how the two totals grow, then press Play to watch the two lanes finish.

Real latencies have heavier tails than the exponential, which makes the slowest call matter even more. Dean and Barroso (2013) called this effect the tail at scale. A simulation confirms the formula. It also reports the 99th percentile (p99), the latency that only 1 turn in 100 exceeds.

The p99 of a parallel turn grows from 9.1 seconds for one call to 13.9 seconds for ten, even though each call averages 2 seconds. The usual remedies are a timeout on each call, and answering without the slowest result when it is optional. Parallelism applies only to independent calls. Calls that depend on each other form a graph, and the time of a run is then set by its critical path, the longest chain of dependent calls, which §24.4 schedules. An interviewer who asks how to make an agent faster expects three answers in this order, namely cache the prefix, run independent calls in parallel, and cut the tail with timeouts.

Retries, idempotency, and budgets

Tools fail in transit. A request times out, an overloaded server answers 429 (too many requests) or 503 (service unavailable), or a connection resets. §24.3 handled failures the model can see and fix, such as a wrong argument, inside the loop. This part handles failures below the model, which the infrastructure should absorb without bothering it. The tool for that is the retry, sending the same request again, and retries have two classic failure modes.

The first is the thundering herd. If a thousand clients see an overload at the same moment and all retry after exactly one second, they overload the server again at exactly one second, and again at two. Exponential backoff doubles the wait after each failure, which spaces the attempts out. Jitter, a random part of the wait, breaks the synchronization. Brooker (2015) simulated 100 clients contending for one resource. Adding jitter to exponential backoff cut their total number of calls by more than half, and "full jitter", a wait drawn uniformly from zero up to the backoff, was a good default.

With a base of 0.5 seconds and a cap of 8, the first five waits are drawn from $[0, 0.5]$, $[0, 1]$, $[0, 2]$, $[0, 4]$, and $[0, 8]$ seconds, and every later wait from $[0, 8]$.

The second failure mode is worse. A timeout does not mean the request failed. It means the client did not hear back. The server may well have done the work and lost only the reply, and a retry then does the work twice. For a search, the cost is a little money. For a payment or an email, it is a visible bug.

How often does this bite? Suppose each attempt is lost before reaching the server with probability $a$, executes but loses its reply with probability $b$, and succeeds otherwise, with probability $p_{\text{ok}} = 1 - a - b$.

The simulation below draws many requests with up to $R$ attempts each and counts how often the server executes one twice or more. It compares the count with an exact formula, derived in the box.

Five attempts give a duplicate rate of 2.02%, about 20,200 per million, barely more than the 19,800 of two attempts. Almost every duplicate is one lost reply followed by one retry that reaches the server, and the second attempt already supplies that retry. The extra attempts add only paths that pair the lost reply with at least one dropped attempt, the likeliest being one drop before it or one drop after it. Each has probability about $ab$, and $2ab = 0.0004$ is the difference between the two rates to the precision shown.

What the extra attempts buy is availability, the chance that the client eventually hears success. It rises from 97% with one attempt to $1 - (a + b)^5 = 1 - 0.03^5 \approx 1 - 2.4 \times 10^{-8}$ with five. So retries without keys buy availability at a duplicate rate close to the rate of lost replies, whatever the number of attempts. With keys, the availability comes free.

The exercise builds an executor that does both. It runs on a fake clock, an object that only pretends time passes, so time is exact and the tests are deterministic.

def call_with_retries(server, clock, rng, payload, key, timeout=2.0, base=0.5, cap=8.0, max_attempts=5, deadline=30.0):
    while True:                                    # placeholder: retries forever, with no backoff and no key
        try:
            return server.call(payload, timeout)
        except Exception:
            pass

Two more mechanisms complete the picture.

• A circuit breaker stops calling a dependency that keeps failing. For an agent it also protects the budget, since a dead tool otherwise consumes retries on every turn. And it gives the model a fast, plain error ("the search service is unavailable") instead of a slow timeout.
• A rate limit caps how fast a client may call, in requests or in tokens per minute, and every model provider and most APIs impose one. The client should pace itself with a token bucket, treat 429 responses as retryable, and honor any retry-after time the server sends.

Both mechanisms are small state machines, programs whose behavior depends on a few remembered values. Both come up as coding problems in interviews for infrastructure roles.

In plain words, the bucket's capacity is the largest burst it allows, and its rate is the long-run pace. Over any window of $\Delta$ seconds the calls spend at most $\beta + \rho\Delta$ tokens, the level at the start of the window plus what drips in during it. A bucket with rate 2 per second and capacity 5 serves 5 calls at once, then one call every 0.5 seconds. The breaker turns a dependency that fails every time into one call per cooldown. Its half-open state is how it notices when the dependency comes back. (The symbols $\rho$ and $\beta$ are local to this part of the section.)

class TokenBucket:
    def __init__(self, rate, capacity):
        self.rate, self.capacity = rate, capacity
    def acquire(self, clock, n=1):
        return 0.0                                  # placeholder: never waits

class CircuitBreaker:
    def __init__(self, threshold, cooldown):

Budgets bound what one run may consume, in steps, tokens, dollars, and wall-clock time, as §24.1 required of every loop. Enforcing a dollar budget well means checking before each call, not after. The worst case of the next call is known in advance.

• Its input costs at most $\kappa_{\text{write}}c_{\text{in}}n_t$, if the cache misses and the whole prompt of $n_t$ tokens is written again.
• Its output costs at most $c_{\text{out}}$ times its maximum output tokens.

So the loop stops when the money spent plus that worst case would pass the budget. A run then never overshoots, and it stops with a recorded reason and its partial progress. For a sense of scale, take Sonnet 4.6 prices, a 100,000-token prompt, and a cap of 4,096 output tokens. The worst-case input is 1.25 times $0.30, or $0.375, and the worst-case output is $0.061, so the loop must hold back about $0.44 before it makes the call.

Seeing what an agent did

An agent that fails in production fails across dozens of model calls and tool calls, often on a machine you cannot reach, and the only evidence is what you recorded. So record every turn. A trace covers one run. Inside it, a span covers each model call, each tool call, and each sub-agent, with a link to its parent span, so the run can be replayed as a tree.

OpenTelemetry, the open standard for traces, has semantic conventions for generative AI, which now live in their own repository and are still marked as in development. They name the common pieces. The operations include chat, execute_tool, and invoke_agent. The attributes include gen_ai.request.model, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, gen_ai.usage.cache_read.input_tokens, gen_ai.tool.name, and gen_ai.tool.call.id. MCP 2026-07-28 documents the _meta keys traceparent and tracestate, so a trace can continue across the protocol into the server.

A dashboard for an agent in production should show these numbers per task type and per release.

Here pass$^k$ is the probability that an agent succeeds on all $k$ of $k$ independent tries at the same task, the reliability measure of §22.5. A canary set is a small fixed set of tasks rerun on every release. The last row is a security signal. A sudden rise in blocked sends usually means an injection campaign, and it belongs in the incident process of §19.7.

Designing a whole agent product, from requirements through tools, permissions, evaluation, and cost targets, is a standard system design interview. A good answer to an agent product design question combines the pieces of this section with two from elsewhere in the book. From this section come a protocol layer, permission and flow policies, a cost and latency budget, and dashboards. From elsewhere come the reliability arithmetic of §24.1 and the evaluation methods of §22.5.

Read more

• Model Context Protocol (2026). Specification, revision 2026-07-28, with its changelog. The authoritative source for every message in this section. Read the tools page and the security best practices, then the changelogs of 2025-11-25 and 2025-06-18 to see how the design evolved.
• JSON-RPC Working Group (2010). JSON-RPC 2.0 Specification. One page, and the source of the id, notification, and error-code rules that MCP and A2A inherit.
• A2A Project (2026). Agent2Agent (A2A) Protocol Specification. Agent Cards, tasks and their states, the three protocol bindings, and an appendix on how A2A and MCP fit together.
• Anthropic (2025). Introducing advanced tool use on the Claude Developer Platform and Code execution with MCP: Building more efficient agents. The measured costs of large tool sets and the two remedies, tool search and tools as code. Their Writing effective tools for agents is the guide to descriptions, and Equipping agents for the real world with Agent Skills explains progressive disclosure, with the open format at agentskills.io.
• Beurer-Kellner et al. (2025). Design Patterns for Securing LLM Agents against Prompt Injections. The six patterns as architectures, with ten case studies of how to apply them. For the attacks on tool descriptions themselves, read Invariant Labs (2025), MCP Security Notification: Tool Poisoning Attacks.
• Debenedetti et al. (2025). Defeating Prompt Injections by Design. CaMeL, the program-plus-labels design that the information-flow exercise simplifies, evaluated on AgentDojo.
• Brooker (2015). Exponential Backoff And Jitter. The simulation behind full jitter, short and still the standard reference. Pair it with Dean and Barroso (2013), The Tail at Scale, on why the slowest of many parallel calls dominates latency.
• Ong et al. (2024). RouteLLM: Learning to Route LLMs with Preference Data. Routers between a strong and a weak model trained on preference data, the starting point for model routing.

Watch next

▶

▶

▶

▶