Chapter 7 · Section 7.6 · 60 min

Retrieval-augmented generation

The GPT you trained in Chapter 6 read eight of Shakespeare's sonnets, and that is everything it knows. Ask it for your company's refund window and it will continue the text in something like iambic pentameter. Train the same architecture on trillions of tokens of web text, as Chapter 8 does, and the model knows a great deal, but the problem only changes shape. Its knowledge stops the day training stops. It has never read anything private. And it can't tell what it doesn't know, so it continues the text fluently anyway, with a plausible refund window or a citation that looks perfect.

Lewis et al. (2020) called knowledge stored in the weights parametric memory. It's broad, but it's frozen, hard to update, impossible to audit, and blind to private data. The fix is a non-parametric memory that the model reads at answer time. Retrieve the passages that are relevant to the input, condition the model on them, and generate. That's retrieval-augmented generation (RAG), and in practice it's the most common way language models are connected to real data.

Everything in this chapter so far was preparation. The encoder (§7.1), the contrastive training (§7.2), the embedding model (§7.3), the index (§7.4), and the hybrid retrieval and reranking (§7.5) are the R. This section adds the G in the order the field found it. First a reader that copies the answer out of a passage, then a generator trained jointly with its retriever, then a frozen base model that reads retrieved text in its context, which is the kind of model you have now. Deployed assistants mostly use instruction-tuned chat models, which Chapter 14 builds, and we preview how they change the prompt. Along the way come the parts that decide whether a RAG system works at all, which are chunking, prompt construction, failure modes, and evaluation.

Retrieve, then read

The oldest version of the idea needs no generation. Open-domain question answering gives a question with no passage attached, and the answer must come from a large corpus such as Wikipedia. DrQA (Chen et al., 2017) split the job into two stages that you can now build entirely from this chapter. A retriever (in DrQA, TF-IDF over words and word bigrams, §7.5) fetches a handful of articles, and a reader, an extractive QA model like the span extractor of §7.1, picks the answer span out of them. DPR (Karpukhin et al., 2020) replaced the lexical retriever with the dense bi-encoder of §7.2. On Natural Questions, the fraction of questions with an answer-bearing passage in the top 20 rose from 59.1 percent with BM25 to 78.4 percent, and with a BERT-base reader the whole system reached 41.5 percent exact match.

Now compare the purely parametric alternative. Roberts, Raffel and Shazeer (2020) fine-tuned T5 to answer the same questions from its weights alone, closed book. The 11-billion-parameter model reached 34.5 percent exact match on Natural Questions (36.6 with extra pretraining aimed at entities), below a retrieve-then-read system whose reader has 110 million parameters. A small model that can look things up beat a model 100 times larger that had to remember. That gap, and the fact that you fix a retrieval system by updating an index rather than retraining weights, is the whole case for retrieval.

Retrieve-then-read has two limits. The reader can only copy a span, so it can't combine facts from two passages or answer in its own words. And the stages are trained separately, so the retriever never learns which passages the reader actually found useful. A generator fixes the first, and treating the retrieved passage as a latent variable fixes the second.

The RAG model, as a probabilistic model

The original RAG paper (Lewis et al., 2020) gives the cleanest way to think about it. Let $x$ be the input (a question) and $y$ the output (an answer of tokens $y_1, \dots, y_T$ ). A retriever $p_\eta(z \mid x)$ assigns probabilities to documents $z$ from the collection. In the paper it was DPR (§7.2), with $p_\eta(z \mid x) \propto \exp(\mathbf{d}(z)^\top\mathbf{q}(x))$ for a document encoder $\mathbf{d}$ and query encoder $\mathbf{q}$ . A generator $p_\theta(y_i \mid x, z, y_{\lt i})$ writes the answer conditioned on the question and one document. The retrieved document is a latent variable, and there are two ways to marginalize it over the top $K$ documents.

RAG-Sequence uses the same document for the whole answer,

p_{\text{seq}}(y \mid x) \approx \sum_{z \in \text{top-}K} p_\eta(z \mid x) \prod_{i=1}^{T} p_\theta(y_i \mid x, z, y_{\lt i}) .

RAG-Token can switch documents at every token,

p_{\text{tok}}(y \mid x) \approx \prod_{i=1}^{T} \sum_{z \in \text{top-}K} p_\eta(z \mid x)\, p_\theta(y_i \mid x, z, y_{\lt i}) .

In plain words, RAG-Sequence asks "which document explains this whole answer?", and RAG-Token asks, word by word, "which documents explain this next word?", so one answer can combine facts from several documents.

The paper trained this end to end by minimizing $-\log p(y \mid x)$ , fine-tuning the query encoder and the generator and keeping the document encoder frozen, because updating it would mean re-embedding and re-indexing all 21 million Wikipedia passages. The generator was BART-large (400M parameters), an encoder-decoder pretrained to reconstruct corrupted text (§8.1), which reads the question and one passage concatenated. It set the state of the art on three open-domain QA benchmarks, with 44.5 percent exact match on Natural Questions for RAG-Sequence against DPR's 41.5 and closed-book T5-11B's 34.5, and its generations were judged more specific, diverse, and factual than a BART model without retrieval.

Training the retriever through the marginal is a lovely piece of math, worth knowing for interviews. Take RAG-Sequence for one example, and write $\ell_z$ for the retriever's logit for document $z$ , so $p_\eta(z \mid x) = \operatorname{softmax}(\boldsymbol{\ell})_z$ , and $g_z = p_\theta(y \mid x, z)$ for the generator's likelihood of the answer given document $z$ . Then

\frac{\partial\,(-\log p_{\text{seq}}(y \mid x))}{\partial \ell_z} = p_\eta(z \mid x) - p(z \mid x, y), \qquad p(z \mid x, y) = \frac{p_\eta(z \mid x)\, g_z}{\sum_{z'} p_\eta(z' \mid x)\, g_{z'}} .

∂ The retriever's gradient is prior minus posterior

Write $\mathcal{L} = -\log \sum_z \pi_z g_z$ with $\pi = \operatorname{softmax}(\boldsymbol{\ell})$ . Since $\partial \pi_{z'}/\partial \ell_z = \pi_{z'}(\mathbb{1}[z = z'] - \pi_z)$ ,

\frac{\partial \mathcal{L}}{\partial \ell_z} = -\frac{\sum_{z'} g_{z'}\,\pi_{z'}(\mathbb{1}[z=z'] - \pi_z)}{\sum_{z'}\pi_{z'} g_{z'}} = -\frac{\pi_z g_z}{\sum_{z'}\pi_{z'}g_{z'}} + \pi_z = \pi_z - p(z \mid x, y).

The fraction is Bayes' rule, the posterior probability that document $z$ was "the one used", given that the answer was $y$ .

In plain words, gradient descent raises the retriever's score for documents that make the correct answer more likely than the retriever currently believes, and lowers it for the rest. No one labels which document is relevant. The generator's likelihood does it.

Core

RAG-Sequence and RAG-Token marginals

Implement rag_marginals(log_prior, log_tok). log_prior is (K,), the log retriever probabilities $\log p_\eta(z \mid x)$ of the top $K$ documents (already normalized). log_tok is (K, T), where log_tok[z, i] is $\log p_\theta(y_i \mid x, z, y_{\lt i})$ . Return (log_p_seq, log_p_tok), the log-likelihoods of the answer under RAG-Sequence and RAG-Token. Work in log space with a stable log-sum-exp, since products of many token probabilities underflow.

import numpy as np

def rag_marginals(log_prior, log_tok):
    raise NotImplementedError

Hidden tests, hints, and AI help, from $1.99 a month.

The hand example in the tests shows the difference. Two equally likely documents, one supporting the first answer token and one the second. RAG-Sequence needs a single document to explain both tokens and gives the answer probability 0.09. RAG-Token combines them and gives 0.25. Answers that synthesize facts from several sources favor the token-level model.

📜 From REALM to today's RAG

REALM (Guu et al., 2020) pretrained a masked language model with a retriever in the loop, backpropagating into the retriever and periodically re-indexing. RAG (Lewis et al., 2020) brought retrieval to generation. Fusion-in-Decoder (Izacard & Grave, 2021) encoded each retrieved passage separately and let the decoder attend to all of them, which scaled to 100 passages. RETRO (Borgeaud et al., 2022) retrieved chunks from a 2-trillion-token database during pretraining, with cross-attention to neighbors. Today's most common setup is simpler than all of these, and it's the one the next section builds. A frozen model gets retrieved passages pasted into its context, with no joint training at all. The probabilistic picture above is still the right mental model of what it's doing.

Retrieval with a frozen base model

Joint training is elegant, but it needs gradients through the generator and a retriever you can afford to update. Often you have neither, only a pretrained model and an index. Two results show that this is enough, and both work with exactly the kind of model you built in Chapter 6, a base model that continues text.

Put the passages in the context. In-Context RALM (Ram et al., 2023) leaves the language model untouched and prepends retrieved text to its input. While the model reads or writes a long text, every few tokens it takes the most recent tokens as a BM25 query, fetches the best passage, and puts it in front. Nothing is trained. Retrieving every 4 tokens with the last 32 tokens as the query, the perplexity gains across GPT-2, GPT-Neo, OPT, and LLaMA models were equivalent to making the model 2 to 3 times larger, and OPT-6.7B with retrieval matched OPT-66B without it on WikiText-103. It works because of what a base model already does. Attention can copy from anywhere in the context (the induction heads of §4.7), so a passage that contains the coming words makes them cheap to predict. Notice also that BM25 beat the off-the-shelf dense retrievers the authors tried for this task, a reminder from §7.5 that the best retriever depends on the job.

Retrieve the next token directly. kNN-LM (Khandelwal et al., 2020) goes one level lower. Run the trained model once over a corpus, and for every position store a key, a hidden state $\mathbf{h}$ from the last block there (in the paper, the 1,024-dimensional input to the final feed-forward layer), and a value, the token that actually came next. At test time, compute the hidden state $\mathbf{h}(x)$ of the current context $x$ , find its $k$ nearest keys $\mathcal{N}$ with the vector search of §7.4, and turn their values into a distribution over the next token $w$ ,

p_{\text{kNN}}(w \mid x) \propto \sum_{(\mathbf{k}_i, v_i) \in \mathcal{N}} \mathbb{1}[v_i = w]\, e^{-\lVert \mathbf{k}_i - \mathbf{h}(x)\rVert^2}, \qquad p(w \mid x) = \lambda\, p_{\text{kNN}}(w \mid x) + (1 - \lambda)\, p_{\text{LM}}(w \mid x),

where $\lambda \in [0, 1]$ is tuned on validation data. In plain words, contexts that look alike to the model vote for what came after them, closer ones louder, and the vote is blended with the model's own prediction. On WikiText-103, with all 103 million training tokens in the datastore, $k = 1024$ , and $\lambda = 0.25$ , test perplexity fell from 18.65 to 15.79 with no training at all. A datastore from another domain works as well. A model trained on Wikipedia scored 34.84 perplexity on books, and adding a datastore of books brought it to 20.47 (with $\lambda = 0.65$ , since the model's own predictions deserve less trust out of domain). The gains concentrate on rare patterns such as names and facts, exactly where parametric memory is weakest.

Interview

The kNN-LM next-token distribution

Implement knn_lm(p_lm, dists, values, lam). p_lm is the model's next-token distribution, shape (V,). dists (k,) holds the squared distances from the current hidden state to its k retrieved keys, and values (k,) the integer next tokens stored with them. Return the (V,) interpolated distribution $\lambda\, p_{\text{kNN}} + (1 - \lambda)\, p_{\text{LM}}$ , with $p_{\text{kNN}}$ a softmax of the negative distances whose mass is summed per token. Tokens never retrieved get no kNN mass. No Python loops.

import numpy as np

def knn_lm(p_lm, dists, values, lam):
    p_knn = np.zeros_like(p_lm)
    p_knn[values] = np.exp(dists) / np.exp(dists).sum()
    return lam * p_knn + (1 - lam) * p_lm

Hidden tests, hints, and AI help, from $1.99 a month.

Prompting a base model with passages

To use In-Context RALM for answering questions rather than lowering perplexity, write the prompt as a document whose natural continuation is the answer, with a solved example first, the few-shot recipe of §6.6 and §6.8.

Passage [1]: The Aster kettle is made by Halden Home, a company based in Oslo.
Passage [2]: Halden Home was founded in 2009 by Nora Lind.
Question: Who founded the company that makes the Aster kettle?
Answer: Nora Lind [2]

Passage [1]: Refunds are available for 30 days. Digital goods are not refundable once downloaded.
Passage [2]: Refunds go back to the original payment method.
Question: Can I get my money back for an e-book I bought last week?
Answer:

The example teaches the format, including the citation in brackets, and generation stops at the end of the line (the stop sequences of §6.4). When the answer is one of a few options, you don't even need to sample. Score each candidate continuation, " No" or " Yes", by its log-likelihood under the model, as in the multiple-choice evaluation of §6.6, and do it once per retrieved passage, weighted by the retriever's probabilities, to get the RAG-Sequence marginal above. A base model follows such formats unreliably, though, and it declines to answer only as far as its examples show it how, which is a large part of why deployed systems use instruction-tuned models.

The pipeline you'll actually build

A production RAG system is two pipelines that meet at the index.

INGESTION (offline, whenever documents change)
  sources -> parse and clean (HTML, PDF, tables) -> deduplicate -> chunk
          -> embed each chunk (dense)  +  index terms (BM25)
          -> store vectors, text, and metadata (source, title, date, access permissions)

QUERY (online, per request)
  question -> (optional) rewrite or expand the query            (Section 7.7)
           -> hybrid retrieval, top 50 to 200, filtered by permissions   (Sections 7.4, 7.5)
           -> rerank with a cross-encoder, keep the best few     (Section 7.5)
           -> fit passages into a token budget and build the prompt with numbered sources
           -> generate with instructions to cite and to abstain
           -> check citations, return the answer with links to sources

Every arrow is a place where quality is won or lost, and the most underrated arrows are the first ones. Parsing a PDF into clean text with its tables intact, or dropping the navigation boilerplate from web pages, often matters more than the choice of embedding model. Let's look at the next one, chunking.

Chunking

You almost never retrieve whole documents. Embedding models have a maximum input length (often 512 tokens, sometimes 8K), and a single vector for a 40-page manual is a blur of every topic in it. Even if you could retrieve whole documents, pasting them into the prompt would waste the context budget and, as we'll see, hurt the model's ability to find the answer. So documents are split into chunks, and each chunk becomes one retrievable unit.

Chunk size is a real trade-off.

	small chunks (a sentence or two)	large chunks (a page)
embedding	sharp, about one thing	blurred over several topics
retrieval precision	high, if the question matches one fact	lower
context in each chunk	often missing (a "this" with no antecedent)	rich
tokens per retrieved result	few	many
typical failure	the answer is split across chunks	the relevant sentence is buried

Common strategies, from simplest to most elaborate, are these.

Fixed-size windows of a set number of tokens, with some overlap between neighbors so a fact near a boundary appears whole in at least one chunk. Simple, and it cuts sentences in half.
Sentence-aware packing. Split into sentences, then pack whole sentences up to the size limit, carrying the last sentences of each chunk into the next as overlap. This is the default worth starting with, and it's the exercise below.
Structure-aware splitting. Respect headings, paragraphs, list items, code blocks, and table rows, and prepend the section path ("Billing > Refunds > Digital goods") to each chunk. For documentation and contracts, this is usually the biggest single improvement.
Semantic chunking. Embed consecutive sentences and start a new chunk where the similarity between neighbors drops, a topic boundary.
Contextual chunks. Before indexing, have an instruction-tuned model write a sentence or two situating each chunk in its document, and prepend it. Anthropic reported that this "contextual retrieval" cut the rate at which the top 20 chunks missed the answer by 35 percent with embeddings alone, by 49 percent when also applied to BM25, and by 67 percent with reranking added (from a 5.7 percent failure rate down to 1.9 percent).
Late chunking (Günther et al., 2024). Run a long-context embedding model over the whole document once, then pool its token embeddings separately over each chunk's span, so every chunk's vector has seen the surrounding text.

Interview

A sentence-aware chunker with overlap

Implement chunk_text(text, max_tokens, overlap), counting tokens as whitespace-separated words. The provided split_sentences(text) returns a list of sentences.

Split into sentences. Any sentence longer than max_tokens words is first cut into consecutive pieces of max_tokens words (the last piece may be shorter), and each piece is treated as a sentence.
Build chunks greedily. Starting at sentence start, add sentences while the chunk's total stays at most max_tokens. Call the first sentence that did not fit end (exclusive).
Emit the chunk as its sentences joined by single spaces. If end is the number of sentences, stop.
The next chunk starts at the smallest index j with start < j <= end such that the sentences j .. end-1 total at most overlap words. This carries as much overlap as fits, in whole sentences, and always makes progress.

Return the list of chunk strings.

def chunk_text(text, max_tokens, overlap):
    words = text.split()
    return [" ".join(words[i:i + max_tokens]) for i in range(0, len(words), max_tokens)]

Hidden tests, hints, and AI help, from $1.99 a month.

Building the prompt

Once you have a few good passages, how you present them matters. With a base model, the prompt is a document to continue, as above. Most deployed systems instead use an instruction-tuned chat model, one fine-tuned to follow written instructions, which is what Chapter 14 builds. Such a model can be told what to do directly, and a solid default template for it looks like this.

You answer questions using only the sources below. Cite the sources you use with
their numbers, like [2]. If the sources do not contain the answer, say you don't know.

[1] Refund policy (help center, updated 2025-03-02)
Refunds are available for 30 days. Digital goods are not refundable once downloaded. ...

[2] Payment methods (help center, updated 2024-11-18)
Refunds go back to the original payment method. Bank transfers can take up to five ...

Question: Can I get my money back for an e-book I bought last week?

Each line earns its place, and each has a counterpart in the base-model format (the solved example shows the citation style, and a second example with the answer "Not in the passages" teaches abstention, less reliably). Only the sources reduces answers from parametric memory. Numbered citations make answers checkable, and you can verify after generation that every cited number exists and that the cited passage actually supports the sentence. Permission to abstain is one of the most effective defenses against confident nonsense when retrieval fails. Titles and dates let the model resolve conflicts between sources and prefer recent ones. The question comes last, after the sources, which is what model providers recommend for long prompts, and it keeps the fixed instructions at the front, where a server can reuse their KV cache across requests (previewed in §7.7). And the token budget is a hard constraint. Every passage costs tokens, latency, and money, and more passages are not always better, which brings us to the failure modes.

Core

Fit passages to a budget and put the best at the edges

Implement select_and_order(passages, budget). passages is a list of (doc_id, n_tokens) in rank order (best first). Walk the list in rank order and keep each passage whose tokens still fit in the remaining budget (skip ones that don't fit, and keep looking). Then order the kept passages so the strongest sit at the two ends of the prompt and the weakest in the middle, ranks 1, 3, 5, and so on from the front, and ranks 2, 4, 6, and so on from the back. Return the list of doc ids in prompt order.

For example, kept ranks [a, b, c, d, e] come out as [a, c, e, d, b].

def select_and_order(passages, budget):
    return [d for d, _ in passages]

Hidden tests, hints, and AI help, from $1.99 a month.

Why the strange ordering? Because of the next section.

Why RAG fails

When a RAG system gives a bad answer, the cause is almost always one of a short list, and the first job in debugging is to find out which. It's also a favorite interview question.

The answer was never retrieved. The relevant chunk wasn't in the top $k$ , because of a vocabulary mismatch, a bad chunk boundary, a query that needed rewriting, or an embedding model that doesn't know the domain. No generator can fix this. Measure it directly with context recall (below), and fix it with hybrid search, reranking, better chunking, or query rewriting (§7.7).
Distraction. The answer was retrieved, but so were passages that look relevant and aren't. Shi et al. (2023) showed that adding a single irrelevant sentence to grade-school math problems substantially reduced the accuracy of LLMs, and passages that are on topic but don't contain the answer tend to hurt more than unrelated ones (Cuconasu et al., 2024). Reranking and keeping fewer, better passages both help.
Lost in the middle. Liu et al. (2023) placed one relevant document among 10, 20, or 30 retrieved documents and moved it around. Accuracy formed a U-shape. It was highest when the relevant document came first or last and dropped significantly when it was in the middle, even for models built for long contexts. For GPT-3.5-Turbo with 20 documents, putting the answer in the middle gave lower accuracy than giving no documents at all (56.1 percent in the closed-book setting). That's the reason for Exercise 7.6c. Put the best passages at the edges, and don't pad the context with marginal ones. Why models behave this way is still debated, and the effect varies across models, which connects to the gap between advertised and effective context length in §12.7.
Knowledge conflicts. The passage says one thing and the model's weights say another. Models sometimes ignore the context in favor of what they memorized, and sometimes follow a wrong passage blindly. Dates, source names, and the instruction to prefer the sources help. Detecting conflicts between sources is a job for the prompt or for a dedicated check.
Unfaithful generation. The model adds details the sources don't contain, or cites a source for a claim it doesn't make. Citation checks catch some of it, and evaluation (below) measures it.
Stale or wrong sources. RAG is only as good as its corpus. Old versions of a policy, duplicates that disagree, and documents that should have been deleted all get retrieved faithfully.
Security and privacy. Retrieved text is untrusted input. A web page or an email can contain instructions ("ignore previous instructions and ...") that the model may follow, which is indirect prompt injection (§19.3). And retrieval must respect permissions, filtering by who is asking before passages reach the prompt, or the model will happily summarize documents the user was never allowed to see (§19.7).

Evaluating RAG

A RAG system has two halves, and a single "the answer looked good" score hides which half failed. So measure them separately. The useful metrics, in the vocabulary of the RAGAS framework (Es et al., 2024), are these. The last column says whether you can compute each one with labels alone, or whether it needs a judge, a language model that reads the text and decides, which in practice is an instruction-tuned model.

metric	question it answers	how it's computed	needs a judge
context recall	did retrieval find what the answer needs?	fraction of labeled gold passages present in the retrieved context (or, with a judge, of the reference answer's claims)	no, with gold passages
context precision or relevance	is the context focused?	fraction of retrieved passages, or sentences, that are relevant, often weighted by rank	no, with labels
faithfulness	is every claim in the answer supported by the context?	a judge splits the answer into short statements and checks each against the context, and the score is supported statements over all statements	yes
answer relevance	does the answer address the question?	a judge writes questions that the answer would answer, and the score is their mean cosine similarity to the real question	yes
answer correctness	is it right?	exact match or token F1 against reference answers for short answers, a judge for long ones	only for long answers

The label-only metrics are the retrieval metrics of §7.5 pointed at RAG, plus the two classic answer metrics from extractive QA. Exact match is 1 when the predicted answer equals a reference after normalization (lowercase, strip punctuation and the articles "a", "an", "the", collapse spaces). Token F1 gives partial credit. Treat the normalized prediction and the reference as bags of tokens, let the overlap be the number of tokens they share counted with multiplicity, and take the harmonic mean of precision (overlap over prediction length) and recall (overlap over reference length), the same harmonic mean as the F1 of §0.11. With several references, a prediction gets its best score against any of them. For "Nora Lind" against the reference "Lind", precision is $1/2$ , recall is 1, and F1 is $2/3$ .

Core

Exact match and token F1, the SQuAD way

Implement normalize(text), exact_match(pred, refs), and token_f1(pred, refs), where refs is a list of reference answers. Normalize by lowercasing, replacing every character that is not a letter, a digit, or whitespace with nothing, removing the whole words a, an, and the, and joining the remaining words with single spaces. Both metrics take the maximum over references. In token_f1, if the prediction or the reference normalizes to no tokens, the score is 1.0 when both are empty and 0.0 otherwise (that is how an abstention is scored against a no-answer reference).

import re
from collections import Counter

def normalize(text):
    return text.lower().strip()

def exact_match(pred, refs):
    return float(any(normalize(pred) == normalize(r) for r in refs))

Hidden tests, hints, and AI help, from $1.99 a month.

Faithfulness and answer correctness are different, and the difference matters. A faithful answer can be wrong (the source was wrong), and a correct answer can be unfaithful (the model knew it anyway). For a system whose job is to answer from your documents, faithfulness is the property you can control and should demand.

Measuring faithfulness and answer relevance takes a judge, which is fast and scalable but biased in known ways, so calibrate it against a few hundred human judgments before trusting it. How to evaluate with a model as the judge, and how to correct its biases, is treated in §14.7, and in more depth with LLM-as-a-judge methods. Two habits make RAG evaluation trustworthy.

Build an eval set from real traffic. A few hundred real questions, each labeled with the passages that answer it and a reference answer. Include unanswerable questions (the system should abstain), questions that need two passages, and questions with tempting near-miss passages.
Log the intermediate steps. For every eval question, keep what was retrieved, what was reranked, what went into the prompt, and what came out, so that when a number drops you can see which stage broke.

Core

A RAG evaluation report

Implement rag_report(records, k). Each record is a dict with the following keys.

"answerable" (bool), whether the corpus contains the answer.
"gold" (a set of passage ids that together contain the answer, empty if not answerable).
"retrieved" (a ranked list of passage ids).
"abstained" (bool), whether the system said it didn't know.
"claims" (a list of bools, one per statement in the answer, true if the statement is supported by the retrieved context, empty if the system abstained).

Return a dict with these keys.

"context_recall", over answerable records, the mean fraction of gold ids found in retrieved[:k].
"faithfulness", over records with at least one claim, the mean fraction of supported claims.
"abstain_when_unanswerable", the fraction of unanswerable records where the system abstained.
"abstain_when_answerable", the fraction of answerable records where it abstained (a false refusal rate).

Use 0.0 for any metric whose set of records is empty.

import numpy as np

def rag_report(records, k):
    raise NotImplementedError

Hidden tests, hints, and AI help, from $1.99 a month.

A whole RAG loop in a few lines

Here's the full query path on a tiny corpus, with BM25 retrieval, budgeted selection, edge ordering, and a few-shot prompt for a base model. The generator is the one piece we don't run here (our Shakespeare GPT would only answer in verse), so the cell stops at the prompt you would send.

Python · cell 1

Look at why BM25 ranked them this way, because it's a lesson in itself. The refund policy came first because it shares "for" and "downloaded" with the question, not because of "refund". Without stemming, "refunds" and "refundable" don't match "refund". The e-book passage came second on "e" and "can", and the gift card passage got in on "for" alone. The passage that answers the question did reach the prompt, but partly by luck. That's why real systems stem or lemmatize for BM25, combine it with dense retrieval, and rerank (§7.5). After that, the model still has to connect "e-book" to "digital goods" and conclude that the answer is no. That's the generator's job, and checking that it did it faithfully is the evaluator's.

In PyTorch

The RAG-Token likelihood with a retriever in the loop is a few lines, and autograd confirms that the retriever's gradient is prior minus posterior.

🔥 In PyTorchfor a machine with PyTorch

And here's the shape of a minimal production loop with a hosted, instruction-tuned model, a preview of what Chapter 14 makes possible. Any chat model API works the same way, and the call is left abstract here.

🎯 Interview questions on this section

When would you use RAG instead of fine-tuning an LLM on your documents?

RAG when the knowledge changes, must be attributable to sources, is private or permissioned, or is large. Updating means re-indexing rather than retraining, answers can cite sources, and access control happens at retrieval. Fine-tuning is better for teaching behavior, style, formats, or a domain's language, not for injecting facts reliably. The two combine well, fine-tuning the model to use retrieved context and cite it.

You only have a frozen base language model and a corpus. How can retrieval help it, with no training?

Two ways. In-context retrieval prepends retrieved passages to the input (In-Context RALM retrieved with BM25 every 4 tokens using the last 32 as the query), and the model's ability to copy from context does the rest. Ram et al. measured gains equivalent to a 2 to 3 times larger model, with OPT-6.7B matching OPT-66B in perplexity. kNN-LM instead stores (hidden state, next token) pairs for a corpus, retrieves the nearest stored states to the current one, and interpolates the neighbors' next-token distribution with the model's, $\lambda p_{\text{kNN}} + (1-\lambda)p_{\text{LM}}$ , which took WikiText-103 perplexity from 18.65 to 15.79 and adapts to a new domain by swapping the datastore. For QA, format the prompt as a few-shot document with passages, or score candidate answers by likelihood.

Closed-book or open-book QA? Give numbers.

Closed-book T5-11B, answering from its weights, got 34.5 percent exact match on Natural Questions. DPR's retrieve-then-read system with a BERT-base reader got 41.5, and RAG-Sequence with a 400M-parameter BART generator 44.5. Retrieval lets a model 25 to 100 times smaller beat one that must memorize, and its knowledge is updated by re-indexing. Closed-book answering still has its place when there is no corpus to search or latency rules out a retrieval step, and the two combine, since a generator's parametric knowledge fills small gaps in what was retrieved.

Explain RAG-Sequence versus RAG-Token, and how the retriever is trained without relevance labels.

Both treat the retrieved document as a latent variable over the top $K$ . RAG-Sequence marginalizes once over documents for the whole answer, RAG-Token at every token, so it can combine facts from several documents. Training minimizes $-\log p(y \mid x)$ , and the gradient with respect to each retriever logit is the prior minus the posterior $p(z \mid x, y) \propto p_\eta(z\mid x)\,p_\theta(y\mid x,z)$ , so documents that make the answer likely get pushed up. The document encoder is usually frozen to avoid re-indexing.

How do you choose a chunk size?

Empirically, on your own eval set. Small chunks give sharp embeddings and precise retrieval but lose context and split answers. Large chunks keep context but blur the embedding and waste prompt tokens. Start with sentence-aware chunks of a few hundred tokens with some overlap, respect document structure (headings, tables, code), add the section path or an LLM-written context line, and measure context recall and answer quality across sizes.

What is "lost in the middle", and what do you do about it?

Liu et al. found a U-shaped accuracy curve. LLMs used relevant information best when it was at the start or end of a long context and worst in the middle, sometimes below their closed-book accuracy. Mitigations are to retrieve fewer, better passages (rerank and cut), put the strongest passages at the edges, keep prompts short, and evaluate your own model, since the effect varies.

Your RAG system gives a wrong answer. How do you debug it?

Separate retrieval from generation. Check whether the needed passage was retrieved (context recall). If not, look at the query, chunking, and retriever (hybrid search, rewriting, reranking). If it was retrieved but the answer is wrong, check where it sat in the prompt, whether distracting passages were present, whether the answer is faithful to the context, and whether the source itself is wrong or stale. Logged intermediate steps make this fast.

How would you evaluate a RAG system?

Build a labeled set from real queries with gold passages and reference answers, including unanswerable and multi-hop questions. Measure retrieval with context recall and recall@k or nDCG, and generation with faithfulness, answer relevance, and correctness, using LLM judges calibrated against human labels. Track abstention on unanswerable questions and false refusals on answerable ones, and keep the eval in CI as a regression suite.

What security problems does RAG introduce?

Retrieved text is untrusted, so documents can carry indirect prompt injections that the model may follow. Treat retrieved content as data, isolate it in the prompt, limit what actions the model can take, and filter suspicious content. Retrieval must also enforce document permissions before anything reaches the prompt, or the system leaks private data to users who couldn't otherwise see it.

🧭 What to remember

RAG gives a language model a non-parametric memory. A retrieve-then-read system with a 110M reader beat closed-book T5-11B on open-domain QA, and the knowledge updates by re-indexing.
A frozen base model already benefits. Prepending retrieved passages (In-Context RALM) was worth a 2 to 3 times larger model, and kNN-LM interpolates next-token votes from a datastore of hidden states.
In the probabilistic view, the document is a latent variable. RAG-Sequence and RAG-Token marginalize it differently, and the retriever's gradient is prior minus posterior.
Chunking is a precision versus context trade-off. Start with sentence-aware chunks with overlap, respect structure, and consider contextual chunks.
Prompts should restrict answers to sources, number them for citations, allow abstention, and fit a token budget with the best passages at the edges.
RAG fails through retrieval misses, distraction, lost in the middle, knowledge conflicts, unfaithful generation, bad sources, and injection or permission leaks. Debug retrieval and generation separately.
Evaluate with context recall, exact match and token F1, abstention rates, and, with a calibrated judge, faithfulness and answer relevance, on a labeled set from real traffic. Next, §7.7 covers the techniques that fix the hardest retrieval failures.

Lewis et al. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. RAG-Sequence, RAG-Token, and end-to-end retriever training.
Liu et al. (2023). Lost in the Middle: How Language Models Use Long Contexts. The U-shaped curve, published in TACL in 2024.
Es et al. (2024). Ragas: Automated Evaluation of Retrieval Augmented Generation. Faithfulness, answer relevance, and context relevance, reference-free.
Chen et al. (2017). Reading Wikipedia to Answer Open-Domain Questions (DrQA). The retriever and reader split that open-domain QA still uses.
Ram et al. (2023). In-Context Retrieval-Augmented Language Models. Retrieval for a frozen language model by prepending passages, with the 2 to 3 times model-size equivalence.
Khandelwal et al. (2020). Generalization through Memorization: Nearest Neighbor Language Models (kNN-LM). Next-token retrieval from a datastore of hidden states.
Izacard & Grave (2021). Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering (Fusion-in-Decoder). Reading many passages at once.
Anthropic (2024). Introducing Contextual Retrieval. Contextual chunks for embeddings and BM25, with measured gains.

Zero to Frontier is made for larger screens

Retrieval-augmented generation

Retrieve, then read

The RAG model, as a probabilistic model

RAG-Sequence and RAG-Token marginals

Retrieval with a frozen base model

The kNN-LM next-token distribution

Prompting a base model with passages

The pipeline you'll actually build

Chunking

A sentence-aware chunker with overlap

Building the prompt

Fit passages to a budget and put the best at the edges

Why RAG fails

Evaluating RAG

Exact match and token F1, the SQuAD way

A RAG evaluation report

A whole RAG loop in a few lines

In PyTorch

Read more

Watch next