Ask a chat model for the birthday of someone moderately obscure, a junior researcher or a minor historical figure. You will often get a date, stated with the same calm fluency as the boiling point of water. Ask again in a fresh chat and you may get a different date, stated just as calmly. Kalai et al. (2025) did exactly this with one of their own birthdays, instructing an open model to answer only if it knew. In three attempts it produced three different dates, all wrong.

Why would a system trained on trillions of tokens of mostly true text make things up, and why would it sound so sure? The tempting answer is that it is a bug that more data or better training will remove. This section argues that the truth is more interesting. Some hallucination is a statistical necessity for any model that is calibrated and has seen most facts only rarely, and much of the rest is something we train in, because the way we grade models rewards guessing over admitting ignorance. Once you see both mechanisms, the fixes stop looking like a bag of tricks. You measure how confident the model should be, you make it say so, you let it abstain, you reward it for abstaining when it should, and you check its claims against sources.

We build from a tiny model that must guess about facts it saw once, to the math of calibration, to measurements on real models.

What counts as a hallucination

The word is used loosely, so let's be precise. A model's output can be wrong relative to two different references. It can be unfaithful to a source it was given (the document it was asked to summarize, the passages retrieved for it, the user's instructions), or it can be false about the world. These are different properties. A summary that faithfully repeats an error in its source document is faithful but not factual. A summary that adds a true detail the document never mentioned is factual but not faithful. §7.6 made the same distinction for retrieval-augmented answers, and argued that for a system meant to answer from your documents, faithfulness is the property you can control.

Maynez et al. (2020), studying abstractive summarization, split faithfulness errors further. An intrinsic hallucination misrepresents information that is in the source (the article says a politician was a mayoral candidate, the summary calls him a former mayor). An extrinsic hallucination adds information that cannot be inferred from the source at all. In their human study, more than 70% of single-sentence summaries of news articles contained a hallucination, most hallucinations were extrinsic, and over 90% of the extrinsic ones were also false. One cause was the training data. The reference summaries in the dataset themselves often contained facts that were not in the article, so models were trained to add them.

For chat models, Huang et al. (2023) proposed a taxonomy that most later work follows. Factuality hallucinations conflict with the world (a wrong entity or relation, or a fabricated claim that cannot be verified anywhere). Faithfulness hallucinations conflict with the input (ignoring the instruction, contradicting the provided context, or contradicting itself).

There is a second axis that matters more for fixing things, the mechanism. Farquhar et al. (2024) single out confabulations, claims that are wrong and arbitrary, meaning the answer changes with irrelevant details such as the random seed. Asked for the target of the cancer drug sotorasib, a model may say KRAS G12C (correct) on one sample and KRAS G12D on the next. Confabulations are different from errors the model makes consistently, which usually come from false or conflicting training data (a common misconception repeated across the web), from optimizing a reward that favors pleasing answers over true ones (the sycophancy of §19.6), or from failures of reasoning. Each mechanism has its own signature and its own remedy, and lumping them together under one word is why "fixing hallucination" sounds harder than it is.

A model that must guess about facts it saw once

Here is the smallest world in which hallucination is unavoidable. There are 20,000 people, each with a birthday. A training corpus mentions people in proportion to how famous they are, a Zipf distribution in which person $i$ is mentioned with probability $p_i \propto 1/i$, so a few people appear constantly and most appear rarely or never. Our model is the best possible student of this corpus. It memorizes every birthday it has seen, perfectly. Then we ask it questions, also about people in proportion to their fame. When it has seen the person, it recalls the right date. When it has not, it has nothing to go on, and if it must answer, it guesses one of 365 days.

How often will it be wrong? Exactly as often as it is asked about someone it never saw, minus the lucky guesses. The interesting question is whether the corpus itself can tell us how often that is, without knowing the true popularity of anyone.

Three columns, nearly identical in every row. The error rate of a model that always answers is the unseen mass, the total probability of the people it never saw (times $364/365$ for the lucky guesses). And the unseen mass is predicted, to within a percentage point, by the singleton rate, the fraction of the corpus taken up by people mentioned exactly once. With 2,000 mentions the model is wrong on about 40% of questions. With 200,000 mentions, a hundred times more data, it is still wrong on about 2%, because the long tail never ends.

Why the singleton rate predicts the unseen mass

This is the Good-Turing estimator, which Alan Turing and I. J. Good developed during their wartime codebreaking and Good published in 1953. It is worth deriving, because it is the heart of the theory that follows.

Hallucinate or be miscalibrated

The toy model hallucinated because we forced it to answer. Kalai and Vempala (2024) showed that a language model is pushed toward the same behavior even without being forced, as long as it is calibrated, meaning that the probabilities it assigns to kinds of statements match how often those statements occur. Think about what calibration demands in the birthday world. In the true distribution of text, statements about people the model never saw make up a fraction $m_0$ of everything written. A calibrated generator, asked to write a sentence about someone, must therefore put total probability about $m_0$ on people it never saw, and for those it can only invent a birthday, true one time in 365. Its hallucination rate is about $m_0 \approx N_1/N$. The only way to put less mass on invented facts is to put more on the facts it knows, and then its probabilities no longer match the world. It must hallucinate or be miscalibrated.

That is the intuition behind their theorem. For arbitrary facts, ones whose truth follows no pattern a model could learn (birthdays, the exact title of an obscure paper), and when no single fact has a large probability, a calibrated model's rate of generating false facts is close to the fraction of facts that appear exactly once in its training data. This holds even if every training sentence is true. It is not a statement about bad data or weak architectures. It is a statement about the long tail. The same paper argues that there is no such statistical pressure for facts that appear many times or that follow systematic rules, such as arithmetic, so errors there have other causes.

Kalai, Nachum, Vempala, and Zhang (2025) sharpened the argument with a reduction to classification. Suppose that for each prompt there is a set $\mathcal{V}$ of valid responses and a set $\mathcal{E}$ of errors. Generating a valid response is at least as hard as answering the yes-or-no question "is this response valid?", which they call Is-It-Valid. Their main theorem, stated here without prompts, is

where $\operatorname{err}$ is the probability that the model generates an error, $\operatorname{err}_{\text{iiv}}$ is the misclassification rate of the classifier that calls a response valid when the model's probability for it exceeds $1/|\mathcal{E}|$, and $\delta$ measures the model's miscalibration at that one threshold. In plain words, if validity is hard to tell apart from error, generation must make errors, unless the model is miscalibrated. For arbitrary facts they prove that, with high probability, the error rate is at least the singleton rate minus terms of order $2/|\mathcal{E}|$ and $(\log N)/\sqrt N$ and minus $\delta$, and that a calibrated model can achieve an error rate close to that bound, so the singleton rate is the right answer, not a loose bound. They also note that $\delta$ is the size of the derivative of the cross-entropy loss with respect to a rescaling of the model's probabilities, which vanishes at any minimum of the loss over a model family that can represent that rescaling. Pretraining with cross-entropy tends to produce calibration, and with it, hallucination on the tail.

Why post-training does not remove it

Pretraining explains why a base model's distribution contains hallucinations. It does not explain why a chat model, which could in principle say "I don't know" whenever it is unsure, so rarely does. Kalai et al. (2025) put the blame on how models are graded. Almost every benchmark scores an answer as right or wrong, and an abstention as wrong. Under that binary grading, consider a question on which the model's best answer is right with probability $c$. Answering earns $c$ points in expectation. Abstaining earns zero. Since $c > 0$ for any guess, always answer is optimal, however unsure the model is. A model trained or selected to score well on such benchmarks learns to bluff, the way a student facing a multiple-choice exam with no penalty for wrong answers learns to never leave a question blank.

The fix they propose is to state the penalty. Suppose a wrong answer costs $k$ points, a right one earns 1, and "I don't know" earns 0. Answering has expected score

which is positive exactly when $c > \frac{k}{1+k}$. Choose $k = \frac{t}{1-t}$ and the break-even confidence is $\frac{t/(1-t)}{1/(1-t)} = t$. In plain words, the instruction "answer only if you are more than $t$ confident, since mistakes cost $t/(1-t)$ points" makes a calibrated model's best policy to answer exactly when its confidence exceeds $t$. A threshold of 0.5 means a wrong answer costs 1 point, 0.75 costs 3, and 0.9 costs 9. Let's check the policy numerically for a model whose confidences are calibrated.

With no penalty the best threshold is zero and the model answers everything. With a penalty the best threshold lands on $k/(1+k)$, and the model answers only the questions it is likely to get right. Kalai et al. surveyed ten influential benchmarks and found that nine use binary grading and give no credit at all for abstaining, while the tenth gives only partial credit. Adding one more "hallucination benchmark" does not change incentives that the main leaderboards set. Changing how the main leaderboards score answers does.

Even metrics designed with abstention in mind can leak. SimpleQA (Wei et al. 2024), which we meet properly below, grades each answer correct, incorrect, or not attempted, and summarizes with an F-score, the harmonic mean of the overall fraction correct and the fraction correct among attempted questions. With $n_c$ correct, $n_i$ incorrect, and $n_a$ not attempted, that harmonic mean works out to

The SimpleQA authors point out, crediting Kalai, that this score can be gamed, since a model gains by guessing whenever it is at least 50% sure, and they recommend a fixed penalty for wrong answers instead. The exact threshold is lower than that, and since the paper does not derive it, here is the calculation, our own. Should a model guess on one more question that it would otherwise skip, if it is right with probability $c$? Guessing moves one question out of $n_a$ into $n_c$ or $n_i$, so the denominator grows by 1 either way and the numerator grows by 2 with probability $c$. The expected score after guessing is $\frac{2n_c + 2c}{2n_c + 2n_i + n_a + 1}$, an expectation over the numerator only, which is exact here because the denominator does not depend on the outcome. Cross-multiplying, this beats $F$ exactly when $2c\,(2n_c + 2n_i + n_a) > 2n_c$, that is, when $c > F/2$. Since $F \le 1$, the paper's 50% rule is a safe sufficient condition, and the real incentive is stronger. A model with an F-score of 0.38 should guess whenever it is more than 19% sure.

Published numbers show how much the grading rule matters. The GPT-5 system card (OpenAI 2025) reports, for SimpleQA without browsing, an accuracy of 22% and a hallucination rate (wrong answers) of 26% for gpt-5-thinking-mini, against 24% and 75% for o4-mini. The card does not list abstentions. The rest of the questions, 52% for gpt-5-thinking-mini and 1% for o4-mini, are the ones each model declined to answer, since the card's two rates leave exactly that remainder. By accuracy alone, o4-mini is the better model. With a wrong answer costing one point, gpt-5-thinking-mini scores $0.22 - 0.26 = -0.04$ and o4-mini scores $0.24 - 0.75 = -0.51$.

def simpleqa_report(grades, t):
    n = len(grades)
    c = grades.count("correct") / n
    return {"correct": c, "incorrect": 1 - c, "not_attempted": 0.0,
            "correct_given_attempted": c, "f_score": c, "penalized": c}

Calibration at the token level

All of this depends on the model knowing how likely it is to be right. A model is calibrated if, among all the answers to which it assigns confidence $c$, a fraction $c$ is correct,

Calibration and accuracy are different things. A model that always says 60% on questions it gets right 60% of the time is perfectly calibrated and useless for deciding which answers to trust. A model that ranks its answers perfectly but says 99% on all of them is badly calibrated and very useful. The ability to rank, to give higher confidence to the answers that are right, is called discrimination, and it is measured by the AUROC of the confidence as a predictor of correctness (§0.11). We will need both.

Why should a pretrained model be calibrated at all? Because cross-entropy is a proper scoring rule. Take a binary event that happens with probability $p$, and suppose the model predicts probability $q$. Its expected log loss is $-p\log q - (1-p)\log(1-q)$, whose derivative in $q$ is $-p/q + (1-p)/(1-q)$. Setting it to zero gives $p(1-q) = q(1-p)$, so $q = p$, and the second derivative $p/q^2 + (1-p)/(1-q)^2$ is positive, so this is the minimum. In plain words, the loss a language model is pretrained on is minimized by reporting true frequencies, so on text like its training data, its next-token probabilities are pushed toward calibration. (Section 0.11 showed the same property for the Brier score.) The same holds for any number of outcomes, which is why §0.5 could call cross-entropy minimization fitting the data distribution.

Kadavath et al. (2022) tested this on models up to 52B parameters. When questions are posed in the right format, multiple choice with lettered options, larger models are well calibrated on the probability they give to the letter of each answer, and calibration improves with scale. Format matters. Replacing one option with "none of the above" hurt both accuracy and calibration. They then asked models to evaluate their own free-form answers, with a prompt asking whether a proposed answer is true and reading off the probability of "True", which they called $\operatorname{P}(\text{True})$. This worked well and improved when the model was first shown several of its own samples for the same question. Finally, they trained models with an extra head to predict $\operatorname{P}(\text{IK})$, the probability that "I know" the answer, from the question alone. It discriminated well, generalized partly to new tasks (its calibration there was weaker), and rose appropriately when relevant source material or hints were placed in the context.

To measure calibration we bin answers by confidence and compare, in each bin, average confidence with accuracy. The expected calibration error of §0.11 weights each bin's gap by its share of the answers,

with $n$ answers and $n_b$ of them in bin $b$. Section 0.11 built it with bins of equal width (Exercise 0.11c), drew the reliability diagrams, and showed that the binned estimate is biased upward by finite samples, so an ECE means little without its number of answers and bins. One choice matters especially for language models. Their confidences crowd near 1, so equal-width bins put most answers in the last bin, where over- and under-confidence can cancel. Kadavath et al. used ten bins with an equal number of answers in each, §0.11's equal-mass bins, and so will we.

What post-training does to calibration

The GPT-4 technical report (OpenAI 2023) contains the cleanest public measurement. On a subset of MMLU, the pretrained GPT-4's confidence in its chosen answer letter was very well calibrated, with an ECE of 0.007. After post-training with RLHF and PPO, on the same questions, the ECE was 0.074, the model now systematically more confident than accurate. Kadavath et al. saw the same thing in their RLHF policies and found that a single fix largely repaired it on several evaluations, dividing the policy's logits by a temperature $\tau = 2.5$.

Why would RL sharpen the distribution like that, and why would a temperature undo it? Here is one simple way to see it. It is this book's toy model, not an account given by the GPT-4 report or by Kadavath et al., and no paper we know of has shown that real rewards take this form. The KL-regularized RL objective has the optimal policy $\pi^\star(y \mid x) \propto \pi_{\text{ref}}(y \mid x)\exp\big(r(x, y)/\beta\big)$ (derived in §17.1), with $\beta$ the KL weight. Suppose the reward, on average, favors answers the reference model already finds likely, because confident, conventional answers are what graders tend to approve, and model this as $r(x, y) = a\log\pi_{\text{ref}}(y\mid x)$ for some $a > 0$. Then

which is the reference distribution with every logit multiplied by $1 + a/\beta$, a temperature below 1. Dividing the policy's logits by $\tau = 1 + a/\beta$ recovers the calibrated reference exactly. Real post-training is not this clean, and Kadavath et al. caution that more intensive RL might distort calibration in ways a temperature cannot repair. But the toy shows how a single temperature could fix so much. Let's reproduce the whole story on multiple-choice questions whose answers are drawn from the pretrained model's own probabilities, so that it is calibrated by construction, with a post-trained model whose logits are 2.5 times sharper.

Accuracy is identical, since multiplying logits by a positive number never changes the argmax. Mean confidence jumps from 0.61 to 0.79, and the ECE from under 0.01 to 0.18. Our sharpening is harsher than GPT-4's, but the shape is the same. Now fit a temperature on half the questions by minimizing the negative log-likelihood of the true answers, and check the other half.

The fitted temperature comes out at 2.52, close to the 2.5 we put in (the difference is sampling noise in 10,000 questions), and the reliability curve of the rescaled model lies back on the diagonal. This is the temperature scaling of §0.11 (Guo et al. 2017), the standard post-hoc fix, and it costs one scalar fitted on a few thousand labeled examples. Its limits matter as much as its strength. It needs labeled data from the distribution you care about, it fixes only a uniform sharpening, and it does nothing for discrimination, because dividing all of a question's logits by the same number never changes which answer comes first.

import numpy as np

def fit_temperature(logits, y):
    return 1.0

Confidence in words

Token probabilities are only available for answers that fit in a token or two, such as a letter for a multiple-choice question. For a sentence-long answer, the probability of the exact string mixes up "is this true" with "is this how I would phrase it", and many APIs do not expose probabilities at all. The natural alternative is to ask. Verbalized confidence means prompting the model to state a number, "your answer and your confidence from 0 to 100%", or training it to do so.

Lin, Hilton, and Evans (2022) fine-tuned GPT-3 to state its confidence in words on arithmetic problems and found that the stated confidences were well calibrated and remained moderately calibrated under distribution shift. For RLHF models, whose token probabilities are often overconfident, Tian et al. (2023) found that verbalized confidences from ChatGPT, GPT-4, and Claude were typically better calibrated than the models' token probabilities on TriviaQA, SciQ, and TruthfulQA, often cutting ECE by about half in relative terms. Others are less sanguine. Xiong et al. (2024) found verbalized confidences generally overconfident, plausibly because models imitate how people talk about their own certainty, and found that combining several sampled answers and their consistency helped more than any single prompt, with no method winning everywhere.

SimpleQA measured both kinds of confidence on frontier models (Wei et al. 2024). Stated confidence correlated with accuracy, larger models were better calibrated than smaller ones (GPT-4o over GPT-4o-mini, o1-preview over o1-mini), and every model's curve sat well below the diagonal, so stated confidence consistently overstated accuracy. The second method asked each question 100 times at temperature 1 and used the frequency of the most common answer as the confidence. That was better calibrated, and for o1-preview frequency roughly matched accuracy. Consistency across samples is a signal the model cannot inflate by choosing words, which brings us to the most principled version of the idea.

Semantic entropy

If a model knows an answer, it gives the same answer every time you sample it, though not always in the same words. If it is confabulating, the answer itself changes. So sample several answers and measure how much they disagree. The naive measure is the entropy of the samples, but it counts "Paris", "It's Paris", and "The capital is Paris" as three different answers. Token-level and string-level entropy confuse uncertainty about wording, which is harmless, with uncertainty about meaning, which is what predicts error.

Semantic entropy (Kuhn et al. 2023, Farquhar et al. 2024) measures the second. Sample $M$ answers $s^{(1)}, \dots, s^{(M)}$ to the question $x$. Cluster them by meaning, putting two answers in the same cluster when each entails the other, meaning each logically implies the other in the context of the question, as judged by a natural-language-inference model or a prompted LLM. One direction is not enough. "Paris" entails nothing about France until the question supplies it, and "Paris, which has 2 million people" says more than "Paris". With clusters $C_1, \dots, C_{|\mathcal{C}|}$, estimate each meaning's probability and take the entropy,

where $p_\theta(s\mid x)$ is the model's probability of the full answer $s$. In plain words, add up the probability of the sampled answers that mean the same thing, renormalize over the meanings you saw, and measure how spread out the meanings are. When probabilities are not available, the discrete variant sets $P(C_i\mid x)$ to the fraction of the $M$ samples that fell in $C_i$. Here are both entropies on two questions, with a string rule standing in for the entailment model.

You can check these by hand. The drug question has meanings in proportions 0.6, 0.3, and 0.1, so $\operatorname{SE} = -(0.6\log 0.6 + 0.3\log 0.3 + 0.1\log 0.1) = 0.306 + 0.361 + 0.230 = 0.898$ nats. The France question has one meaning and $\operatorname{SE} = 0$, while its naive entropy, 1.030, is nearly as high as the drug question's 1.280. Naive entropy would flag the question the model knows cold almost as strongly as the one it is confabulating.

Farquhar et al. (2024) evaluated semantic entropy with $M = 10$ samples on question answering (TriviaQA, SQuAD, BioASQ, NQ-Open) and math word problems (SVAMP), with LLaMA 2 Chat at 7B, 13B, and 70B, Falcon Instruct at 7B and 40B, and Mistral Instruct 7B. As a detector of wrong answers, averaged over the 30 combinations of task and model, semantic entropy reached an AUROC of 0.790, against 0.691 for naive entropy, 0.698 for $\operatorname{P}(\text{True})$, and 0.687 for a supervised regression on the answer's embedding. The discrete variant performed about as well. For paragraph-length biographies from GPT-4, they broke each paragraph into individual factual claims, generated questions that each claim answers, and computed discrete semantic entropy over the model's answers to those questions, which detected the confabulated claims better than asking the model to check itself.

Three limits come with it. First, it detects only confabulations. A model that has learned a misconception gives the same wrong answer every time, with zero semantic entropy. Second, it is expensive, about $M$ times the generation cost plus up to $M^2/2$ entailment checks per question, and Kossen et al. (2024) report a 5-to-10-fold increase in compute. Their semantic entropy probes, linear probes trained to predict semantic entropy from the hidden states of a single generation, cut the overhead to nearly nothing while keeping much of the detection performance. Third, it depends on the entailment judge, and long answers must first be decomposed into claims.

import numpy as np

def cluster(answers, entails):
    seen = {}
    return [seen.setdefault(a, len(seen)) for a in answers]      # placeholder: exact strings only

def semantic_entropy(answers, logps, entails):
    p = np.exp(np.asarray(logps, dtype=float)); p /= p.sum()

Abstention and selective prediction

A confidence score earns its keep when it is used to decide what to answer. Selective prediction (Geifman and El-Yaniv 2017) formalizes this. A model with confidence $c_i$ on question $i$ answers when $c_i > t$ and abstains otherwise. Two numbers describe the result. The coverage is the share of questions answered, and the selective risk is the error rate among those,

where $y_i = 1$ if answer $i$ is correct. Sweeping $t$ from high to low traces the risk-coverage curve, from answering only the most confident question to answering everything. Its area, the AURC, is the mean of the selective risk over all $n$ coverage levels when questions are accepted in order of decreasing confidence. Lower is better. The best possible ordering, accepting all correct answers first, gives the oracle curve, which is zero until coverage reaches the accuracy, and the gap between a model's AURC and the oracle's (the excess AURC) isolates how well the confidence ranks. Farquhar et al.'s rejection-accuracy curve is the same curve upside down, accuracy instead of error. Geifman and El-Yaniv also showed how to choose $t$ on held-out data so that the selective risk stays below a target with high probability, guaranteeing a 2% top-5 error on ImageNet with probability 99.9% at almost 60% coverage.

Let's draw the curves for the three models of our calibration experiment, plus the oracle.

The three curves nearly coincide, with AURC within 0.002 of each other, even though their ECEs differ by more than a factor of twenty. Sharpening and temperature scaling change confidence values but barely change their order across questions, and the risk-coverage curve depends only on the order. (With two options the order would be exactly unchanged. With four, the largest probability is not quite a monotone function of a single number, so a few questions swap places.) Calibration tells you what a confidence of 0.8 means. Discrimination tells you which answers to drop. They are different properties, measured by different metrics, and fixed by different methods.

Calibration comes back the moment wrong answers cost something. Let's give our three models the instruction "answer only if you are more than 75% confident", so each wrong answer costs $0.75/0.25 = 3$ points.

The calibrated models answer about a quarter of the questions, are right on almost 90% of those, and score about $+0.15$. The overconfident model believes it clears the bar on 61% of questions, is right on only 74% of them, just under the break-even rate of 75%, and scores below zero. Same knowledge, same ranking, and the overconfident model would be better off abstaining on everything. This is the practical case for calibration. It is what makes a stated threshold mean what it says.

import numpy as np

def risk_coverage(conf, correct):
    order = np.argsort(conf)                                  # placeholder: least confident first
    k = np.arange(1, len(conf) + 1)
    return k / len(conf), np.cumsum(1 - np.asarray(correct)[order]) / k

def aurc(conf, correct):

Abstention has costs too. A model that says "I don't know" to questions it could answer is the factuality version of the over-refusal of §19.4, and it is just as annoying. So report coverage next to accuracy, and evaluate abstention on questions that are answerable as well as on ones that are not.

Measuring factuality

The measurements so far assumed we know which answers are right. For real models, building that ground truth is the hard part, and the benchmarks differ in what they check and how.

Short answers, SimpleQA and TruthfulQA

SimpleQA (Wei et al. 2024) is 4,326 short questions with single, indisputable answers that do not change over time, such as a date, a name, or a number. Each question was written by one AI trainer with a supporting web page and answered independently by a second trainer, and only questions where the two agreed were kept. Questions were also collected adversarially, kept only if at least one of four GPT-4-era model responses got them wrong. A prompted grader labels each response correct, incorrect, or not attempted. Here are some of the paper's results, in percent of all questions.

Read the rows with the grading rule in mind. GPT-4o-mini almost never abstains and is wrong on nine questions in ten. Claude-3.5-sonnet answers far fewer questions correctly than GPT-4o but abstains on a third of them, and with a one-point penalty for wrong answers it scores $28.9 - 36.1 = -7.2$ against GPT-4o's $38.2 - 60.8 = -22.6$. Every model in the table has a negative score under that penalty, and the authors note that at a penalty of 9, which requires 90% precision on attempted questions to break even, none of them would be positive. SimpleQA also shows that knowledge tracks size, with each large model ahead of its small sibling, consistent with the Good-Turing picture, since a bigger model can memorize more of the tail.

Newer models are often evaluated alongside private sets of the same kind. The o3 system card (OpenAI 2025) reported a hallucination rate on PersonQA, questions about publicly available facts about people, of 33% for o3 against 16% for o1, while o3's accuracy was also higher, 59% against 47%. The card's explanation was that o3 makes more claims overall, so it gets more right and more wrong. That is what a reward that does not charge for wrong answers would produce.

TruthfulQA (Lin, Hilton, and Evans 2022) tests something narrower, imitative falsehoods, false answers that are likely under the training distribution because people often say them. It has 817 questions in 38 categories ("What happens if you smash a mirror?"). The best model in the original paper was truthful on 58% of questions against 94% for people, and in a striking reversal of the usual scaling, larger models were generally less truthful, because they were better at reproducing common misconceptions. The GPT-4 report found its base model only slightly better than GPT-3.5 on the multiple-choice version, and large gains after RLHF. Treat it with care, for several reasons.

• It is adversarial against a particular model. 437 of the questions were kept only if GPT-3 175B answered them falsely, and the other 380 were written with that experience. Some of the inverse scaling reflects the filtering target.
• Truthful is not the same as informative. Answering "I have no comment" to every question is perfectly truthful, which is why the generation task also scores informativeness with a separate judge.
• The multiple-choice versions can be gamed. In the original MC1 format, incorrect options were sometimes paraphrases of each other, so picking the odd one out beats chance without any knowledge. The authors released a two-option version in January 2025 to remove that shortcut (Evans, Chua, and Lin 2025).
• It is old, public, and small. The GPT-4 report states that its RLHF data was not checked for contamination with TruthfulQA. A benchmark of 817 public questions is easy to train on by accident, the general problem treated with contamination detection.

Long answers, FActScore, LongFact, and SAFE

Short-answer benchmarks miss how chat models are used, which is to write paragraphs that mix many claims. One wrong detail in a biography should not make the whole biography wrong, and a paragraph with ten true claims and one false one is better than a paragraph with two true and one false. FActScore (Min et al. 2023) makes this precise. Break a response $y$ into atomic facts $\mathcal{A}_y$, short statements that each carry one piece of information, check each against a knowledge source $\mathcal{K}$, and score the fraction supported,

In plain words, FActScore is the average, over the prompts the model did not decline, of the share of its claims that check out. The original study used biographies of people with Wikipedia as $\mathcal{K}$, a setting where support is mostly unambiguous. Human annotators found FActScores of 42.5% for InstructGPT, 58.3% for ChatGPT, and 71.5% for PerplexityAI, which searches the web and still had more than a quarter of its claims unsupported. Errors were more common for rarer people and for facts later in a generation. An automated estimator that splits the response with a language model and checks each fact against retrieved Wikipedia passages matched the human scores to within 2%, cheap enough to score thousands of generations.

Notice what FActScore leaves out. It measures precision, so a model that declines often, or writes three safe sentences instead of thirty, scores well. LongFact and SAFE (Wei et al. 2024) add recall. LongFact is 2,280 prompts across 38 topics that ask for long, fact-dense answers. SAFE (Search-Augmented Factuality Evaluator) splits a response into individual facts, makes each one self-contained, checks that it is relevant to the prompt, and rates it by issuing Google searches and reasoning over the results. On about 16,000 facts it agreed with crowdsourced annotators 72% of the time, and on a random 100 of the disagreements it was right 76% of the time, at more than 20 times lower cost. With $S(y)$ supported and $N(y)$ unsupported facts (irrelevant ones are dropped) and a target number $K$ of supported facts a user would want, the metric is

and $F_1@K = 0$ when $S(y) = 0$. Across 13 models, larger models were generally more factual. The comparison of PaLM-2-L-IT with and without RLHF is a nice illustration of what recall adds. The RLHF model averaged 72.9 supported facts per response against 13.2, and its $F_1@64$ was 91.0 against 31.1, mostly because it said much more.

def factscore(responses):
    facts = [l for r in responses if r for l in r]
    return facts.count("S") / max(len(facts), 1), 1.0         # placeholder: pooled over all facts

def f1_at_k(responses, K):
    raise NotImplementedError

Automated fact-checkers are themselves models with error rates. Decomposition can split a claim wrongly or drop its context ("he won the award" without saying who), the knowledge source may not cover rare entities, and the verifier can be fooled by fluent text. When you compare two models on an automated factuality score, validate the checker on a few hundred human labels and correct for its errors as §19.4 did for safety judges.

Retrieval and citation, and how they fail

If hallucination comes from facts the model never learned, give it the facts. Retrieval-augmented generation (§7.6) puts relevant passages in the context and asks the model to answer from them and cite them. This changes the problem from factuality to faithfulness, which is easier to check, and it moves knowledge out of the weights, where it is frozen and unverifiable, into documents that can be updated and inspected. Practitioners regard it as the most effective mitigation available. It also fails in measurable ways.

Liu, Zhang, and Liang (2023) audited four commercial generative search engines. On average only 51.5% of generated sentences were fully supported by their citations, and only 74.5% of citations supported the sentence they were attached to, while the answers read as fluent and authoritative. On the ALCE benchmark (Gao et al. 2023), even the best systems lacked complete citation support 50% of the time on long explanatory questions. Legal research tools built on retrieval, some advertised as avoiding hallucination, were found to hallucinate between 17% and 33% of the time (Magesh et al. 2024). And recall that FActScore's search-based PerplexityAI was at 71.5%.

The causes are the failure modes of §7.6, seen from the hallucination side. When retrieval misses, the model falls back on its weights and writes a plausible answer with a citation to a passage that does not support it. When the retrieved passages conflict with what it memorized, it sometimes trusts the wrong one. When a passage is on topic but does not answer the question, the model often stretches it. The mitigations follow. Verify after generation that every claim is entailed by the passage it cites, using the same entailment models as semantic entropy, and drop or flag claims that are not. Instruct and train the model to abstain when the passages do not contain the answer. And evaluate faithfulness and abstention on unanswerable questions, as §7.6's evaluation recipe does.

Fixes at decoding time and at training time

Decoding. Greedy or low-temperature decoding removes the sampling variation behind some confabulations, at the price of hiding the uncertainty that semantic entropy would have measured, and it does nothing for consistent errors. A more interesting decoding fix is DoLa (Chuang et al. 2024), which contrasts layers. Project the hidden state of an early layer $j$ through the output head to get a next-token distribution $q_j$, compare it with the final layer's $q_L$, and decode from

with $\alpha = 0.1$. In plain words, boost the tokens whose probability grew most between the early layer and the last, among tokens the final layer already finds plausible. The premature layer $j$ is chosen at each step, from a bucket of candidate layers picked on validation data, as the one whose distribution differs most from the final layer's by Jensen-Shannon divergence. The intuition, supported by their analysis, is that early layers capture what is fluent and later layers inject factual knowledge, so the difference emphasizes the knowledge. DoLa raised the TruthfulQA scores of LLaMA models by 12 to 17 absolute points with no retrieval and no training. Prompting methods work in the same spirit at a coarser grain. Chain-of-verification (Dhuliawala et al. 2023) has the model draft an answer, write verification questions about its own claims, answer them independently, and revise, and SelfCheckGPT (Manakul et al. 2023) flags sentences that are not supported by other samples, a cheap cousin of semantic entropy.

Training. §14.1 showed that fine-tuning on facts the base model does not know is learned slowly and raises hallucination elsewhere (Gekhman et al. 2024). Kang et al. (2024) found the mechanism behind a related effect. When a fine-tuned model meets an unfamiliar question, its made-up answer tends to imitate the answers it was trained to give on the unfamiliar fine-tuning examples. So the way those examples are labeled controls how the model hallucinates, and labeling them with an expression of uncertainty teaches it to be uncertain on unfamiliar questions. R-Tuning (Zhang et al. 2024) builds this into the data. Run the base model on the fine-tuning questions and split them by whether its own answer matches the label. Every training target keeps the labeled answer and then appends a question, "Are you sure you accurately answered the question based on your internal knowledge?", followed by "I am sure" for questions the base model got right and "I am unsure" for those it got wrong. The model thus learns to express uncertainty exactly on the questions outside its parametric knowledge. This refusal-aware tuning answered known questions better, expressed uncertainty more often on unknown ones, and improved calibration, and the authors found that the refusal ability generalized to other tasks, a meta-skill in their words.

Preference optimization can target factuality directly. Tian et al. (2023) generated several responses per prompt, scored each automatically for factuality, either FActScore-style against retrieved Wikipedia or without any reference by using the model's own consistency across samples, and ran DPO (§17.2) on the resulting preference pairs. At 7B scale, compared with Llama-2-chat, factual error rates fell by 58% on biographies and 40% on medical questions.

For RL with verifiable rewards (§18.3), the analysis of the grading section applies word for word. A reward of 1 for a correct answer and 0 for anything else is binary grading, and its optimal policy never abstains. A reward of 1 for correct, 0 for "I don't know", and $-t/(1-t)$ for wrong makes the optimal policy answer exactly when its probability of being right exceeds $t$. The o3 PersonQA numbers and the gpt-5-thinking-mini SimpleQA numbers above look like the behavior each kind of reward predicts, though neither system card describes its reward in that detail. Kang et al. add a caution for long-form factuality RL. When the reward model itself hallucinates about unfamiliar facts, it corrupts the training signal, and controlling how the reward model handles unfamiliar inputs made RL factuality fine-tuning work better.

Interpretability is starting to show where in the network this happens. In Claude 3.5 Haiku, Lindsey et al. (2025) found a default circuit that pushes the model toward saying it cannot answer, and features that fire on known entities and inhibit that default. Hallucinations appeared when those features fired on a name the model recognized but knew little about, switching off the default without the facts to back it up, a model that knows of someone without knowing about them. Tools for reading circuits like this one are the subject of circuit analysis.

In PyTorch

In practice temperature scaling is fitted with a few steps of L-BFGS on the log-temperature, which keeps $\tau$ positive without constraints. The model's logits are computed once and cached, so fitting costs nothing compared with running the model. The block below builds a calibrated four-way classifier, sharpens its logits as post-training would, fits $\tau$ on half the data, and reports the ECE on the other half.

The fitted $\tau$ comes out near 2.4, and the held-out ECE falls from 0.18 to about 0.01 with accuracy untouched. For a deployed LLM, logits would be the model's scores for the answer options of a few thousand labeled multiple-choice questions from the target distribution, and the fitted $\tau$ would be applied when reporting confidence, not when sampling text.

The end of Part IV

Hallucination is where this chapter's threads meet. Grading that rewards guessing is a specification problem, the optimizer doing exactly what the reward asked. Factuality numbers are as easy to misread as safety numbers, and need the same intervals, the same corrected judges, and the same care about what was counted. And a model that reports its uncertainty truthfully is being honest in the plainest sense, saying what it knows and no more.

It also closes Part IV, and with it the story of how a model that predicts the next token becomes an assistant people can use. Supervised fine-tuning taught it the format and the persona (Chapter 14). Reinforcement learning gave it a way to learn from consequences (Chapter 15), preference learning taught it what people prefer (Chapters 16 and 17), and verifiable rewards taught it to reason (Chapter 18). This chapter asked what can go wrong at each step and how much we can trust the result. The honest answer is that the techniques work well enough to deploy, with layers of defense around them, and not well enough to stop worrying. The specification can be wrong, the learned goal can differ from it, evaluations are lower bounds, oversight is hardest exactly where it matters most, a calibrated model must still guess about the long tail, and every defense is eventually tested by someone trying to break it.

The questions that remain open are the field's most interesting research problems. Can we evaluate behavior we cannot check? Can we read a model's goals, and what it knows, from its weights instead of its words? Reading activations directly, where $\operatorname{P}(\text{IK})$ heads and semantic entropy probes come from, is the subject of probing, and sparse autoencoders decompose those activations into features such as the known-entity features behind some hallucinations. Models that act in the world raise every question of this chapter with higher stakes, where code agents running in sandboxes meet prompt injection and need the control protocols of §19.7, and where agent evaluations must measure long, open-ended behavior reliably.

There is also a practical question that every aligned model faces the moment it is deployed. Each safeguard in these sections, a classifier at the input, a streaming check at the output, a monitor reading the reasoning, ten samples for semantic entropy, costs compute on every request, and so does the model itself. Serving a frontier model to millions of people at an acceptable price depends on fast, cheap inference, which starts from the fact that generating each token is limited by how fast weights can be read from memory rather than by arithmetic, and on tricks such as speculative decoding, in which a small model drafts several tokens and the large model checks them all in one parallel pass. Safety and efficiency meet there, because a safeguard that is too expensive to run on every request will not be run.

Read more

• Kalai and Vempala (2024). Calibrated Language Models Must Hallucinate. The theorem linking calibration, the Good-Turing estimate, and hallucination on arbitrary facts.
• Kalai, Nachum, Vempala, and Zhang (2025). Why Language Models Hallucinate. The reduction to Is-It-Valid classification, the singleton-rate bound, and the case for confidence targets in evaluation.
• Kadavath et al. (2022). Language Models (Mostly) Know What They Know. Calibration with scale and format, P(True), P(IK), and temperature-repaired RLHF policies.
• Farquhar, Kossen, Kuhn, and Gal (2024). Detecting hallucinations in large language models using semantic entropy. Semantic entropy, bidirectional entailment clustering, and its evaluation across models and tasks.
• Wei et al. (2024). Measuring short-form factuality in large language models. SimpleQA, its grading, the F-score loophole, and calibration by stated confidence and answer frequency.
• Min et al. (2023). FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation. Atomic facts, human and automated factual precision of biographies.
• Wei et al. (2024). Long-form factuality in large language models. LongFact, the SAFE evaluator, and F1@K.
• Lin, Hilton, and Evans (2022). TruthfulQA: Measuring How Models Mimic Human Falsehoods. Imitative falsehoods and inverse scaling, worth reading with its caveats in mind.

Watch next

▶

▶

▶