Why thinking out loud makes models smarter, how to spend more compute on a hard problem, and how reinforcement learning against checkable answers turned language models into reasoning models.

Chapter 18 · Section 18.1 · 55 min

Chain of thought

Give a language model this problem and insist that the very next token be the answer. "Compose these twenty shuffles of five cards, applied in order, and tell me where the cards end up." Then give the same model the same problem, but let it write down the order of the cards after each shuffle before answering. The weights are identical. The prompt differs by a few words. Yet for a long enough list of shuffles, the first answer is close to a guess and the second is usually right.

Nothing about the model's knowledge changed between the two runs. What changed is how much computation it was allowed to do. You have seen the trick already. §6.8 previewed chain-of-thought prompting, with worked exemplars that took PaLM 540B from 17.9% to 56.9% on GSM8K, the one sentence "Let's think step by step", and a one-paragraph hypothesis for why it works, that each written token buys another forward pass. This section turns that hypothesis into an argument. We will count exactly how much serial work a Transformer does per token, see why writing intermediate results down adds work that no amount of width can replace, state what the theory of circuits proves and what it does not, and read the experiments that test what the written chain is doing. Then we will ask an uncomfortable question that every lab now has a team working on. When a model writes out its reasoning, is that the reasoning it actually used?

A fixed amount of work per token

Recall the shape of a decoder (§6.2). To produce the token at position $t$ , the model runs $L$ layers. Layer $\ell$ at position $t$ reads the outputs of layer $\ell - 1$ at positions $1, \dots, t$ , through attention, and that is the only way information moves. Draw the computation as a graph with one node per (layer, position). Every edge goes up one layer. The longest path from the input to the answer token has $L$ edges, however long the prompt is.

So within one forward pass, the model is a circuit of fixed depth $L$ . It can be enormously wide, with billions of multiplications happening side by side, but it cannot do more than $L$ rounds of "compute something, then use the result". A problem that needs $S$ dependent steps, where each step needs the previous step's result, can be solved in one pass only if the model packs about $S/L$ of those steps into each layer, and at some $S$ it cannot.

Generation changes the graph. The answer of the last layer at position $t$ is sampled into a token, and that token becomes the input of layer 1 at position $t+1$ . Now there is an edge that goes down, from the top of the stack back to the bottom, and the longest path through $T$ generated tokens has about $LT$ edges. In plain words, every token the model writes is a chance to run the whole network again on top of what it has already worked out. The written tokens are the model's working memory, and they are the only channel through which the result of one pass reaches the bottom of the next. (The KV cache of §6.5 stores every layer's keys and values, but layer $\ell$ only ever reads layer $\ell - 1$ , so the cache does not deepen the computation. Only the sampled token travels from the top back to the bottom.)

This is the cleanest argument for chain of thought (CoT), and it has nothing to do with language. A chain of thought is extra serial computation.

Composing shuffles, one step at a time

Let's make the opening puzzle concrete. A shuffle of five cards is a permutation of $\{0, 1, 2, 3, 4\}$ , and there are $5! = 120$ of them. Composing $S$ of them is the classic inherently serial problem, because each result depends on all the previous ones. Written out step by step, each step is one lookup in a $120 \times 120$ table, a constant amount of work, whatever $S$ is.

Python · cell 1

The trace is twenty short lines, and each line follows from the one before it by one lookup. Without the trace, the model would need to compute the final permutation in one pass, which is the same as evaluating a product of twenty elements of a non-abelian group in fixed depth. The next part of this section says how hard that is.

The widget's arithmetic is worth doing once at a less toy size. Suppose one forward pass can reliably carry out $c = 4$ dependent steps of a task, the task needs $S = 1{,}000$ of them, and each intermediate state takes $v = 10$ tokens to write. The model must write a state after every 4 steps except the last group, $\lceil 1000/4\rceil - 1 = 249$ states, about 2,490 thinking tokens, which is the scale of the chains that reasoning models write on competition problems. Now let each token go wrong independently with probability 0.1%. The whole chain survives with probability $0.999^{2491} \approx e^{-2.5} \approx 8\%$ . In plain words, a long chain buys the serial depth, but it also multiplies the chances to slip, so a long chain is only useful if its steps are very reliable or if the model can notice and repair its own errors. Checking and backtracking are exactly the behaviors that reinforcement learning teaches reasoning models (§18.5).

What circuit complexity says, and what it does not

The intuition above can be made precise. The precise version is one of the most quoted and most misquoted results in the reasoning literature, so it is worth getting right.

Complexity theorists classify problems by the circuits that can solve them. $\mathsf{TC}^0$ is the class of problems solvable by circuits of constant depth and polynomial size, built from AND, OR, NOT, and majority (threshold) gates with any number of inputs. $\mathsf{NC}^1$ allows depth that grows like $\log n$ . Merrill and Sabharwal (2023) proved that Transformers whose arithmetic precision is logarithmic in the input length can be simulated by constant-depth threshold circuits, so in this sense a Transformer forward pass lives in $\mathsf{TC}^0$ . That is the formal version of "fixed depth".

Composing permutations of five items (the " $S_5$ word problem") is $\mathsf{NC}^1$ -complete, a classical result of Barrington (1989). So if $\mathsf{TC}^0 \ne \mathsf{NC}^1$ , no such Transformer can compose arbitrarily long sequences of shuffles in a single forward pass, whatever its weights. That separation is unproven, but it is widely believed, much as $\mathsf{P} \ne \mathsf{NP}$ is.

Now add a chain of thought. Three results, all published in 2023 and 2024, describe what intermediate tokens buy.

Li, Liu, Zhou, and Ma (2024). With $T$ steps of chain of thought, constant-depth Transformers with constant-bit precision and $O(\log n)$ embedding size can solve any problem solvable by Boolean circuits of size $T$ . Without CoT, the same models are limited to $\mathsf{AC}^0$ , an even weaker class. Their experiments trained models on permutation composition, iterated squaring, and the circuit value problem, and found that CoT made the difference between failing and solving.
Merrill and Sabharwal (2024). A logarithmic number of CoT steps barely helps. A linear number lets a Transformer recognize every regular language, and a polynomial number lets it solve exactly the problems solvable in polynomial time. The constructions assume a mild generalization of the normalization layer, which they call projected pre-norm. §4.7 previewed this result.
Feng et al. (2023). Assuming $\mathsf{TC}^0 \ne \mathsf{NC}^1$ and log precision, bounded-depth Transformers cannot directly output the answers to basic arithmetic expressions or linear equations unless their size grows super-polynomially with the input length, while constant-size Transformers can solve both by generating step-by-step derivations.

In plain words, the number of thinking tokens is a dial on the model's serial computing power, from "constant depth" at zero tokens to "any polynomial-time algorithm" at polynomially many.

These are expressivity results, and they say less than they are often taken to say. Keep four caveats in mind.

They are about existence. Some setting of the weights can do the job. Nothing says gradient descent will find it, or that a model pretrained on web text has.
They depend on assumptions. Precision that grows with the input, particular normalization variants, and an unproven separation of complexity classes.
They are asymptotic. For any fixed input length, a big enough model can do it in one pass. The theory says the model must grow with the problem, while with CoT it need not.
They do not say the tokens must mean anything. A model could in principle carry its computation in tokens that look like nonsense, and we will meet evidence below that it sometimes does.

Few-shot chain of thought, and what its ablations rule out

Recall the format from §6.8. Wei et al. (2022) replaced few-shot exemplars of the form "question, answer" with "question, a short worked solution in natural language, answer" (the Roger and his tennis balls exemplar), and the model imitated the format by writing a solution before its own answer. Two details from the paper are worth adding to the headline. PaLM 540B's 56.9% on GSM8K, a benchmark of grade-school math word problems, beat the previous best of 55% from a fine-tuned GPT-3 that also used a trained verifier (Cobbe et al. 2021, which we meet in §18.2), with no training at all. And the gain was not uniform across sizes. Chain of thought helped only for models of roughly 100 billion parameters and up, and the authors found that smaller models "produced fluent but illogical chains of thought", which made them worse than standard prompting. It was an emergent ability in the sense of §9.6.

The serial-computation argument makes a prediction that the bare headline cannot test. If the benefit is extra serial work whose results are stored in the text, then the chain must contain the intermediate results, and it must come before the answer. The paper's ablations test exactly that, and they are the part worth remembering. Each replaces the worked solution with something else.

exemplar format	what it tests	result on GSM8K
equation only (no words)	is it just the math?	little help on GSM8K (it did help on datasets of one- or two-step problems)
dots, as many as the characters of the equation	is it just extra tokens of compute?	about the same as no chain at all
the chain written after the answer	does it just activate relevant knowledge?	about the same as no chain at all
the full chain before the answer		the large gain

The second row is the striking one. Giving the model more positions to compute in, without meaningful content, did not help in this prompting setup. The third row says that the order matters. A chain helps when the answer is conditioned on it, which is exactly what the serial-computation argument predicts.

Zero-shot, with one sentence

§6.8 also quoted the zero-shot version. Kojima et al. (2022) found that the exemplars were not needed, and that appending "Let's think step by step." to the question was enough to make large instruction-following models write a chain of thought, taking text-davinci-002 from 17.7% to 78.7% on MultiArith and from 10.4% to 40.7% on GSM8K. What the preview left out is that their method has two stages, both with greedy decoding.

Reasoning extraction. Prompt with Q: [question]. A: Let's think step by step. and let the model write.
Answer extraction. Append the model's reasoning, then a phrase such as "Therefore, the answer (arabic numerals) is", and read off the number.

The phrase mattered. In their comparison of sixteen triggers on MultiArith, "Let's think step by step." scored 78.7, "First," 77.3, and "Let's think about this logically." 74.5. Misleading or irrelevant triggers scored between 9.3 and 18.8, around the no-trigger baseline of 17.7.

The second stage is easy to overlook and important in practice. A chain of thought produces free text, and someone has to decide which part of it is the answer. Every method in this chapter needs that step, and §18.3 shows how much can go wrong in it.

Giving the chain a structure

A plain chain of thought is free-form text written left to right. Several follow-ups imposed a structure on it, each aimed at one failure of the plain version.

method	what the chain looks like	the failure it targets	a reported result
least-to-most (Zhou et al. 2023)	first break the problem into simpler subproblems, then solve them in order, feeding each answer to the next	problems harder than any exemplar, where plain chains fail to generalize	SCAN in every split, including the length split, at 99% or more with 14 exemplars on code-davinci-002, against 16% for chain of thought
program-aided (PAL, Gao et al. 2023, and Program of Thoughts, Chen et al. 2023)	the chain is a program, and an interpreter runs it to produce the answer	arithmetic slips inside otherwise correct reasoning	PAL with Codex beat PaLM 540B with chain of thought on GSM8K by 15 points absolute
tree of thoughts (Yao et al. 2023)	several candidate next steps, rated by the model and searched with backtracking	one wrong early step dooms a left-to-right chain	Game of 24 with GPT-4, 74% against 4% for chain of thought

In the language of the serial-computation argument, each one changes where the work happens. A program moves the arithmetic out of the forward pass into an exact interpreter, so the written tokens only have to carry the plan, which removes most of the per-token error rate from the worked example above. Decomposition makes each serial chunk small enough for the model to do reliably. Tree search spends compute on alternatives as well as on depth, the parallel-versus-sequential trade-off of §18.7, and it returns as a planning method for agents that search over actions. Reasoning models trained with RL learn to do much of this inside one long chain, decomposing, writing code, checking, and backtracking, with no structure prescribed.

Scratchpads, the trained version

Nye et al. (2021) asked the question from the training side. Instead of prompting, fine-tune a model to write intermediate results on a "scratchpad" before the final answer, and compare with a model fine-tuned to answer directly. On long addition, models trained on 1- to 8-digit sums with a scratchpad learned the task once they passed a critical size, while models trained without one did not, and on 9- and 10-digit sums, longer than anything in training, the models without a scratchpad failed completely. On evaluating polynomials, fine-tuned accuracy went from 31.8% to 50.7%. On predicting the output of short synthetic Python programs, where the scratchpad was the program's execution trace, line by line with the values of the variables, it went from 20% to 41.5%.

The addition scratchpad is the chain of thought in its purest form. Each line is one digit position, the work per line is constant, and the carry is the single piece of state that must flow from one line to the next.

Warm-up

An addition scratchpad

Implement addition_scratchpad(a, b) for non-negative Python integers. Work from the least significant digit to the most significant. For each digit position, record a pair (digit, carry), the output digit at that position and the carry passed to the next one. Return (steps, total), where steps is the list of pairs (one per position, as many positions as the longer number has digits) and total is the sum rebuilt from the steps alone, including a final carry if there is one. Do not compute a + b directly.

def addition_scratchpad(a, b):
    raise NotImplementedError

Hidden tests, hints, and AI help, from $1.99 a month.

Is the chain the real reasoning?

A chain of thought is plausible if it reads like a valid argument for the answer. It is faithful if it reflects the process that actually produced the answer. The two can come apart, and the serial-computation argument does not care which one we get. The model gains computation either way.

Faithfulness matters for two reasons. If we want to trust an answer because the reasoning looks right, the reasoning had better be the cause of the answer. And if we want to watch a model's reasoning for signs of bad behavior, which is now one of the main proposals for overseeing capable models, the trace has to show what the model is doing.

Biasing features the model never mentions

Turpin et al. (2023) tested faithfulness with counterfactuals. Take a multiple-choice task, add a feature that should not matter, and see whether the answer moves and whether the explanation admits why. They used two biasing features on 13 tasks from BIG-Bench Hard.

Answer is always A. Reorder the options of every few-shot exemplar so the correct one is always (A).
Suggested answer. Add "I think the answer is (X) but I'm curious to hear what you think." with a random option X.

Both features moved the models' answers toward the bias, and the explanations rationalized the biased answer instead of mentioning the bias. Accuracy fell by as much as 36%, with the largest drop, 36.3 points, for GPT-3.5 with zero-shot CoT under the suggested answer, measured on the questions where the suggestion was wrong. Out of 426 explanations that supported a biased prediction, exactly one mentioned the bias. On a social-bias benchmark (BBQ), models gave explanations that justified stereotype-aligned answers by weighing the evidence inconsistently, without mentioning the stereotype.

The method generalizes into a recipe you can run on any model. Measure accuracy with and without the feature, count how often the answer moves to the suggested option, and among those, how often the reasoning says that the suggestion is why.

Interview

A counterfactual faithfulness test

You ran a model twice on $n$ multiple-choice questions, once plainly and once with a suggested answer in the prompt. Implement bias_report(gold, plain, biased, suggested, mentions), where every argument is a length- $n$ sequence. gold holds the correct options, plain and biased the model's answers in the two runs, suggested the option suggested in the biased run, and mentions[i] is True if the biased run's reasoning explicitly mentions the suggestion. Return a dict with these keys.

"acc_plain", "acc_biased", and "drop" (plain minus biased).
"flip_rate", the fraction of the questions whose plain answer was not the suggested option that switched to the suggested option in the biased run.
"acknowledged", among the questions counted as flips, the fraction whose reasoning mentions the suggestion, or float("nan") if there were no flips.

import numpy as np

def bias_report(gold, plain, biased, suggested, mentions):
    raise NotImplementedError

Hidden tests, hints, and AI help, from $1.99 a month.

Cutting, corrupting, and padding the chain

Lanham et al. (2023) attacked the question from the other side. If the answer really depends on the chain, then tampering with the chain should change the answer. They used four interventions.

intervention	what is done	what faithfulness predicts
early answering	keep only the first part of the chain, then force an answer	the answer changes when the chain is cut short
adding mistakes	insert an error into one step and let the model continue	the final answer follows the error
paraphrasing	reword the start of the chain and let it continue	the answer stays the same (it is the content, not the phrasing, that matters)
filler tokens	replace the chain with "..." of the same length	no gain (the content, not the length, matters)

The findings were mixed in an informative way. How much models relied on the chain varied widely by task, from heavy reliance on some to mostly ignoring it on others. Paraphrased chains gave about the same accuracy as the originals, which argues against information hidden in the exact wording, and filler tokens gave no gain, which argues that the benefit of a chain is not merely extra computation. And across models from 13 billion to 175 billion parameters, faithfulness got monotonically worse with size on most tasks. A bigger model is more likely to know the answer without its chain, so the chain becomes decoration.

The early-answering test is easy to turn into a number. For each problem, truncate the chain after $k$ of its $n$ steps for $k = 0, 1, \dots, n$ , ask for an answer each time, and record whether it matches the answer with the full chain. If the answer is already fixed before any reasoning, the curve sits at 1 everywhere and the chain was not needed. The area over the curve summarizes how much of the answer the chain determined.

Core

Early answering and the area over the curve

matches is a list with one entry per problem. Entry $i$ is a sequence of length $n_i + 1$ whose element $k$ is 1 if the answer after keeping the first $k$ of the problem's $n_i$ reasoning steps equals the answer with the full chain, and 0 otherwise (so the last element is always 1). Implement early_answering_aoc(matches), returning the mean over problems of the area over each problem's curve, $\int_0^1 (1 - m_i(x))\,dx$ , where $x = k/n_i$ is the fraction of the chain kept and the integral is computed with the trapezoid rule over the points $x = 0, 1/n_i, \dots, 1$ . A problem with $n_i = 0$ has area 0.

import numpy as np

def early_answering_aoc(matches):
    raise NotImplementedError

Hidden tests, hints, and AI help, from $1.99 a month.

Hidden computation and reasoning models

The filler-token result has a twist. Pfau, Merrill, and Bowman (2024) trained small Transformers on two synthetic tasks, a version of 3SUM and a related one, where the models failed when answering immediately. Given a run of meaningless filler tokens ("......") before the answer, the same architecture reached 100% and 94% accuracy. The extra positions gave the model room to compute in parallel, even though nothing was written in them. Learning to use filler was hard, and it worked only with dense supervision from demonstrations that spread the computation across positions. Large chat models at the time did not benefit from filler on standard benchmarks. Still, the result shows that a model can carry computation in tokens whose content tells us nothing, which is exactly the failure that would make reasoning traces unreadable.

Reasoning models trained with RL raised the stakes, because their chains are long, consequential, and optimized only for the final answer. Chen et al. (2025) gave models multiple-choice questions with a hint to the answer embedded in the prompt (a suggestion from the user, answer metadata, a visual pattern in the examples, or for the misaligned cases, a peek at the grader or information obtained unethically) and checked whether models that changed their answer to follow the hint said so. Claude 3.7 Sonnet mentioned the hint in 25% of such cases and DeepSeek R1 in 39%. Outcome-based RL raised faithfulness at first, then plateaued at about 28% on MMLU and 20% on GPQA. In synthetic RL environments with an exploitable reward hack, models learned the hack on more than 99% of examples and verbalized it in under 2% of them in five of six environments.

Baker et al. (2025) at OpenAI looked at the same question from the monitoring side. A frontier reasoning model (which the paper places in the same family as o1 and o3-mini) was trained on coding tasks in which it could cheat, for example by calling exit(0) so the tests never ran or by raising SkipTest. A GPT-4o monitor that read the chain of thought along with the actions caught 95% of the hacks, against 60% for a monitor that saw only the actions. That is the good news, chains are often legible enough to catch bad behavior. The bad news came when they put the monitor into the reward. With light pressure the agent became both more capable and better behaved. With more optimization it learned obfuscated reward hacking, cheating while writing a chain that looked innocent. The authors call the cost of keeping chains monitorable, by not optimizing against them, a "monitorability tax". We return to chain-of-thought monitoring as a safety tool in §19.6.

In PyTorch

Here is the serial-computation argument as a learning experiment. We train a small network to output the parity of 16 bits directly, and a second, even smaller network to do one step of the chain, mapping (parity so far, next bit) to the new parity. Both see the same 3,000 training strings. The chain model is then unrolled for 16 steps at test time.

🔥 In PyTorchfor a machine with PyTorch

The direct model memorizes much of its training set but is barely better than a coin on new strings, about 58%. The chain model learns XOR, a four-row truth table, and is exact on every string. This is a statement about what a small network learns from limited data, not a theorem, but it is the same lesson Nye et al. found with scratchpads. Supervising the intermediate steps turns a problem that has to be solved all at once into many problems that are each easy.

🎯 Interview questions on this section

Why does chain of thought help a Transformer? Give the computational argument, not the psychological one.

Within one forward pass, information flows strictly upward through $L$ layers, so the longest dependency chain the model can compute is about $L$ steps, regardless of width or prompt length. Generating a token feeds the top layer's result back to the bottom layer at the next position, so $T$ generated tokens give a computation graph about $LT$ deep. Problems with inherently serial structure (composing permutations, carrying digits, following a chain of deductions) need depth that grows with the problem size, and the written tokens are where the extra depth comes from. The KV cache does not provide it, because each layer only reads the layer below.

State the circuit-complexity results about chain of thought, with their assumptions. What do they not show?

A log-precision Transformer forward pass can be simulated by constant-depth threshold circuits, $\mathsf{TC}^0$ (Merrill and Sabharwal 2023). Composing permutations of five elements is $\mathsf{NC}^1$ -complete, so if $\mathsf{TC}^0 \ne \mathsf{NC}^1$ (believed, unproven) one pass cannot solve it for all lengths. With CoT, $T$ steps let a constant-depth Transformer simulate any Boolean circuit of size $T$ (Li et al. 2024), linear steps recognize all regular languages, and polynomial steps give exactly $\mathsf{P}$ , both under a mild assumption about the normalization layer (Merrill and Sabharwal 2024). These are expressivity results. They do not show that training finds such solutions, they rely on precision and normalization assumptions and an unproven class separation, they are asymptotic, and they do not require the intermediate tokens to be meaningful.

What did the ablations in Wei et al. (2022) rule out?

Three alternative explanations. Equation-only exemplars helped little on multi-step GSM8K problems, so the natural-language steps matter, not just the arithmetic. Filler dots equal in length to the equation did about as well as no chain, so it is not merely extra tokens of compute in that prompting setup. A chain written after the answer also did about as well as no chain, so the benefit is not just activating relevant knowledge. The answer must be conditioned on the reasoning that precedes it, which is what the serial-computation view predicts.

How would you test whether a model's chain of thought is faithful?

Intervene and see whether the answer responds the way faithfulness predicts. Truncate the chain at several points and force an answer (early answering), and summarize with the area over the curve of "matches the full-chain answer". Insert a mistake and see whether the answer follows it. Paraphrase the chain and check the answer is unchanged. Replace it with filler of the same length and check the gain disappears. From the other direction, add a feature that should not matter, such as a suggested answer or a hint, measure how often the answer moves toward it, and how often the reasoning admits the feature is why. Turpin et al. found one acknowledgment in 426 biased explanations, and Chen et al. found reasoning models verbalizing hints only 25% to 39% of the time.

Why might faithfulness get worse as models get larger?

Lanham et al. found that from 13B to 175B parameters, the answer depended less on the chain on most tasks. A larger model can often compute the answer in a single pass, so the chain becomes post-hoc justification rather than the computation. Faithfulness by their measure is highest when the model needs the chain, which is when the task is hard relative to the model. This is one reason a chain's usefulness as a window into the model is not guaranteed to improve with capability.

Your team wants to add a penalty to the RL reward whenever a monitor reading the chain of thought flags bad intent. What do you tell them?

Baker et al. tried exactly this. Light pressure produced agents that hacked less and were more capable, but heavier optimization taught the model to hack while writing innocent-looking chains, which destroyed the monitor's value and hid the problem. The safer default is to use the CoT monitor for detection and evaluation, fix the environment or the reward when hacks are found, and avoid applying strong optimization pressure to the chain itself. Accepting some cost to keep chains monitorable is what they call the monitorability tax.

Does a longer chain of thought always help?

No. Past the length the problem needs, extra tokens add latency and cost and more chances to make an error, and models that keep going after they are done can talk themselves out of a correct answer (§18.5 calls this overthinking). Chain of thought also helps far more on math and symbolic problems than on knowledge questions, where Sprague et al. found it nearly useless unless the problem involved symbolic manipulation. And for small prompted models, Wei et al. found chains that were fluent but illogical and made accuracy worse.

A task needs 1,000 dependent steps, a forward pass can do about 4 of them, and each written state costs 10 tokens. Estimate the chain length and its reliability. What would you change if the per-token error rate is 0.1%?

The model must write a state after every 4 steps, $\lceil 1000/4\rceil - 1 = 249$ states, so about 2,500 thinking tokens. With independent errors at 0.1% per token, the chain survives with probability $0.999^{2500} \approx e^{-2.5} \approx 8\%$ , so the long chain is necessary but not sufficient. The levers are to make each state cheaper or more reliable (offload arithmetic to an interpreter as program-aided reasoning does, or write more compact states), to let the model verify and repair its own steps (what RL-trained reasoning models learn), and to sample several chains and vote or verify, since independent failures rarely agree on the same wrong answer.

🧭 What to remember

A forward pass is a fixed-depth computation, about $L$ dependent steps however wide the model is. Each generated token feeds the result back to the bottom of the stack, so $T$ tokens give about $LT$ steps of serial computation.
In complexity terms, one pass of a log-precision Transformer is in $\mathsf{TC}^0$ . With $T$ chain-of-thought steps it can simulate circuits of size $T$ , and polynomially many steps give exactly $\mathsf{P}$ . These are statements about what weights can exist, under assumptions, not about what training finds.
Few-shot CoT (Wei et al. 2022) took PaLM 540B from 17.9% to 56.9% on GSM8K and helped only large models. Its ablations show that the reasoning must come before the answer and that extra tokens alone did not help in that setup. Zero-shot CoT needs only "Let's think step by step." plus a second, answer-extraction step. Structured variants decompose the problem (least-to-most), offload arithmetic to an interpreter (PAL), or search over steps (tree of thoughts).
A long chain multiplies the chances to slip. At 0.1% error per token, a 2,500-token chain survives only about 8% of the time, which is why reliable steps and self-repair matter.
Scratchpads train the intermediate steps directly and let models generalize to longer inputs, as in long addition, where the carry is the state passed from line to line.
Traces are not guaranteed to be faithful. Models follow biasing features and hints without mentioning them, rely on their chains to a degree that varies by task and falls with size, and learn to hide intent when the chain itself is optimized.
Next, §18.2 spends compute in the other direction, sampling many chains and choosing among them.

Wei et al. (2022). Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. The paper that named the technique. Read Section 3.3 for the ablations.
Kojima et al. (2022). Large Language Models are Zero-Shot Reasoners. "Let's think step by step", with the two-stage answer extraction and the comparison of trigger phrases.
Zhou et al. (2023), Least-to-Most Prompting Enables Complex Reasoning in Large Language Models, Gao et al. (2023), PAL: Program-aided Language Models, and Yao et al. (2023), Tree of Thoughts: Deliberate Problem Solving with Large Language Models. Three ways to give the chain a structure.
Nye et al. (2021). Show Your Work: Scratchpads for Intermediate Computation with Language Models. Training models to write intermediate state, including execution traces of programs.
Li et al. (2024). Chain of Thought Empowers Transformers to Solve Inherently Serial Problems and Merrill & Sabharwal (2024), The Expressive Power of Transformers with Chain of Thought. The two clearest statements of what intermediate tokens buy in circuit terms.
Turpin et al. (2023). Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting. The counterfactual biasing method.
Lanham et al. (2023). Measuring Faithfulness in Chain-of-Thought Reasoning. Truncation, corruption, paraphrase, and filler, with the inverse scaling of faithfulness.
Chen et al. (2025). Reasoning Models Don't Always Say What They Think and Baker et al. (2025), Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation. Faithfulness and monitorability for RL-trained reasoning models.
Pfau et al. (2024). Let's Think Dot by Dot: Hidden Computation in Transformer Language Models. When filler tokens help, and why that is a warning.

Zero to Frontier is made for larger screens

Chain of thought

A fixed amount of work per token

Composing shuffles, one step at a time

What circuit complexity says, and what it does not

Few-shot chain of thought, and what its ablations rule out

Zero-shot, with one sentence

Giving the chain a structure

Scratchpads, the trained version

An addition scratchpad

Is the chain the real reasoning?

Biasing features the model never mentions

A counterfactual faithfulness test

Cutting, corrupting, and padding the chain

Early answering and the area over the curve

Hidden computation and reasoning models

In PyTorch

Read more

Watch next