In September 2024 OpenAI released o1, a model that "thinks" before it answers, and said little about how it worked beyond two plots. Accuracy rose smoothly with the compute spent on reinforcement learning, and it rose smoothly again with the compute spent thinking at test time. In January 2025 DeepSeek released R1, which matched o1 on several math and coding benchmarks, with open weights and a paper that described the recipe. The paper's most surprising result was a model that had never seen a single example of human reasoning, trained with nothing but the GRPO of §18.4 and a program that checked final answers, that learned on its own to think for thousands of tokens, to check its work, and to go back when it noticed a mistake.

This section walks through that recipe stage by stage, with the numbers. Then it separates what is publicly known about other labs' reasoning models from what is guessed, takes on the live debate about whether RL teaches models anything new, and ends with the practical problem every deployed reasoning model has, controlling how long it thinks.

What makes a model a reasoning model

A reasoning model, as the term is used now, is a language model trained, mostly by RL against verifiable rewards (§18.3), to write a long chain of thought before its answer. Three things distinguish it from a chat model asked to "think step by step" (§18.1).

• The chain is long and self-directed. Thousands or tens of thousands of tokens, containing plans, attempts, checks, and revisions that no prompt asked for.
• It is optimized for the answer. Only the final answer is rewarded, so the chain is whatever helps produce correct answers, readable or not.
• Compute at inference becomes a dial. Longer thinking gives better answers up to a point, so the model's cost per query depends on how hard it thinks, and products expose that as a setting.

The chain is usually separated from the answer by special tokens, such as <think> and </think>, and many products show users a summary of it rather than the raw text.

R1-Zero, reinforcement learning from a base model

DeepSeek-R1-Zero started from DeepSeek-V3-Base, the pretrained mixture-of-experts model with 671B total and 37B active parameters (§13.5), with no supervised fine-tuning at all. The prompt was a short template saying, in effect, that the assistant first reasons privately and then answers, with the reasoning and the answer wrapped in <think> and <answer> tags. The template deliberately said nothing about how to reason, so that whatever reasoning appeared would be the policy's own. The reward was purely rule-based, an accuracy reward from an answer checker (or compiler and tests, for code) plus a format reward for using the tags. As §18.3 noted, DeepSeek used no neural reward model because of the risk of reward hacking at this scale. The algorithm was GRPO.

On AIME 2024, R1-Zero's pass@1 rose during training from 15.6% to 71.0%, and to 86.7% with majority voting over 64 samples, in the original version of the paper. (The later version published in Nature reports a final pass@1 of 77.9% for a longer training run, with the same 86.7% under self-consistency, so quote numbers with their version.) Two things happened along the way that nobody had programmed.

The responses grew long. The average length rose steadily through training, from hundreds of tokens to thousands, with no length reward. The model was rewarded only for correct answers and found that longer thinking produced more of them (a toy model below reproduces this), which is the test-time compute effect of §18.7 discovered by the optimizer.

New behaviors appeared. Reflection (revisiting earlier steps), exploring alternative approaches, and checking intermediate results grew in frequency. The paper shows an intermediate checkpoint stopping in the middle of a solution with "Wait, wait. Wait. That's an aha moment I can flag here." and then re-deriving the step. This "aha moment" became the emblem of the paper.

R1-Zero also had problems that made it unusable as a product. Its reasoning was hard to read, and it mixed languages, drifting between English and Chinese within one chain. The full R1 recipe exists to fix those problems while keeping the reasoning.

The later published version of the paper gives the training details. For R1-Zero, the learning rate was $3\times10^{-6}$, the KL coefficient 0.001, the group size 16, and the maximum response length 32,768 tokens until step 8,200 and 65,536 after. Training ran about 10,400 steps with 32 questions per step, and the reference model was refreshed every 400 steps. The paper also reports the cost of the RL itself, at an assumed $2 per H800 GPU hour. R1-Zero took about 101K GPU hours ($202K), creating the SFT data about 5K ($10K), and R1 about 41K ($82K), for a total near $294K. That excludes pretraining the base model, which the DeepSeek-V3 report put at 2.788M H800 GPU hours, about$5.6M, for the official training run alone.

Why the chains grow

Why would an optimizer that is paid only for correct answers make its responses longer? Because on hard problems more thinking raises the chance of being right, and nothing in the reward charges for tokens. Here is the smallest model of that. Each of three prompts, easy, medium, and hard, lets the policy choose a thinking length from 250 to 8,000 tokens, and the success probability $0.95(1 - e^{-\ell/d})$ for a length $\ell$ rises faster on an easy prompt (difficulty scale $d$ = 200 tokens) than on a hard one ($d$ = 3,000). The policy starts uniform over lengths, and each step samples a group of 8 lengths per prompt and applies the group-mean REINFORCE update of Exercise 18.4d with a 0/1 reward.

The medium and hard prompts move to the longest chains, and accuracy on the hard prompt doubles, from 0.42 to 0.88, almost its best possible 0.884. That is the length growth of R1-Zero in miniature, found by the optimizer with no length reward. Look at the other two prompts, though. The easy prompt ends near 2,900 tokens and the medium one near 7,600, while 1,000 and 4,000 tokens would already give each of them 0.94. Once success is nearly certain, the reward stops distinguishing lengths and nothing pushes back, so the lengths wander upward. Outcome-only RL grows chains where length helps and lets them drift where it does not, which is the seed of the overthinking problem at the end of this section.

The DeepSeek-R1 recipe

R1 adds three stages around a second round of R1-Zero-style RL. It is worth knowing in order, because most open reasoning models since have followed a version of it.

1. Cold start. Collect thousands of long chain-of-thought examples in a readable format, with a summary at the end, and fine-tune DeepSeek-V3-Base on them. This gives RL a starting point whose reasoning is already legible, and the paper argues a better one than the raw base model.
2. Reasoning-oriented RL. Run GRPO on math, code, science, and logic prompts with rule-based rewards, plus a language consistency reward, the proportion of words in the chain of thought that are in the target language. The paper reports that this reward slightly degraded benchmark performance and kept it anyway, because readable reasoning in one language was worth more.
3. Rejection sampling and SFT. When RL nears convergence, sample many responses per prompt from the RL checkpoint, keep only correct ones (checked by rules, or for some data by DeepSeek-V3 as a judge comparing with the reference), and filter out chains with mixed languages, long paragraphs, and code blocks. That gives about 600K reasoning samples. Add about 200K non-reasoning samples (writing, factual QA, self-cognition, translation), mostly reused from DeepSeek-V3's own SFT data. Then fine-tune the base model, DeepSeek-V3-Base, for two epochs on the roughly 800K samples. Note what is being trained. Not the RL checkpoint, but a fresh copy of the base, on the RL checkpoint's best outputs, which is rejection sampling fine-tuning (§18.2) used as a distillation step.
4. RL for all scenarios. A final RL stage that combines rule-based rewards on reasoning prompts with reward models for helpfulness and harmlessness on general prompts, the RLHF of Chapter 16 running alongside RLVR.

The result scored 79.8% pass@1 on AIME 2024 and 97.3% on MATH-500, and reached a Codeforces rating of 2,029, above 96.3% of human participants.

def filter_traces(samples, reference, max_keep=4, max_par_chars=1500):
    return [s for s in samples if reference in s][:max_keep]

Distillation into small models

The same 800K samples did something else that changed the field. DeepSeek fine-tuned six open dense models on them, Qwen2.5 at 1.5B, 7B, 14B, and 32B and Llama 3 at 8B and 70B, with SFT only and no RL. DeepSeek-R1-Distill-Qwen-7B scored 55.5% on AIME 2024, and DeepSeek-R1-Distill-Qwen-32B scored 72.6% on AIME 2024 and 94.3% on MATH-500, far beyond what those base models could do with prompting.

The control experiment is the important part. DeepSeek also ran large-scale R1-Zero-style RL directly on Qwen2.5-32B-Base. That model, DeepSeek-R1-Zero-Qwen-32B, reached 47.0% on AIME 2024, against 72.6% for the distilled version of the same base. The paper's conclusion is that small models trained with RL alone may not even reach the performance of distillation, and that distilling from a stronger reasoner is both cheaper and better at this scale. One reading is that the strong model's RL discovered reasoning patterns that a small model's RL, exploring from a weaker starting point, would rarely find. Yue et al. (2025), whose skeptical results about RL appear below, found that distillation, unlike RLVR in their experiments, could give a model reasoning patterns its base did not have.

Two follow-ups showed how little data the transfer can need when the student is already strong. LIMO (Ye et al. 2025) fine-tuned Qwen2.5-32B-Instruct on 800 carefully chosen problems with high-quality reasoning traces and reported 63.3% on AIME 2024 and 95.6% on MATH-500 in the revised paper (the first version used 817 examples and reported 57.1% and 94.8%). s1 (Muennighoff et al. 2025) used 1,000 and is the subject of §18.7. Both argue that long-form reasoning is mostly elicited from knowledge that pretraining already put there. Distillation can also be done on-policy, with the student generating and the teacher scoring every token of the student's output, which is how modern distillation methods avoid training the student only on text it would never write itself.

Does RL teach new reasoning, or sharpen what is there?

This is the most interesting open question in the area, and the evidence points in both directions.

The skeptical case is Yue et al. (2025). Across several model families, benchmarks, and six RLVR algorithms, RL-trained models beat their base models at pass@1, but the base models had higher pass@$k$ at large $k$ (they used $k$ up to 256 as a proxy for "can it ever solve this"). The two curves crossed. On Minerva math with a 32B model, the base model led by about 9 points at $k = 128$. During RL, pass@1 rose while pass@256 fell. Their reading is that RLVR makes the model sample its already-reachable correct solutions more reliably, narrowing its distribution, and that the solvable set is bounded by the base model. DeepSeekMath reported the same pattern a year earlier, RL raising maj@$k$ but not pass@$k$.

The other side is ProRL (Liu et al. 2025, at NVIDIA). With much longer RL training, KL control, periodic resets of the reference policy, and a diverse set of tasks, their RL-trained model beat the base model across pass@$k$ evaluations, including on tasks where the base model failed regardless of how many samples it drew, and the improvement in the reasoning boundary grew with training duration and was largest where the base model was weakest.

Both can be true. RL with a group baseline can only reinforce what it samples, so early in training, or with a short budget, sharpening dominates. Whether it later expands the solvable set may depend on training length, exploration (entropy, clip-higher, KL), and the task mix. The widget below makes the arithmetic of the debate concrete.

The practical lesson for evaluating any RL result is to report the whole pass@$k$ curve, not just pass@1, and to compare against the base model at large $k$.

def crossover(c_base, c_rl, n, ks):
    b = np.array([np.mean(c_base) / n] * len(ks)); r = np.array([np.mean(c_rl) / n] * len(ks))
    return b, r, None

What is known about other labs' reasoning models

Most frontier reasoning models are closed, and the public record is a mix of official statements, model cards, benchmark reports, and a great deal of informed guessing. It is worth keeping the categories apart, especially in an interview.

What OpenAI has said about o1. Its announcement (OpenAI 2024) states that a large-scale RL algorithm teaches the model to think productively using its chain of thought, that performance improves smoothly with more RL (train-time compute) and with more time spent thinking (test-time compute), and that the raw chains of thought are hidden from users, a decision it said weighed user experience, competitive advantage, and the value of being able to monitor an unaltered chain. It reported 74% on AIME 2024 with a single sample per problem, 83% with consensus among 64 samples, and 93% when re-ranking 1,000 samples with a learned scoring function, an 89th-percentile Codeforces ranking, and accuracy above human PhD-level experts on GPQA. The o1 system card added safety evaluations, including an experiment in which a monitor model read o1-preview's chains of thought looking for deception.

o3. Announced in December 2024, o3 scored 75.7% on the ARC-AGI-1 semi-private evaluation within the benchmark's public compute limit and 87.5% with a configuration using 172 times more compute, according to the ARC Prize team, who noted that the o3 released in April 2025 was not the same configuration they had tested. OpenAI's open-weight gpt-oss models (August 2025) are reasoning models whose effort is set to low, medium, or high in the system prompt.

Other labs. Google released Gemini 2.0 Flash Thinking in December 2024 and made the Gemini 2.5 models thinking models in March 2025, with user-set thinking budgets arriving in 2.5 Flash. An advanced Gemini with "Deep Think", which explores several solutions in parallel before answering, achieved a gold-medal score at the 2025 International Mathematical Olympiad (35 of 42 points), working end to end in natural language within the time limit, and OpenAI announced that an experimental reasoning model also reached gold-medal-level performance on the same problems, under the same time limits and without tools, as graded by former medalists it engaged rather than by the competition's coordinators. Anthropic described Claude 3.7 Sonnet (February 2025) as the first "hybrid" reasoning model, one model that answers directly or thinks first, with the raw thinking shown to users and a thinking budget settable through the API. Qwen3 (2025) documented a four-stage pipeline, long chain-of-thought cold start, reasoning RL, thinking mode fusion (training one model to support both thinking and non-thinking modes), and general RL, along with a thinking budget. Kimi k1.5 described long-context RL and methods for making long-chain models concise.

What is speculation. The training data, reward design, RL algorithm, and model sizes of the closed models have not been published. In particular, OpenAI has not said that o1 uses Monte Carlo tree search or a process reward model at inference, and the public descriptions are of a single model generating one long chain of thought. Claims to the contrary, common in 2024, were guesses. So were early claims that o1 was trained with process supervision, a plausible guess given Lightman et al. (2023) but not one OpenAI confirmed. The best public evidence on that question is DeepSeek's. The R1 paper has a section of unsuccessful attempts, which reports that a process reward model, a learned grader of every step rather than only the final answer, and Monte Carlo tree search over reasoning steps both failed to help at R1's scale. Why grading steps is attractive, and why it is so hard to make work inside RL, is the subject of §18.6.

Long chains, overthinking, and length control

Long chains are the source of reasoning models' accuracy and also of their cost, latency, and some of their failures. Overthinking is the name for spending far more thinking than a problem needs. Chen et al. (2024) found that o1-like models used, on average, 1,953% more tokens than conventional models on very simple questions, and on "what is 2 + 3" one model produced 13 separate solutions. Their self-training approach cut the tokens used on MATH500 by 48.6% for QwQ-32B-Preview while keeping its accuracy. Overthinking is also not only waste. A model that keeps going after it has the answer can talk itself out of it, the effect the "uses the whole budget" setting of §18.1's widget models.

Several kinds of control have been tried.

Length rewards during RL. Kimi k1.5 (Kimi Team 2025) added a group-relative length reward. For the $k$ responses to one prompt, with lengths $\ell_i$ between the group's minimum $\ell_{\min}$ and maximum $\ell_{\max}$, let

and zero for every response if all have the same length. In plain words, the shortest response in the group gets $+0.5$ and the longest $-0.5$, correct answers are rewarded for being short, and wrong answers are only ever penalized for being long, never rewarded for being short, so the model cannot gain by answering wrong quickly. The length reward is added to the correctness reward with a weight, and Kimi warmed it up because it slowed training early on.

import numpy as np

def length_reward(lengths, correct):
    L = np.asarray(lengths, float)
    return -L / L.max()

Back to the toy model of chain growth from earlier in this section, now with the Kimi length reward added to the correctness reward at two weights.

At weight 0.1 the easy prompt shrinks from about 2,900 tokens to 1,100 and the medium one from 7,600 to 4,000, while the hard prompt keeps its long chain, and average accuracy moves only from 0.925 to 0.920. At weight 0.3 the length reward starts to outbid correctness, and the hard prompt is cut to about 4,700 tokens at a cost of 15 points of accuracy. Because the reward is relative to the group, it never says how long is long enough. It only says "shorter than your siblings", and its weight decides how much accuracy that is worth.

Long-to-short methods. Kimi k1.5 also compared ways to turn a long-chain model into a concise one, merging the weights of long- and short-chain models, shortest rejection sampling (sample several responses and fine-tune on the shortest correct one), DPO with short correct answers preferred over long ones, and a second RL stage with the length reward. The cell below implements the two simplest ingredients of these pipelines, R1's language-consistency reward and shortest rejection sampling.

Telling the model how long to think. L1 (Aggarwal and Welleck 2025) trained a model with RL on prompts that end in an instruction such as "Think for 1024 tokens.", rewarding correctness minus a penalty proportional to the gap between the requested and the actual length. The resulting 1.5B model followed length budgets closely, outperformed s1's budget forcing by up to 100% relative (20 points absolute) at matched lengths, and beat GPT-4o at equal reasoning lengths. Product controls work at a coarser grain. Thinking budgets in tokens (Claude, Gemini, Qwen3), effort levels (gpt-oss), and thinking-or-not switches in hybrid models all let the caller trade accuracy for cost, and inference-time interventions such as budget forcing (§18.7) do it without retraining.

In PyTorch

Two pieces of reward post-processing come up in every long-chain RL loop and are worth having batched. One is the group-relative length reward above, and the other is DAPO's dynamic-sampling filter (§18.4), which drops groups whose responses all got the same correctness reward before the update.

The weight on the length reward is a real decision. Too small and lengths keep growing, too large and the model learns to stop before it has finished thinking, which costs accuracy on exactly the hard problems that needed the long chain.

Read more

• DeepSeek-AI (2025). DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. The recipe, R1-Zero, distillation, and the unsuccessful attempts. The later version adds training details and costs.
• OpenAI (2024). Learning to reason with LLMs. The o1 announcement, the train-time and test-time compute plots, and the decision to hide raw chains.
• Kimi Team (2025). Kimi k1.5: Scaling Reinforcement Learning with LLMs. Long-context RL, the length reward, and long-to-short methods.
• Yue et al. (2025). Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? and Liu et al. (2025), ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models. The two sides of the sharpening debate.
• Gandhi et al. (2025). Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRs. Why some base models improve under RL and others do not.
• Ye et al. (2025). LIMO: Less is More for Reasoning. Long reasoning elicited by 800 examples (817 in the first version).
• Chen et al. (2024). Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMs and Aggarwal & Welleck (2025), L1: Controlling How Long A Reasoning Model Thinks With Reinforcement Learning. Overthinking measured, and length control trained.
• Yang et al. (2025). Qwen3 Technical Report. Thinking mode fusion and thinking budgets in one model.

Watch next

▶

▶

▶