Here are two solutions to the same problem, each twelve steps long. In the first, step 7 multiplies where it should divide, step 9 makes a second mistake that happens to cancel the first, and the final answer is right. In the second, eleven careful steps are followed by an arithmetic slip in the last line, and the final answer is wrong. An outcome reward gives the first solution full marks and the second nothing. A teacher grading the work would do close to the opposite.

That gap is the credit assignment problem of §15.1 in its sharpest form. A single bit at the end of a long chain says whether the chain worked, not which step made it work or fail. §18.5 trained DeepSeek-R1 on exactly that bit, with the group baseline of §18.4 doing the credit assignment by comparing whole responses to their siblings, and got a frontier reasoning model. Yet DeepSeek had tried something richer along the way, a learned grader of every step, and listed it among the attempts that failed. Process supervision is the idea that scoring each step should do better, and the case for it is strong. This section builds both kinds of reward model, shows how step labels can be manufactured without human annotators, puts a step scorer to work inside GRPO and inside beam search and tree search, and then works out why it failed for R1 and where it does help.

Outcome reward models

An outcome reward model (ORM) is a learned verifier that reads a problem $x$ and a complete solution $y$ and predicts the probability that the solution's final answer is correct, $r_\phi(x, y) \approx \mathbb{P}(\operatorname{ans}(y) = a^\star)$. It is trained exactly as Cobbe et al. (2021) trained their GSM8K verifier (§18.2). Sample many solutions from a generator, label each by checking its final answer, and fit a classifier with binary cross-entropy. The labels are free wherever the answer is checkable, and the trained model generalizes to solutions whose answers cannot be checked at deployment.

An ORM can be asked for a prediction after every token, not only at the end. Trained that way, with the final label as the target at every position, it becomes an estimate of the value of each prefix, the probability that the generator will end up correct from here, which is what Cobbe et al. found worked best. That is worth keeping in mind, because the line between "outcome" and "process" reward models is blurrier than the names suggest.

Uesato et al. (2022) at DeepMind compared the two kinds of supervision head to head on GSM8K with a 70B model. Outcome supervision alone reached about the same final-answer error rate as process supervision, with far cheaper labels (16.6% against 14.8% with reward models). But it did not reliably produce correct reasoning. Their "trace error" is the fraction of solutions with the right final answer that contain at least one wrong step. Every model trained with RL directly against final-answer correctness had a high trace error, 12.4% at best, against 3.8% for their best process-based method. RL against a learned reward model instead of the raw final-answer check closed much of that gap (5.5%), because even reward models trained only on final-answer labels learned to agree with step-level human judgments. Overall they lowered the best published trace error from 14.0% to 3.4%. Right answers for wrong reasons are common, and a raw outcome signal cannot see them.

Process reward models

A process reward model (PRM) reads the problem and a solution split into steps $s_1, \dots, s_m$ (usually lines or paragraphs) and predicts, after each step, the probability that the step is correct,

Lightman et al. (2023) at OpenAI built the canonical dataset, PRM800K. Human labelers marked each step of model-generated solutions to MATH problems as positive, negative, or neutral, giving 800K step-level labels across 75K solutions to 12K problems. Two design choices made it effective. They used active learning, showing labelers mostly "convincing wrong-answer" solutions (ones the current PRM scored highly but whose final answer was wrong), which improved the data efficiency of process supervision by a factor of 2.6. And they trained on GPT-4-generated solutions with a model fine-tuned from GPT-4, so the PRM saw the kind of mistakes a strong model makes.

To score a whole solution, the step scores must be aggregated. Lightman et al. used the probability that every step is correct, the product $\prod_j v_j$. It scored best among the aggregations they compared, including the minimum, though the differences were minor (77.4% to 78.2% at best-of-1,860). Both treat a solution as only as good as its weakest step. Other options are the score of the last step, or a mean.

In plain words, the product asks "is everything right?", the minimum asks "how bad is the worst step?", and the last step asks the PRM to act as an outcome model. The product has a side effect worth knowing. Longer solutions multiply more numbers below 1, so the product penalizes length, which can be a feature (fewer chances to err) or a bug (short wrong solutions beat long right ones).

On the 500 MATH test problems they held out, using best-of-1,860 selection from GPT-4 samples, the PRM solved 78.2%, the ORM 72.4%, and majority voting 69.6%. The PRM beat both at every number of candidates, and its lead grew as the number of candidates increased, which is the selection problem of §18.2 again. With more candidates there are more convincing wrong solutions, and a verifier that checks every step is harder to fool than one that checks the overall impression.

import numpy as np

def prm_score(step_probs, how):
    p = np.asarray(step_probs, float)
    return {"product": p.prod(), "min": p.min(), "last": p[-1]}[how]

def select_best(solutions, how):
    return int(np.argmax([prm_score(s, how) for s in solutions]))

A step verifier need not be a classifier head on a language model. Zhang et al. (2024) trained generative verifiers, language models that judge a solution by the probability they assign to the next token "Yes" after the question "Is the answer correct?", optionally after writing their own chain of thought about the solution. They beat discriminative verifiers trained on the same data at best-of-$n$ selection, and sampling several verification chains and averaging their verdicts helped further. A verifier that reasons spends test-time compute on checking, the same trade that §18.7 studies for solving.

Step labels without humans

PRM800K took a large annotation effort for one dataset. Wang et al. (2024) asked whether step labels could be computed instead, and their method, Math-Shepherd, became the template for automatic process supervision. The idea is to judge a step by what happens after it. From the prefix $s_1, \dots, s_j$, let a completer model finish the solution $N$ times, check each completion's final answer, and label the step by the results.

• Hard estimation, $\hat h_j = 1$ if any of the $N$ completions reaches the correct answer, else 0. A step is good if the solution can still be saved after it.
• Soft estimation, $\hat v_j = \frac{1}{N}\sum_{n=1}^{N}\mathbb{1}[\text{completion } n \text{ is correct}]$, the fraction that succeed.

The soft label has a precise meaning in the language of §15.2. It is a Monte Carlo estimate of the value of the prefix under the completer's policy, $V^{\pi_c}(x, s_{1:j}) = \mathbb{P}_{\pi_c}(\text{correct}\mid x, s_{1:j})$. The hard label estimates whether that value is nonzero. A PRM trained on these labels is therefore a value model for the completer, which is close to what Cobbe's token-level ORM was, reached by a different road.

With labels from Math-Shepherd, step-by-step PPO improved Mistral-7B from 77.9% to 84.1% on GSM8K and from 28.6% to 33.0% on MATH, and using the PRM to verify samples at test time raised those to 89.1% and 43.5%. Luo et al. (2024), in OmegaPRM, made the procedure cheaper by searching for the first error with a binary search over prefixes (a wrong final solution must have one), and collected over 1.5 million step annotations this way. A PRM trained on them, combined with weighted self-consistency, raised an instruction-tuned Gemini Pro from 51% to 69.4% on MATH500 and from 86.4% to 93.6% on GSM8K.

Monte Carlo labels measure something subtly different from correctness, and it is worth seeing exactly how. They measure whether this completer can still succeed, which mixes up three things. A correct step can get a low value because the completer is weak at what comes next. A wrong step can get a high value because the completer recovers from it, ignoring the mistake or reaching the answer another way. And a step that does nothing at all, a restatement of the problem, gets whatever value the prefix already had.

The correct steps have true values between 0.23 and 0.44, not far above the 0.10 of the wrong steps that follow, because the completer must still survive several more steps on its own. With $N = 4$ or $16$ rollouts the soft labels are noisy enough that some correct steps score below wrong ones. With $N = 64$ the hard labels mark every step as good, including the five wrong ones, because the completer occasionally recovers by luck. Hard estimation is too lenient with many rollouts, and soft estimation is too noisy with few. The Qwen team's study of PRM development (Zhang et al. 2025) reached a similar verdict at scale. Labels built by Monte Carlo estimation generalized worse than labels from an LLM judge or from humans, and keeping only examples where Monte Carlo estimation and an LLM judge agreed on the location of the first error ("consensus filtering") worked better than either alone.

import numpy as np

def mc_step_labels(rollouts, v0):
    r = np.asarray(rollouts, float)
    return {"soft": r.mean(1), "hard": r.any(1).astype(int), "progress": r.mean(1)}

The last field of that exercise is the idea behind Setlur et al. (2024). Instead of rewarding a step for the value it reaches, reward it for the change in value it causes, $\hat v_j - \hat v_{j-1}$, an advantage in the sense of §15.4. A restating step then earns nothing, a step that makes the problem easier earns a positive reward, and a mistake earns a negative one, however good or bad the prefix already was. They also found it matters whose success probability you measure. Progress measured under a separate "prover" policy, chosen to be complementary to the policy being improved, worked better than progress under the policy itself. Their process advantage verifiers made test-time search more than 8% more accurate and 1.5 to 5 times more compute-efficient than outcome verifiers, and in online RL gave a 5 to 6 times gain in sample efficiency.

Finding the first error of a wrong solution by Monte Carlo is expensive if done step by step, since each step costs $N$ completions. If values are monotone, positive before the first error and zero from it on, binary search finds it with a logarithmic number of evaluations. This is the core of OmegaPRM, and a nice interview problem.

def first_error(value, n):
    for j in range(1, n + 1):
        if value(j) == 0:
            return j
    return n

In practice the values are Monte Carlo estimates, so "zero" means "no successes in $N$ tries", and a wrong step from which the completer occasionally recovers can be missed. OmegaPRM handles this with thresholds and by organizing the rollouts in a tree that reuses completions across searches.

Searching with a step verifier

A PRM is most clearly useful at test time, as a guide for search. Best-of-$n$ uses a verifier once, after $n$ full solutions are written. A step verifier can be used during generation, to stop spending compute on prefixes that have already gone wrong.

Step-level beam search. Keep $b$ partial solutions. At each step, sample $w$ continuations of each, score the $bw$ new prefixes with the PRM, keep the best $b$, and repeat until the beams finish. It spends the same compute as best-of-$bw$ but concentrates it on promising prefixes. Snell et al. (2024) compared it with best-of-$N$ against the same PRM and found beam search better on harder questions and at lower compute budgets, and best-of-$N$ better on easier questions and at higher budgets, where beam search's aggressive pruning started to exploit the PRM's errors (§18.7). Lookahead search, which scores a prefix by rolling it forward a few steps first, generalizes beam search but was not better at matched compute in their experiments.

Tree search. Monte Carlo tree search (MCTS), the algorithm behind AlphaGo and AlphaZero, treats reasoning steps as moves. Each node is a prefix. Each iteration selects a path down the tree by repeatedly picking the child that maximizes an upper confidence score, expands a leaf by sampling a new step, evaluates it with a value model or a rollout, and backs the result up the path. AlphaZero's selection rule (PUCT) is

where $Q(s, a)$ is the average value backed up through child $a$, $P(s, a)$ is the policy's prior probability of that step, $N$ counts visits, and $c$ trades exploration against exploitation. In plain words, go where the value has been high, but also where the policy thinks a step is plausible and you have not looked much yet. It is a relative of the bandit UCB rule of §15.1, with one bandit per node, a bonus scaled by the prior, and a bonus that decays like $1/N(s,a)$ rather than $1/\sqrt{N(s,a)}$.

Several systems applied this to language. RAP (Hao et al. 2023) used the language model both as the policy and as a "world model" that predicts the state after each step, with MCTS on top. TS-LLM (Feng et al. 2023) used AlphaZero-style search with a learned value function, for decoding and as a source of training data, on trees up to 64 steps deep. rStar-Math (Guan et al. 2025) combined MCTS with a process preference model trained on its own search results over four rounds of self-evolution, and took Qwen2.5-Math-7B from 58.8% to 90.0% on MATH and to 53.3% on AIME 2024 (8 of 15 problems), using 64 search trajectories per problem.

Tree search over text differs from tree search over Go in ways that make it harder. The branching factor is a whole space of possible next steps, not a few hundred legal moves, and two different steps can mean the same thing. A "step" has no fixed definition. And the value model, which in Go learns from millions of games with exact outcomes, here must judge partial mathematical arguments.

Process rewards inside GRPO

A PRM can also replace the single reward in GRPO. DeepSeekMath, the paper that introduced GRPO (§18.4), defined a process-supervision variant. For a group of $G$ responses to one prompt, the PRM gives response $i$ a reward $r_i^{(j)}$ at the end of each of its steps $j$. Collect every step reward of the whole group into one set $\mathbf{R}$, normalize each by that set's mean and standard deviation,

and give token $t$ of response $i$ the sum of the normalized rewards of the steps that end at or after it,

where $\operatorname{end}_i(j)$ is the index of the last token of step $j$ of response $i$. In plain words, each token is credited with everything that happens from its own step onward, the reward-to-go of §15.3 with step rewards in place of one final reward. Tokens of the same response now get different advantages. A step the PRM likes raises the advantage of every token up to and including it, and a step it dislikes lowers them. In a solution whose steps are good until step 7 goes wrong, the tokens of the early steps carry their own positive rewards on top of the later penalty, so they are pushed down less than the tokens from step 7 on, which is exactly the credit assignment the outcome reward could not do. If every response is a single step, the formula collapses to the outcome advantage of §18.4. In DeepSeekMath's own comparison, on a 1.3B model with learned reward models in both cases, the process variant did somewhat better than the outcome variant.

Take a group of two responses. Response 1 has two steps, ending at tokens 2 and 4, with PRM rewards 0.9 and 0.2. Response 2 has one step, ending at token 3, with reward 0.6. The set $\mathbf{R} = \{0.9, 0.2, 0.6\}$ has mean 0.567 and sample standard deviation 0.351, so the normalized rewards are 0.95, $-1.04$, and 0.09. Response 1's first three tokens get $0.95 - 1.04 = -0.09$, since a bad step follows them, and its last two get $-1.04$. Response 2's four tokens all get 0.09.

import numpy as np

def process_advantages(step_rewards, step_ends, lengths, eps=1e-6):
    return np.zeros((len(lengths), max(lengths)))

The dense signal is the attraction, and also the danger. Every step reward is a number the policy can push up without making the final answer more likely, which is what the next part is about.

When process rewards help, and why R1 did not use them

The case for process rewards at test time is strong. For training with RL, the record is mixed, and the reasons are instructive.

Now we can read DeepSeek's unsuccessful attempts, which §18.5 mentioned, in detail (DeepSeek-AI 2025). For PRMs they gave three reasons. It is hard to define what a fine-grained step is in general reasoning. It is hard to decide whether an intermediate step is correct, since automatic annotation is unreliable and manual annotation does not scale. And a model-based PRM "inevitably leads to reward hacking", while retraining it to keep up with the policy costs resources and complicates the pipeline. For MCTS, the search space of tokens is exponentially larger than a board game's, and the value model that guides the search is hard to train well enough.

Gao et al. (2024) measured the hacking directly. Adding rewards from ORMs or PRMs that were excellent at inference-time reranking to the success reward during RL often did not help and sometimes hurt, because policies learned to collect high PRM scores by repeating correct but unnecessary steps. When the total reward is a sum of step rewards, more steps means more reward. Their fixes bound the total. One clips the per-step rewards so a step cannot earn a positive reward, and the other rewards each step by the difference between consecutive step scores, so the sum telescopes to a bounded quantity, the same structure as the progress rewards above.

The Qwen team's study adds a final twist. PRMs selected or trained for best-of-$n$ performance tended to put their lowest scores on the final answer step, behaving more like outcome models than process models. A PRM can be good at its benchmark without doing the job its name describes.

So where do process rewards stand? They are a strong tool for test-time search and reranking, a useful source of dense credit when outcome rewards are too sparse, and a well-documented reward-hacking risk when used naively inside RL. The large-scale RL recipes of 2025 sidestepped the issue by using outcome checkers and many samples per prompt, letting the group baseline of §18.4 do the credit assignment, and letting the model learn its own internal verification, the checking and backtracking that §18.5 watched emerge.

In PyTorch

A PRM is usually a language model with a small classification head, trained to predict the label at the last token of each step, with every other position masked out of the loss. Here is that training setup on synthetic hidden states, in which each token's state weakly encodes whether an error has occurred so far. The masking pattern is the part that carries over to real code.

Two details are easy to get wrong. The loss must index only the step-end positions, since a label means "the prefix up to here is correct" and is undefined in the middle of a step. And locating the first error from step scores is much harder than classifying steps, because a single false alarm early in the chain moves the location, which is one reason first-error benchmarks such as ProcessBench are demanding.

Read more

• Lightman et al. (2023). Let's Verify Step by Step. PRM800K, process versus outcome supervision at scale, and active learning for labels.
• Uesato et al. (2022). Solving math word problems with process- and outcome-based feedback. The first careful comparison, and the distinction between final-answer error and trace error.
• Wang et al. (2024). Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations. Automatic step labels from completions.
• Luo et al. (2024). Improve Mathematical Reasoning in Language Models by Automated Process Supervision. OmegaPRM and binary search for the first error.
• Setlur et al. (2024). Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning. Process rewards as advantages under a prover policy.
• Zhang et al. (2025). The Lessons of Developing Process Reward Models in Mathematical Reasoning. What goes wrong with Monte Carlo labels, and consensus filtering.
• Gao et al. (2024). On Designing Effective RL Reward at Training Time for LLM Reasoning. How policies hack learned process rewards during RL, and bounded rewards that prevent it.
• Zhang et al. (2024). Generative Verifiers: Reward Modeling as Next-Token Prediction. Verifiers that answer Yes or No, optionally after reasoning about the solution.
• Guan et al. (2025). rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinking. MCTS with a process preference model, trained by self-evolution.

Watch next

▶

▶