PPO for language models, as RLHF runs it, keeps four networks in memory, the policy being trained, a frozen reference copy, a reward model, and a value model (§16.4). The value model is the awkward one. It is usually as large as the policy, it must be trained alongside it, and its job in reasoning RL is close to impossible. After every prefix of a long solution, it must predict the probability that the final answer will turn out correct. A bad value model makes bad advantages, and bad advantages make a bad policy.

In early 2024 the DeepSeekMath paper proposed throwing the value model away (Shao et al. 2024). For each prompt, sample a group of $G$ responses, score each one, and use the group's own average reward as the baseline. A response is good if it beat its siblings. They called it Group Relative Policy Optimization (GRPO), and a year later it trained DeepSeek-R1. Around the same time, Ahmadian et al. (2024) argued from the RLHF side that PPO's machinery was unnecessary for language models and that a REINFORCE estimator with a leave-one-out baseline, RLOO, did better. The two methods are closer than their names suggest, and understanding exactly how close is the fastest way into this whole family of algorithms.

This section derives the group baseline from the policy gradient, writes the GRPO objective term by term, then looks hard at the details that later papers found to matter, the normalizations that introduce biases (Dr. GRPO), the tricks that make long-chain RL stable (DAPO), and the move from token-level to sequence-level ratios (GSPO). It ends with a complete implementation.

The policy gradient with a group baseline

Treat a whole response as one action, as §15.7 does for terminal rewards. With prompt $x$, response $y \sim \pi_\theta(\cdot\mid x)$, and a scalar reward $R(x, y)$, the policy gradient theorem with a baseline (§15.4) is

for any baseline $b(x)$ that does not depend on $y$. The natural baseline is the prompt's value, the expected reward $V(x) = \mathbb{E}_y[R(x, y)]$, which with 0/1 rewards is the success probability $p(x)$ of §18.2. PPO learns $V$ with a network. GRPO and RLOO estimate it from other samples of the same prompt.

The leave-one-out baseline

Draw $G$ responses $y_1, \dots, y_G$ independently from $\pi_\theta(\cdot\mid x)$ with rewards $r_1, \dots, r_G$. For response $i$, use the mean reward of the other $G - 1$ responses,

In plain words, each response is judged against the average of its siblings, and pushed up if it did better and down if it did worse.

This is unbiased, and the proof is the baseline lemma of §15.4 with one extra observation. Given $x$, the baseline $b_i$ is a function of $y_j$ for $j \ne i$ only, and those are independent of $y_i$. So

because the expected score is zero. Each term of $\hat{\mathbf{g}}_{\text{RLOO}}$ is therefore an unbiased estimate of $\nabla_\theta\mathbb{E}_y[R]$, and so is their average. The baseline costs nothing beyond the extra samples, which RLOO needs anyway, and it is an unbiased estimate of $V(x)$ built from data the policy just produced. This is the estimator of Kool, van Hoof, and Welling (2019), "Buy 4 REINFORCE Samples, Get a Baseline for Free!", applied to language models by Ahmadian et al.

The group mean is the leave-one-out baseline, rescaled

Now use the plain group mean $\bar r = \frac1G\sum_j r_j$ as the baseline instead, which includes response $i$'s own reward. §15.4 warned that this makes the baseline depend on the action. Here is exactly how much. Since $\sum_{j\ne i}r_j = G\bar r - r_i$,

So $r_i - \bar r = \frac{G-1}{G}(r_i - b_i)$. In plain words, subtracting the group mean gives exactly the leave-one-out advantages shrunk by the constant factor $(G-1)/G$. The resulting gradient estimate points in an unbiased direction and is $1 - 1/G$ times too short on average, a "bias" that any learning rate absorbs. With $G = 16$, the factor is 0.94. The worry in §15.4 was correct in principle and harmless in practice.

The leave-one-out estimator is unbiased (ratio 1), the group-mean estimator is unbiased up to the factor $7/8$, and both have less variance than plain REINFORCE. The group mean's variance looks lower still, but only because its estimates are shrunk by $7/8$. Multiply the leave-one-out variance by $(7/8)^2 \approx 0.77$ and you get the group-mean number, so the two are the same estimator at different scales.

import numpy as np

def rloo_advantages(rewards):
    r = np.asarray(rewards, float)
    return r - r.mean(1, keepdims=True)

GRPO's advantage

GRPO uses the group mean and also divides by the group's standard deviation,

with a small $\epsilon$ to avoid dividing by zero. For outcome rewards, every token of response $i$ gets this same advantage, $\hat A_{i,t} = \hat A_i$. The paper does not say which standard deviation. At the time of writing, TRL and verl both use the sample standard deviation, with Bessel's correction (dividing by $G - 1$), and we use that convention. With $G = 16$ the two conventions differ by only 3%, but the difference is enough to fail an exact test.

A worked example makes the numbers concrete. Take $G = 8$ responses of which $k = 2$ are correct, with rewards 1 and 0. The mean is $0.25$. The sample standard deviation is $\sqrt{\frac{2(0.75)^2 + 6(0.25)^2}{7}} = \sqrt{12/56} = 0.463$. So each correct response gets $\hat A = 0.75/0.463 = 1.62$ and each wrong one gets $-0.25/0.463 = -0.54$. The advantages sum to zero, as they always do. The two correct responses share a push up of $2 \times 1.62 = 3.24$ and the six wrong ones share a push down of the same total.

Notice what happens when all $G$ responses get the same reward. The numerator is zero for everyone, and the group contributes no gradient at all. With 0/1 rewards, a prompt that the policy solves with probability $p$ produces a mixed group, one with any signal, with probability

For $G = 8$, a prompt at $p = 0.05$ is informative only 34% of the time, and one at $p = 0.95$ the same. Prompts that are too easy or too hard for the current policy waste their rollouts. This fact drives much of the engineering that follows, from DAPO's dynamic sampling to curricula that keep the training set near the frontier of what the model can do.

import numpy as np

def group_advantages(rewards, normalize_std=True, eps=1e-6):
    r = np.asarray(rewards, float)
    return r - r.mean()

The GRPO objective

DeepSeekMath writes the objective at the token level, borrowing PPO's clipped surrogate (§15.6). Sample the group from the policy as it was before the update, $\pi_{\theta_{\text{old}}}$, and define each token's probability ratio

Then GRPO maximizes

where $|y_i|$ is the number of tokens in response $i$, $\epsilon$ is the clip range, and $\hat D_{i,t}$ is a per-token estimate of the KL divergence to the reference policy,

In plain words, for every token of every response, take PPO's clipped surrogate with the response's group-relative advantage, subtract a penalty for drifting from the reference model, average the tokens within each response, and average the responses.

Each piece has a reason, and two of them have consequences that took the field a year to notice.

• The ratio and the clip are PPO's. They let the same batch of samples be reused for several gradient steps without the policy running away from the distribution that produced them. When the batch is used for a single step, as in DeepSeekMath's own runs, the ratio is exactly 1 at the update and the clip never binds. The gradient of $\rho_{i,t}$ at $\theta = \theta_{\text{old}}$ is $\nabla\log\pi_\theta(y_{i,t}\mid\cdot)$, so GRPO reduces to REINFORCE with group-normalized advantages plus the KL gradient.
• The KL term is the $k_3$ estimator of §0.5, with $q = \pi_\theta$ and $p = \pi_{\text{ref}}$, whose value is a never negative estimate of $D_{\mathrm{KL}}(\pi_\theta\Vert\pi_{\text{ref}})$, unbiased when the tokens are sampled from $\pi_\theta$ itself (here they come from $\pi_{\theta_{\text{old}}}$, so it is a close approximation). GRPO adds it to the loss. RLHF with PPO instead subtracts a per-token KL from the reward (§16.4), which then flows through the advantage estimate. DeepSeekMath gives the reason, that putting it in the loss avoids complicating the calculation of $\hat A_{i,t}$. The choice has a cost that §15.7 works out in its table of $k_1$, $k_2$, and $k_3$. Differentiated at the sampled tokens, $k_3$'s gradient is $(1 - \pi_{\text{ref}}/\pi_\theta)\nabla\log\pi_\theta$, whose expectation is the gradient of the KL with its arguments swapped (the forward KL), $D_{\mathrm{KL}}(\pi_{\text{ref}}\Vert\pi_\theta)$, not of the reverse KL $D_{\mathrm{KL}}(\pi_\theta\Vert\pi_{\text{ref}})$ that the objective writes down and the estimate reports. GRPO's penalty still pulls the policy toward the reference, but through a different divergence. (Used as a reward instead, $k_3$ would be wrong too. Only $k_1$ in the reward gives the reverse KL's gradient.)
• The $1/|y_i|$ average weights each response equally regardless of its length. This is the first of the two consequences.
• The std in $\hat A$ rescales each prompt's gradient by how varied its group's rewards were. This is the second.

DeepSeekMath trained its 7B model with $G = 64$ samples per question, a learning rate of $10^{-6}$, $\beta = 0.04$, a maximum length of 1,024 tokens, a batch of 1,024, and a single policy update per batch, on about 144K chain-of-thought questions from GSM8K and MATH. Its rewards came from a learned reward model trained on DeepSeekMath-Base, not from a program, since the rule-based rewards of RLVR (§18.3) became the norm only with R1. GRPO took the instruction-tuned model from 82.9% to 88.2% on GSM8K and from 46.8% to 51.7% on MATH. The paper also defines a process supervision variant, in which a process reward model (§18.6) scores each step, the step rewards are normalized across the whole group, and each token's advantage is the sum of the normalized rewards of the steps from its own onward. And it describes iterative GRPO, which periodically retrains the reward model on fresh policy samples (with 10% of historical data replayed) and resets the reference model to the current policy.

import numpy as np

def grpo_loss(logp, logp_old, logp_ref, adv, mask, eps_low=0.2, eps_high=0.2, beta=0.04, agg="seq_mean", max_len=None):
    ratio = np.exp(logp - logp_old)
    surr = ratio * adv[:, None]
    loss = -(surr * mask).sum() / mask.sum()
    return loss, np.zeros_like(logp)

Two biases, and Dr. GRPO

Liu et al. (2025) looked at R1-Zero-style training closely and argued that two of GRPO's normalizations bias the optimization in ways nobody intended. Their fix, which they called Dr. GRPO ("GRPO done right"), removes both.

The length bias

Under the $\frac{1}{|y_i|}\sum_t$ average, the total weight of response $i$ on the gradient is $\hat A_i/G$, whatever its length, so each of its tokens gets $\hat A_i/(G|y_i|)$. Follow that through for the two signs of the advantage.

• Correct responses ($\hat A_i > 0$). A short one gets a larger push per token than a long one. The update favors brevity among correct answers.
• Wrong responses ($\hat A_i < 0$). A long one gets a smaller push down per token than a short one. Take two wrong responses with $\hat A = -1$ in a group of $G = 8$, one of 100 tokens and one of 1,000. Each token of the short one is pushed down by $1/800$ and each token of the long one by $1/8000$, ten times less. The update penalizes long wrong answers less, so wrong answers drift longer.

The second effect is the troubling one, because response length grows during reasoning RL for good reasons too (§18.5), and the bias makes some of that growth an artifact of the loss rather than useful thinking. Dr. GRPO replaces the per-response average with a sum over tokens divided by a constant, the maximum generation length (any constant works, it just rescales the gradient). DAPO, independently, divides by the total number of tokens in the group, $\sum_i|y_i|$, so every token in the group counts equally, and many implementations average over all tokens in the batch. Either way, every token in a group (or batch) gets the same weight, so a long response is no longer underweighted per token relative to a short one beside it.

The same issue appeared in §15.3 in a different guise. Dividing each episode's sum by its own length changes the objective, while dividing by a constant or by the batch's total token count only rescales the step, and the analysis above says precisely how the first one changes it.

The difficulty bias

The std normalization multiplies each prompt's contribution by $1/\operatorname{std}(r)$. With 0/1 rewards and large $G$, the sample std approaches $\sigma(x) = \sqrt{p(1-p)}$, where $p = p(x)$ is the prompt's success probability. The mean-centered estimator's expected gradient is $\frac{G-1}{G}\nabla_\theta p$, from the identity above. So in the large-$G$ limit,

using $\frac{d}{dp}\arcsin\sqrt p = \frac{1}{2\sqrt{p(1-p)}}$. In plain words, GRPO does not maximize the average accuracy $\sum_x p(x)$. It maximizes $\sum_x 2\arcsin\sqrt{p(x)}$, which weights a unit of progress on a prompt by $1/\sqrt{p(1-p)}$, smallest at $p = 0.5$ and growing without bound as $p$ approaches 0 or 1. Prompts the model almost always gets right, and prompts it almost always gets wrong, are upweighted relative to the ones in the middle.

Is that bad? It is a different objective, and not necessarily a worse one. Upweighting hard prompts can be useful, and the finite-$G$ effect partly cancels it, since a prompt near $p = 0$ usually produces an all-wrong group with no signal at all. But it is not the objective anybody wrote down, and Liu et al. call it a question-level difficulty bias. Dr. GRPO uses the plain mean-centered advantage $r_i - \bar r$. REINFORCE++ (Hu 2025) keeps a normalization but computes the mean and std over the whole global batch instead of per prompt, which removes the per-prompt reweighting.

The mean-only column is exactly $\frac{63}{64}\,dp/dz$, the true gradient up to the constant. With the std and $G = 64$, the expected gradient is close to $\sqrt{p(1-p)} = (dp/dz)/\sqrt{p(1-p)}$ (somewhat below it at $p = 0.02$, where $0.98^{64} \approx 27\%$ of groups are still all wrong), which at $p = 0.02$ is about six times the true gradient and at $p = 0.5$ only twice it, so the hard prompt gets about three times the relative weight. With $G = 8$ the hard prompt rarely produces a mixed group, and its expected gradient falls.

Liu et al. also studied the base models that this kind of RL starts from. While most base models need a template to answer a question rather than keep writing text, Qwen2.5 base models answered best with no template at all, scoring about 60% higher on average than with the usual 4-shot prompt, which the authors suspect reflects pretraining on concatenated question-answer text. (Their other finding about base models, on where the famous "aha moment" comes from, belongs with the recipe it questions, in §18.5.) Their minimal recipe, Dr. GRPO on Qwen2.5-Math-7B with MATH problems of levels 3 to 5, reached 43.3% on AIME 2024 with 27 hours on 8 A100 GPUs.

DAPO's four fixes

ByteDance's DAPO paper (Yu et al. 2025) set out to reproduce R1-Zero-scale results openly and found that naive GRPO on Qwen2.5-32B reached only 30 points on AIME 2024. Four changes, plus dropping the KL term, took it to 50, beating the 47 that DeepSeek reported for R1-Zero-style training of the same base model, in half the training steps. The ablation adds them one at a time.

Clip-higher. PPO's symmetric clip limits how far one update can raise a token's probability, to $(1 + \epsilon)$ times its old value. For a token the policy already likes, $\pi_{\text{old}} = 0.9$, that is no constraint, since $0.9 \times 1.2$ exceeds 1. For an unlikely token, $\pi_{\text{old}} = 0.01$, it caps the new probability at 0.012, an absolute gain of 0.002, so the rare tokens that start a new line of reasoning can grow only slowly while the already-likely ones are effectively unconstrained. The authors linked this asymmetry to entropy collapse, the policy becoming nearly deterministic early and exploring nothing new. They decoupled the clip range, keeping $\epsilon_{\text{low}} = 0.2$ for decreases and raising $\epsilon_{\text{high}} = 0.28$ for increases.

Dynamic sampling. Groups in which every response is right, or every response is wrong, have zero advantage and contribute nothing except noise in the batch statistics. DAPO over-samples prompts and discards those whose groups are all correct or all wrong, refilling until the batch is full of mixed groups. Formally, the objective is taken only over prompts with $0 < |\{i : y_i \text{ correct}\}| < G$. This cost more generation per step and paid for itself in steps, the largest single gain in the ablation.

Token-level loss. As discussed above, normalize by the total number of tokens, $\frac{1}{\sum_i|y_i|}\sum_i\sum_t$, so long responses are not underweighted per token.

Overlong reward shaping. Responses cut off at the length limit have no answer and get a wrong-answer reward even when the reasoning was going well. Overlong filtering masks their loss entirely. Soft overlong punishment (implemented in §18.3's PyTorch block) adds a length penalty that ramps in over the last 4,096 tokens before the 20,480-token limit.

DAPO also used a rule-based reward of $+1$ for a correct answer and $-1$ otherwise, $G = 16$ samples per prompt, a constant learning rate of $10^{-6}$, 512 prompts per rollout step, and 16 gradient updates per rollout step, so its ratios do move away from 1 and its clipping does bind. It removed the KL penalty, arguing that a model trained to produce long chains of thought is supposed to move far from its starting distribution.

GSPO, ratios at the sequence level

Qwen's GSPO paper (Zheng et al. 2025) attacked a different part of the objective, the token-level ratio itself. Importance sampling corrects for a mismatch between two distributions by reweighting samples, and it works when you average over many samples from the same distribution. A token-level ratio $\rho_{i,t}$ reweights a single sampled token at a single context, which cannot perform a meaningful distribution correction and adds noise. The noise accumulates over long responses, the clip interacts with it, and in their experiments it led to training collapse, especially for mixture-of-experts models, whose routing changes between the old and new policy make individual token ratios jump around (§13.6). GRPO on MoE models needed routing replay, recording which experts each token used during sampling and forcing the same experts when computing the new probabilities.

GSPO replaces the token ratio with a length-normalized sequence ratio,

the geometric mean of the token ratios, and clips at the sequence level,

In plain words, the unit of reward is a whole response, so the unit of importance weighting and clipping should be a whole response too. The length normalization keeps $s_i$ near 1 for long responses, which is why GSPO's clip ranges are tiny, $3\times10^{-4}$ below and $4\times10^{-4}$ above in their experiments (their GRPO baseline used 0.2 and 0.27). GSPO clipped about two orders of magnitude more tokens than GRPO and still trained more efficiently, it made routing replay unnecessary for MoE models, and Qwen credits it for the improvements in its Qwen3 models. A variant, GSPO-token, allows per-token advantages while keeping the sequence-level ratio's value.

The wild token shifts the mean log-ratio by only $\log 4/1000 \approx 0.0014$, so the length-normalized ratio stays within a fraction of a percent of 1. The raw sequence ratio, the product of all thousand token ratios, is at the mercy of a thousand small fluctuations. It came out at 4.97 here, but its logarithm has standard deviation $0.1\sqrt{1000} \approx 3.2$, so a factor of 20 either way is typical, and the spread grows with length, which makes it useless as an importance weight.

The family at a glance

ReMax (Li et al. 2023) is the one row not met above. It uses the reward of the policy's greedy response as the baseline, which costs one extra generation per prompt instead of a group.

Libraries such as TRL, verl, and OpenRLHF expose most of these choices as configuration flags, so in practice "GRPO" names a family, and a paper that reports "GRPO" results needs to say which members of the family it used.

A toy training run

Here is the whole idea on a problem small enough to watch. A "reasoning task" has $P$ prompt types, and for each type the model chooses among $A$ strategies, each with its own probability of producing a correct answer. The policy is a table of logits, one row per prompt type. Each step samples a group of $G$ strategies per prompt, observes 0/1 rewards, and takes one step of REINFORCE with the group-mean baseline, the Dr. GRPO advantage. For a softmax policy the score of action $a$ is $\mathbf{e}_a - \boldsymbol\pi$ (§15.3), so the update for one prompt's logits $\mathbf{z}$ is

import numpy as np

def group_update(z, actions, rewards, lr):
    return z

def train(S, steps, G, lr, seed):
    S = np.asarray(S, float)
    return np.zeros_like(S), [float(S.mean())] * (steps + 1)

In PyTorch

Now the real thing at toy scale, a complete GRPO loop on an autoregressive policy with variable-length responses, masking, a reference model, the $k_3$ KL penalty, DAPO's asymmetric clip and token-level loss, and two gradient steps per batch so the ratio and the clip actually do something. The task is a tiny verifiable one. The prompt is a number $t$ from 1 to 4, the vocabulary is the tokens 1, 0, and an end token, and the reward is 1 exactly when the response is $t$ copies of 1 followed by the end token. A uniformly random policy solves $t = 1$ with probability $(1/3)^2 \approx 11\%$ and $t = 4$ with probability $(1/3)^5 \approx 0.4\%$. (The randomly initialized network is not exactly uniform, and the printed starting accuracies are estimates from 256 samples each.)

Every line maps to a piece of the objective. The group statistics are computed per prompt with view(-1, G). The mask stops at the first end token, so padding contributes nothing. The old and reference log-probabilities are computed once, without gradients, before the updates. And the hardest prompts, solved a few times in a thousand at the start, are learned because the few groups that contain a success give that success a large positive advantage.

What a GRPO step costs at scale

At real scale the gradient is the cheap part. Take DAPO's batch, 512 prompts per rollout step with $G = 16$ responses each, on a 32B-parameter model, and suppose the responses average 5,000 tokens (reasoning RL often runs longer). Count FLOPs with the $2N$ per generated token and $6N$ per trained token of §9.2, where $N$ is now the parameter count.

Generation is only a sixth of the FLOPs, yet practitioners report that it takes most of the wall-clock time, because decoding is memory-bound. Each decode step reads every weight and the whole KV cache to produce one token per sequence (§6.5, §11.2), while the training passes process all 41M tokens as large matrix multiplications. And the step cannot finish until the longest response does, so a few 20,000-token responses can hold up a batch of 5,000-token ones. Three engineering consequences follow, and interviewers like to ask about all three.

• Separate engines. Rollouts run in a fast inference engine (vLLM or SGLang) and the loss in a training framework, with fresh weights copied to the sampler after each update. The two compute slightly different probabilities for the same token, because their kernels, precision, and batching differ, so samples are a little off-policy even when the batch is used for a single update. Yao et al. (2025) measured this mismatch and corrected it with truncated importance sampling, weighting each token's loss by $\min(\pi_{\text{trainer}}/\pi_{\text{sampler}}, C)$ for a cap $C$.
• Asynchronous rollouts. Generating batch $k+1$ while training on batch $k$ keeps the GPUs busy, at the price of samples that are one or more policy versions stale. The ratio and clip in GRPO's objective were designed for exactly this kind of staleness.
• No critic. The value model would be another 32B network to train, 512 GB of weights, gradients, and Adam state at the 16 bytes per parameter of §10.1, plus its forward and backward passes. Dropping it is a large part of GRPO's appeal.

Read more

• Shao et al. (2024). DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. GRPO's origin, Section 4, with the outcome and process variants and the analysis of why RL works in Section 5.2.
• Ahmadian et al. (2024). Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs. RLOO, and the argument that PPO's machinery is unnecessary for language models.
• Kool, van Hoof & Welling (2019). Buy 4 REINFORCE Samples, Get a Baseline for Free!. The leave-one-out baseline.
• Liu et al. (2025). Understanding R1-Zero-Like Training: A Critical Perspective. Dr. GRPO, the length and difficulty biases, and what base models already do before RL.
• Yu et al. (2025). DAPO: An Open-Source LLM Reinforcement Learning System at Scale. Clip-higher, dynamic sampling, token-level loss, and overlong shaping, with a full ablation.
• Zheng et al. (2025). Group Sequence Policy Optimization. Sequence-level ratios and clipping, and why token-level ratios destabilize MoE training.
• Li et al. (2023). ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Models. The greedy-response baseline.
• Hu (2025). REINFORCE++: Stabilizing Critic-Free Policy Optimization with Global Advantage Normalization. Normalizing advantages over the global batch instead of per prompt.

Watch next

▶

▶

▶

▶