The RLHF algorithm fits on one page, and §16.4 wrote that page. Making it work is another matter. When Huang et al. (2024) set out to reproduce OpenAI's 2020 summarization results with open models, they ended up documenting more than twenty implementation details, from which token the reward is read at to whether dropout is on, any one of which could quietly break the result. Labs that ran RLHF at scale tell the same story.

This section is the practical companion to the rest of the chapter. It collects the hyperparameters published recipes actually used, explains reward and advantage whitening and the scale problem behind them, lists the stability tricks that matter and the numbers to watch during a run, shows how to evaluate the trained model honestly, and places RLHF in the post-training stacks labs use today, next to rejection sampling, DPO, and RL with verifiable rewards.

What published recipes used

Here are four documented PPO-based recipes side by side. "n/a" means the paper does not state the value, not that it was absent.

Several patterns are worth absorbing. Learning rates are small, from $10^{-6}$ to about $10^{-5}$, and usually below the same model's SFT rate (Llama 2 used $10^{-6}$ for PPO against $2\times10^{-5}$ for SFT, and Stiennon et al. about a quarter of their SFT rate), because a policy-gradient step moves the distribution the model samples from, and the next batch of data with it. Huang et al. are the exception that shows the range, using $3\times10^{-6}$ for SFT, reward model, and PPO alike. Batches are large (512 responses), because per-token advantages are noisy. Training is short, a few hundred iterations, because the gold reward peaks (§16.5). And the KL coefficients look small but are not comparable across papers, for a reason the next part explains.

Whitening, and why the reward scale is arbitrary

A Bradley-Terry reward model is identified only up to a shift per prompt and is measured in units of annotator noise (§16.3). The scale matters for RL in a precise way. Multiply the reward by a constant $c > 0$ and the RLHF objective becomes

the original objective with $\beta$ replaced by $\beta/c$, multiplied by a constant that does not change the maximizer. In plain words, the reward scale and $\beta$ are one knob, not two. A reward model with twice the spread behaves like half the $\beta$. That is why $\beta = 0.05$ in one paper and $0.005$ in another can mean the same thing, and why practitioners normalize.

Three normalizations are in common use, at three different places.

Fixing the reward model's offset once. Shift the reward model's bias so a reference set of responses averages zero (labeler demonstrations in InstructGPT, the dataset's reference summaries in Stiennon et al. and Huang et al.). Ziegler et al. (2019) also scaled to unit variance on samples from the reference policy. This fixes both the offset and the scale before RL starts, so $\beta$ has a stable meaning.

Whitening the rewards in each batch. Normalize the batch's scores to zero mean and unit variance before building the shaped rewards. It keeps the effective $\beta$ constant as the reward distribution shifts during training, at the cost of making the reward relative to the batch. Huang et al. (2024) list it as optional and found it made outputs shorter and lowered raw preference rates, with similar results once length was controlled. Llama 2 first inverted the sigmoid, mapping each reward to the log-odds scale, and then whitened.

Whitening the advantages. Normalize $\hat A_t$ over the real tokens of the batch before the PPO loss. This is nearly universal, and Xu et al. (2024) found it one of three factors that decided whether PPO worked well, alongside large batches and an exponential moving average update of the reference model. It makes the size of the policy step independent of the reward scale, so the learning rate and clip range keep their meaning.

The detail that trips people up is the mean. Whitening advantages should subtract the mean, so roughly half the tokens are pushed up and half down. Whitening rewards often should not subtract it, rescaling only, because the sign of the reward relative to zero carries information when the reward model was normalized to a reference. Libraries expose this as a flag, often called shift_mean.

import numpy as np

def masked_whiten(x, mask, shift_mean=True, eps=1e-8):
    return (x - x.mean()) / np.sqrt(x.var() + eps)      # placeholder: statistics include padding

Stability tricks that matter

Most of these appear in at least two of the recipes above, and each one fixes a specific failure.

• Initialize the value model from the reward model, so the critic starts out knowing how good partial responses look, and train it on its own (Stiennon et al., InstructGPT, Huang et al.). A critic initialized from the policy starts out knowing nothing about reward and produces useless advantages for the first part of training.
• Turn dropout off in all four models. Otherwise $\log\pi_\theta$ and $\log\pi_{\text{old}}$ differ on the same tokens before any update, the ratio is not 1 at the first step, and the KL reward is random (Huang et al.).
• Penalize responses that do not end. A fixed negative score when no end-of-sequence token appears (Huang et al. used $-1$) stops the policy from running to the length limit, where the reward model's verdict on a truncated response is meaningless.
• Use large batches and few epochs. Xu et al. (2024) found batch size gave the largest gains among the factors they studied, especially for code. InstructGPT trained one epoch per batch.
• Clip the value function and the gradients. Value clipping keeps the critic from chasing one noisy batch, and global gradient-norm clipping (§8.8) guards against the occasional huge advantage.
• Mix in pretraining gradients to reduce the alignment tax. InstructGPT's PPO-ptx added the language-modeling loss on pretraining data, scaled by $\gamma_{\text{ptx}} = 27.8$ (§16.4 writes the objective), and this reduced the performance regressions on public NLP datasets, although some (DROP, SQuADv2, translation) remained.
• Update the reference slowly instead of freezing it. Xu et al. found an exponential moving average update of the reference model helped, since a fixed reference over-regularizes a policy that has legitimately improved. This changes the objective from "stay near SFT" to "do not move too fast", so use it knowingly. It is a different thing from InstructGPT's moving average of the policy's weights (decay 0.992), which smooths the model being trained and leaves the reference fixed.
• Stop early, on held-out evaluations. Llama 2 trained 200 to 400 iterations and stopped based on held-out prompts. Because overoptimization is invisible in the training curves, a checkpoint schedule and an independent evaluation are part of the algorithm.

What to watch during a run

Every quantity in the training loop is computed against the proxy, so a run's dashboard has to be read for internal consistency, and then checked against something external.

Two of these deserve definitions. The approximate KL between the updated policy and the rollout policy is usually estimated on the rollout tokens with the $k_3$ estimator of §0.5, the mean of $(\rho_t - 1) - \log\rho_t$ with $\rho_t = \pi_\theta/\pi_{\text{old}}$, which is nonnegative and low-variance when the policies are close. The explained variance of the critic is $1 - \operatorname{Var}(\hat R - V)/\operatorname{Var}(\hat R)$ over real tokens, 1 for a perfect critic and 0 for one no better than predicting the mean.

import numpy as np

def ppo_diagnostics(logp_new, logp_old, values, returns, mask):
    return {"approx_kl_k1": 0.0, "approx_kl_k3": 0.0, "explained_var": 0.0}

In PyTorch these utilities are a few lines each, and every RLHF library has its own copy. Here they are on a fake minibatch, the way a trainer would compute them after each update.

The whitened advantages have mean 0 and standard deviation 1 over the real tokens, and a small update gives a $k_3$ KL around $10^{-3}$ nats and almost nothing clipped, what a healthy first minibatch should look like.

The simulator below runs a small RLHF loop live, in the spirit of §16.4's NumPy toy, but with a reward model that has a length bias the true reward does not share. Watch the proxy score, the true score, the KL, and the length together, the way you would watch a real dashboard.

Evaluating the result

A trained policy has to be judged by something other than the reward model it was trained against. Five kinds of evidence are standard, and a serious report includes several.

Head-to-head win rates. Show people (or an LLM judge, with its position and length biases handled as in §14.7) responses from the new model and a baseline, usually the SFT model, and report the win rate with its error bar, a paired one when both models answer the same prompts (§0.11). InstructGPT's 85 ± 3% against GPT-3 is this kind of number. Judges prefer longer answers, so raw win rates reward length. Length-controlled win rates, previewed in §14.7, fit a model of the judge's preference that includes the length difference as a feature and then report the predicted preference with that feature set to zero. Length control raised AlpacaEval's Spearman correlation with Chatbot Arena rankings from 0.94 to 0.98 (Dubois et al. 2024).

The reward-KL frontier. Plot an independent measure of quality (a held-out reward model, a judge's win rate) against the KL from the reference, for several checkpoints or several $\beta$ values. One method beats another only if it gets more quality at the same KL. Comparing at different KLs mixes up a better optimizer with one that only moved further, and the DPO paper compared methods exactly this way (Rafailov et al. 2023).

Capability regressions. Run the standard benchmarks the SFT model was good at, to measure the alignment tax. InstructGPT found regressions on several public datasets and reduced them with PPO-ptx. Bai et al. (2022) found severe alignment taxes for small models, while their 13B and 52B RLHF models did better than the base models on zero-shot evaluations and about the same on few-shot ones.

Side effects. RLHF reduces output diversity compared with SFT, while generalizing better to new inputs (Kirk et al. 2023). It can hurt calibration, as the GPT-4 report showed for multiple-choice confidence after post-training (OpenAI 2023). And it can shift refusal rates in either direction, so measure both harmful compliance and over-refusal (§19.4).

Reading samples. Every number above is an average, and reward hacks are usually obvious once you read twenty responses from the final checkpoint next to the SFT model's.

The judge prefers the new model 71% of the time, but most of that is length. With the length term removed the estimate falls to 61%, close to the 57% that the quality difference alone implies (the rest is sampling noise in 800 judgments). This is a simplified version of the AlpacaEval estimator, which also models how hard each instruction is.

import numpy as np

def lc_win_rate(prefers_model, len_model, len_base, iters=50):
    raw = float(np.mean(prefers_model))
    return raw, raw                      # placeholder: no length control

How RLHF fits in a modern post-training stack

RLHF with PPO was the whole of post-training in 2022. Today it is one tool among several, and published stacks combine them differently.

Four ideas recur.

Rejection sampling fine-tuning (also called best-of-$n$ distillation or expert iteration) samples $K$ responses per prompt, keeps the best by the reward model, and fine-tunes on the winners with the ordinary SFT loss. It uses the reward model without the RL machinery, is stable, and captures much of best-of-$n$'s gain (§16.5). Llama 2 used it before PPO, and Llama 3 made it a core stage. WebGPT had earlier found that best-of-64 against a reward model beat its RL policy, which it attributed partly to the extra inference compute and to the reward model being more robust to rejection sampling than to RL (Nakano et al. 2021).

Direct preference optimization skips the reward model and the RL loop and trains the policy on preference pairs with a classification loss derived from the same KL-regularized objective (§17.2). It needs two models instead of four and no generation during training, which is why Llama 3 chose it, reporting that it needed less compute for large models and performed better. The evidence on quality is mixed. Ivison et al. (2024) found PPO beat DPO by up to 2.5% on math and 1.2% in general domains, with the preference data mattering more than either algorithm. Xu et al. (2024) found that well-tuned PPO beat DPO and other methods in every testbed they tried, from dialogue to competitive code. The trade-offs between online and offline preference learning are the subject of §17.5.

RL with verifiable rewards replaces the reward model with a program wherever correctness can be checked, which removes the learned proxy and with it most of §16.5's problems (and introduces new ones, the verifier's bugs). It powers reasoning models, usually with GRPO or RLOO, which drop the value model by using other samples of the same prompt as the baseline (§18.4). The same simplification works with a learned reward too. Ahmadian et al. (2024) found that REINFORCE-style methods, including a leave-one-out baseline, outperformed PPO for RLHF on preference reward models while needing no critic. They argue that a policy starting from a strong SFT model needs far less of PPO's variance control than an agent learning a game from scratch. Preference reward models remain for everything that has no checker, such as tone, helpfulness, and safety.

AI feedback scales the preference data itself. Constitutional AI generates harmlessness preferences from written principles (§19.2), and LLM judges label preferences cheaply (§16.2).

Read more

• Huang et al. (2024). The N+ Implementation Details of RLHF with PPO: A Case Study on TL;DR Summarization. The most complete list of what has to be right, with hyperparameters and ablations.
• Zheng et al. (2023). Secrets of RLHF in Large Language Models Part I: PPO. Which PPO components matter for stability in RLHF, and the PPO-max recipe built from them.
• Xu et al. (2024). Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study. Advantage normalization, large batches, and a moving reference, and evidence that tuned PPO wins.
• Ivison et al. (2024). Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedback. The relative importance of data, algorithm, reward model, and prompts.
• Ahmadian et al. (2024). Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs. Why the critic and much of PPO's machinery can be dropped for RLHF.
• Dubois et al. (2024). Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators. The regression behind length-controlled win rates.
• Lambert et al. (2024). Tulu 3: Pushing Frontiers in Open Language Model Post-Training. A fully open modern stack of SFT, DPO, and RL with verifiable rewards.
• Lambert (2025). Reinforcement Learning from Human Feedback. The book-length treatment of everything in this chapter, kept current with the field.

Watch next

▶

▶