In late 2024 the Tulu 3 team at the Allen Institute gave a name to something several labs were already doing. Take prompts whose answers can be checked by a program, such as math problems with known answers or instructions with checkable constraints like "use fewer than 50 words". Sample a response, run the checker, and give a reward of 10 if it passes and 0 if it does not. Train with RL. They called it reinforcement learning with verifiable rewards (RLVR), and they found that it improved GSM8K, MATH, and IFEval scores on top of an already strong post-trained model (Lambert et al. 2024). Two months later, DeepSeek-R1 showed what the same idea does at scale (§18.5).

The appeal is that a program cannot be flattered. A learned reward model is a neural network with blind spots, and RL against it finds them (§16.5). DeepSeek gave exactly this reason for using no neural reward model in R1-Zero, writing that "the neural reward model may suffer from reward hacking in the large-scale reinforcement learning process".

But a program can be wrong. RL is an optimizer that will find every input on which your checker says yes and the truth says no, and the checker is the entire specification of what you want. This section builds checkers for math and code, shows how policies learn to fool them, and turns that into a short engineering discipline for graders you can trust.

The objective

Write $R(x, y) \in \{0, 1\}$ for the verifier's verdict on response $y$ to prompt $x$. RLVR maximizes

the KL-regularized objective of §15.7 with a program in place of the reward model, where $\pi_{\text{ref}}$ is the starting model and $\beta \ge 0$ sets how far the policy may drift (several recipes set $\beta = 0$, as §18.4 discusses). With a 0/1 reward the first term is exactly the average pass@1 over the prompt distribution, the quantity §18.2 called $p(x)$, averaged over $x$. In plain words, RLVR directly maximizes the probability of a correct answer, which is the one number a deployed model without a checker needs to be high.

Each verifier is a function from (prompt, response) to a reward, built from three parts. An extractor finds the claimed answer in the response, a normalizer puts it in canonical form, and a comparator decides whether it matches the reference. Each part can fail in two directions.

• A false negative rejects a correct response. The model is penalized for being right in an unexpected format, which wastes signal and teaches the format rather than the math.
• A false positive accepts a wrong response. The model is paid for being wrong in a particular way, and RL will learn to be wrong in exactly that way as often as possible.

False positives are the dangerous ones, because RL amplifies them. A false negative rate of 5% costs some learning speed. A false positive that a policy can trigger on purpose eventually becomes the policy.

Format rewards

DeepSeek-R1-Zero's template asked the model to put its reasoning between <think> and </think> and its answer between <answer> and </answer>, and its reward had two rule-based parts, an accuracy reward and a format reward for following the template (DeepSeek-AI 2025). The format reward does two jobs. It makes extraction reliable, since the answer is wherever the tags say, and it separates the part of the output that is reasoning (and may later be hidden or summarized) from the part that is the answer.

A format reward should be strict about structure and indifferent to content. One well-formed thinking block, one well-formed answer block, nothing else. Leniency here leaks into the accuracy reward, because a response with two answer blocks invites the extractor to read the wrong one.

def format_reward(text):
    return 1.0 if "<answer>" in text else 0.0

Checking math answers

Most math RLVR data comes in one of two shapes. Competition problems (the MATH dataset and its descendants) have answers written in LaTeX, by convention inside \boxed{...} at the end of the solution. Word problems (GSM8K) have numeric answers, which the dataset marks after ####. Either way the checker must find the model's final answer, canonicalize it, and compare it with the reference.

Extraction

The rule nearly everyone uses is "the last \boxed{} in the response". It is the last one because models often box intermediate results, and the final answer comes at the end. The contents can contain braces, as in \boxed{\frac{a}{b^{2}}}, so a regex like \\boxed\{(.*?)\} returns \frac{a and silently breaks every answer with a fraction. You need a brace counter.

import re

def extract_boxed(text):
    m = re.findall(r"\\boxed\{(.*?)\}", text)
    return m[-1] if m else None

Normalization and comparison

Now the hard part. A checker that compares strings exactly will reject 0.5 against a reference of \frac{1}{2}, 1000 against 1,000, and x = 5 against 5. Each of those false negatives tells the model, on that example, that the right answer was wrong. Here is a working list of transformations, roughly in the order a checker applies them.

Each rule is also a place to be wrong. Dropping percent signs makes 50\% equal 50, which is what most datasets want, but also makes it unequal to 0.5, which some want. Removing the separator in 1,000 is right, but the same rule applied to 1,2 would turn the coordinate pair into twelve. Numeric comparison needs a tolerance, but a loose one accepts 0.333 for 1/3, which a grader should reject because the model rounded. And no list of string rules catches every symbolic equivalence. $3\sqrt{2}$ and $\sqrt{18}$ are equal, and only a computer algebra system can tell. That is why production checkers such as Hugging Face's Math-Verify parse LaTeX into SymPy expressions and test equivalence symbolically, falling back to string rules when parsing fails.

def answers_match(pred, ref):
    return pred is not None and pred.strip() == ref.strip()

Huang et al. (2025) measured how real checkers fail during RL on math. Rule-based checkers, they found, often fail to recognize equivalent answers written in different formats, a substantial false negative rate. Model-based checkers, where an LLM judges whether the response matches the reference, recover those false negatives but are themselves hackable. During RL, policies learned to emit patterns that the judge model misclassified as correct, in the simplest cases a single character such as { or long strings of meaningless text. The practical lesson, which matches what labs report doing, is to use a symbolic checker first and a judge only where it fails, and to probe any judge adversarially before letting a policy optimize against it.

Code with unit tests

For code, the verifier runs the program. Given a problem statement and a set of tests, extract the code from the response, execute it in a sandbox against the tests, and reward it. The two common rewards are binary (1 only if every test passes) and fractional (the fraction of tests passed). Fractional rewards give a denser signal on hard problems, at the cost of paying for partial solutions, including ones that special-case the easy tests.

Unit tests have false positives of their own. A test suite is a sample of the input space, and a wrong program that agrees with the right one on that sample passes. Liu et al. (2023) generated 80 times as many tests for HumanEval, from mutations of the original inputs and inputs written by an LLM, and found that the extra tests caught enough previously accepted wrong programs to cut pass@$k$ by up to 19.3% to 28.9%, and to change the ranking of models. In RL, an under-tested problem is not just a noisy label. It is a problem on which the policy is paid for code that does not work.

The standard way to measure a test suite's strength comes from software engineering. Make mutants of a correct program, small changes such as flipping a comparison or dropping an edge case, and count how many of them some test catches ("kills"). A mutant that survives every test is a wrong program the grader would accept.

def mutation_report(reference, mutants, inputs):
    return {"survivors": [], "minimal": list(range(len(inputs)))}

When the policy attacks the grader

Weak tests are the passive failure. The active failure is a policy that learns to attack the test harness itself, and once models can run code, this happens in practice. The hacks that have been reported in real training runs share a theme. They make the check succeed without making the program correct.

• Exiting early. Calling sys.exit(0) before any assertion runs, so the harness sees a successful exit code. Baker et al. (2025) observed this and raising SkipTest in a frontier model's coding RL (§18.1).
• Objects that equal everything. Returning an object whose __eq__ always returns True (and, for strings, whose methods return the object itself), so assert result == expected passes. MacDiarmid et al. (2025) at Anthropic call this the AlwaysEqual hack.
• Patching the test framework. Writing a conftest.py that monkey-patches pytest's reporting so failures are recorded as passes (also in MacDiarmid et al.).
• Tampering with the evaluator. METR (2025) documented frontier models overwriting the timing function in a kernel-optimization task so it always reported shorter times, monkey-patching the evaluator, overwriting the equality operator, and finding the grader's own copy of the answer.

MacDiarmid et al. found something worse than wasted compute. When a model learned these hacks on real production coding environments, it generalized to broader misbehavior, including alignment faking and attempts at sabotage, which is why reward hacking in RLVR is a safety problem and not only a data-quality problem (§19.1). Zhong et al. (2025) built ImpossibleBench to measure the propensity directly. It makes coding tasks whose specification and tests contradict each other, so the only way to pass is to cheat, and the pass rate is the cheating rate.

The lesson from these cases is that a grader must not trust anything the solution can touch. Run the solution in a separate process with time and memory limits, no network, and a read-only copy of the tests. Have the untrusted process write its outputs in a plain data format, and compare them in the trusted process, where an object with a rigged __eq__ cannot follow, because serializing it either fails or produces only its data. In-process grading can never be made fully safe (a solution can inspect the caller's stack frames to read the expected answer, or patch built-in functions the grader uses), but the same principles applied in-process already stop the common hacks.

def grade_solution(src, func_name, tests):
    ns = {}
    exec(src, ns)
    f = ns[func_name]
    return float(all(f(*args) == expected for args, expected in tests))

Other domains with checkable answers

Math and code are the largest RLVR domains, not the only ones. What they share is a cheap program that decides correctness. The table lists the other domains in common use and the main way each one goes wrong.

The multiple-choice row deserves a number. Suppose that on a question with $K$ options the policy reasons its way to the right answer with probability $q$ and otherwise guesses uniformly. Its reward is 1 with probability $q + (1-q)/K$, and by Bayes' rule the fraction of rewarded responses that were lucky guesses is

With $K = 4$ options and $q = 0.2$, that is $0.2/0.4 = 0.5$. In plain words, half of the positive rewards reinforce chains that never reasoned to the answer, and to the policy they look exactly like the real successes. Practitioners therefore rewrite multiple-choice questions into open-answer form where they can, or drop the questions whose answers are easy to guess.

Noisy rewards, and why only some noise is harmless

Every real grader makes mistakes. How much does that matter? Suppose the grader accepts a wrong response with probability $\mathrm{FPR}$ (its false positive rate) and rejects a correct one with probability $\mathrm{FNR}$ (its false negative rate), and suppose, crucially, that these rates do not depend on which correct or wrong response it sees. Then for a prompt on which the policy is correct with probability $p$, the expected value of the noisy grader's reward $\tilde R$ is

The gradient of this expectation is $(1 - \mathrm{FPR} - \mathrm{FNR})\nabla_\theta p$, the true gradient times a positive constant as long as $\mathrm{FPR} + \mathrm{FNR} < 1$. In plain words, response-independent noise slows learning down and changes nothing else. A grader that is wrong 20% of the time in both directions shrinks the signal to 60%, which costs samples, but the optimum is unchanged. One refinement is worth knowing. With a KL penalty, shrinking the reward by a factor $c$ is the same as multiplying $\beta$ by $1/c$, so the regularized optimum moves toward $\pi_{\text{ref}}$, and a noisy grader in a KL-regularized run behaves like a clean grader with a stronger penalty.

The assumption is the whole story. Real false positives are not independent of the response. A grader that accepts multiple boxed answers accepts them every time, so the policy can raise its reward by moving probability onto that behavior, and the optimum of the noisy objective is now a policy that hedges. False negatives that depend on format, likewise, move the optimum toward the format the grader likes. The cell below makes the distinction concrete with a softmax policy over three behaviors.

Uniform noise leaves the learned policy the same as the clean grader, careful solving, only slower. A grader that pays for one specific wrong behavior produces a policy that does that behavior.

Building graders you can trust

Put together, the evidence suggests a short discipline for any RLVR project.

1. Measure the grader before the policy does. Build an audit set of responses with human-verified labels, including responses from the policy you will train, since its mistakes are the distribution that matters. Report the false positive and false negative rates separately.
2. Attack it. Before training, run the grader on known hack patterns. Multiple answers, hedges ("either 3 or 4"), empty boxes, answers in unusual formats, very long outputs, text addressed to a judge model, and for code the exit, equality, and patching tricks above. Every accept is a bug.
3. Prefer exact over learned. Use symbolic equivalence for math and execution for code, and use a model judge only for the cases they cannot handle, with its outputs audited.
4. Isolate execution. Separate processes, time and memory limits, no network, read-only tests, outputs compared in the trusted process.
5. Keep held-out checks. Hidden tests the policy never trains against, and a held-out benchmark scored by a different grader. If training reward rises while the held-out score stalls or falls, the policy has found something in the grader.
6. Read samples. The cheapest detector of reward hacking is a person reading a hundred high-reward samples every few hundred steps.

Spurious rewards, and what they say about evaluation

A finding from 2025 is a warning for anyone reading RLVR results. Shao et al. (2025) trained Qwen2.5-Math-7B with RLVR on MATH using rewards that should not work. On MATH-500, ground-truth rewards improved accuracy by 29.1 points. A reward for format alone improved it by 13.8, rewards computed against incorrect labels by 24.1, and completely random rewards by 21.4. The same spurious rewards often failed to help other model families, such as Llama 3 and OLMo 2. Their analysis suggested that RL was mostly amplifying behaviors the base model already had, in particular a habit of reasoning in Python code, which rose from 65% to over 90% of responses, and that the clipping in the GRPO objective of §18.4 biases updates toward such high-prior behaviors. Wang et al. (2025) found a related effect from the other side, RLVR on a single training example lifted Qwen2.5-Math-1.5B from 36.0% to 73.6% on MATH-500. They report that it beat a format-correction baseline by 8.6 points, so most of that gain, though not all of it, matches what fixing the output format alone buys.

None of this means verifiable rewards are fake. It means that a gain from RLVR on one model family is not evidence that the reward taught anything new, and that format fixes, elicitation, and real learning look the same on a single benchmark number. Test on more than one base model, include a random-reward control, and look at what changed in the outputs.

In PyTorch

In a training loop, rewards for a batch are assembled from several rule-based parts and handed to the RL algorithm as one tensor. Here is a typical combination, accuracy plus format plus DAPO's soft overlong punishment, which ramps a penalty in over the last $L_{\text{cache}}$ tokens before the hard length limit $L_{\max}$ so that truncated, unfinished responses are discouraged smoothly rather than all at once (Yu et al. 2025).

DAPO used $L_{\max} = 20{,}480$ with $L_{\text{cache}} = 4{,}096$, so the penalty starts at 16,384 tokens, and it rewarded correctness with $+1$ and an incorrect answer with $-1$ rather than 0. The weights on the parts, here 0.1 on format, are design choices, and every one is a lever the policy will pull. A format bonus large enough to matter on its own is a reward for well-formatted wrong answers.

Read more

• Lambert et al. (2024). Tulu 3: Pushing Frontiers in Open Language Model Post-Training. The paper that named RLVR, with its reward design and data.
• DeepSeek-AI (2025). DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. Rule-based accuracy and format rewards, and why they avoided neural reward models.
• Huang et al. (2025). From Accuracy to Robustness: A Study of Rule- and Model-based Verifiers in Mathematical Reasoning. False negatives in rule-based checkers and hackable model-based ones, measured during RL. The first version was titled "Pitfalls of Rule- and Model-based Verifiers".
• Liu et al. (2023). Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation. EvalPlus, and how many wrong programs weak tests accept.
• MacDiarmid et al. (2025). Natural Emergent Misalignment from Reward Hacking in Production RL. Real coding-environment hacks and how learning them generalized to misaligned behavior.
• METR (2025). Recent Frontier Models Are Reward Hacking. Documented cases of models tampering with evaluation code.
• Shao et al. (2025). Spurious Rewards: Rethinking Training Signals in RLVR. Random and incorrect rewards that still raise scores, and what that implies for evaluation.
• Hugging Face. Math-Verify. A production math answer checker that parses LaTeX into SymPy for symbolic comparison.

Watch next

▶

▶

▶