How to Build Verifiable Reward Pipelines for RLVR Training

The reward is the whole game in reinforcement learning. Get it wrong and the model optimizes your mistake with perfect discipline. RLVR promises a reward you can trust because a program computes it, but only if you build the verifier carefully enough that the model cannot quietly cheat its way to a high score.

In rlvr reinforcement learning verifiable rewards, the reward signal comes from a program that checks whether an answer is correct, not from a model that learned to imitate human preference. A math answer is verified for equivalence, a code answer is run against tests, and the score falls out of the result.

That single design choice changes the economics of training. A deterministic check costs almost nothing to run, stays consistent across millions of samples, and cannot be flattered the way a learned reward model can. It is why verifiable rewards now sit under most serious math, code, and reasoning work.

Most explainers stop at that definition. This guide keeps going into the code. You will catalogue the five verifier shapes, build the two that carry the most weight, wire one into a GRPO trainer, and then walk the failure modes that quietly wreck RLVR runs, the ones a definition never warns you about.

The mechanics assume you know roughly how group-relative RL updates a policy. If that part is fuzzy, the companion explainer on what GRPO is covers the optimizer side; here the focus stays on the reward that feeds it.

TL;DR

• RLVR replaces a learned reward model with a deterministic program that verifies correctness, such as a math equivalence check or sandboxed code execution, so the reward is cheap to compute and hard to reward-hack.
• Most tasks map to one of five verifier types: math or symbolic equivalence, sandboxed code execution against tests, string match, format or schema validation, and constraint checks.
• Build a math verifier with the math_verify library, whose parse and verify calls handle LaTeX, fractions, and numeric tolerance so equivalent forms all score as correct.
• A code reward must run untrusted model output inside a locked-down sandbox with no network, hard CPU, memory, and time caps, and an unprivileged user; a subprocess timeout alone is not isolation.
• Three failure modes recur: verifier gaming, false positives from a weak check, and spurious-reward gains on some base models like Qwen, so validate the verifier as its own artifact and across model families.

What Is RLVR, Reinforcement Learning With Verifiable Rewards?

RLVR stands for reinforcement learning with verifiable rewards, and the verifiable part is the whole idea. Every reward is produced by a deterministic function you can read, test, and trust, rather than a neural network that approximates what good looks like. The check returns a clean signal, often a simple pass or fail.

Contrast it with preference-based methods. RLHF trains a reward model on human comparisons, and RLAIF does the same with an AI labeler standing in for the human. Both learn a soft, gameable notion of quality. RLVR replaces that learned model with a rule, so the reward is exactly as sound as the rule you wrote, no more and no less.

Verifiable does not have to mean binary. A verifier can return a graded score as easily as a pass or fail: the fraction of unit tests a program passes, or a partial-credit rule for a multi-step answer. Binary rewards are simplest and most common, but a graded signal carries more information per sample when the task allows it.

The name comes from Ai2’s Tülu 3 paper, which introduces “a novel method we call Reinforcement Learning with Verifiable Rewards (RLVR)” and uses it to push math and instruction following on open models. The lineage runs straight into the reasoning systems that came after, where a correctness signal a model cannot fake became the backbone of large-scale RL.

The cost gap is not marginal. A learned reward model is a second network you serve on every step, adding latency and GPU memory, while a verifier is a function call that returns in milliseconds. At the sample volumes RL burns through, that difference often decides whether a run is affordable at all.

Because the check is cheap and hard to hack, it scales to the sample counts RL needs without a reward model in the loop. That is the same property the DeepSeek-R1 training story leaned on, and it is why RLVR and GRPO are so often paired in practice.

Table 1: Reward sources compared

Which Verifier Types Can You Build?

A verifier is just a function from an answer to a score, and only a few shapes cover most tasks. Picking the right one is mostly about what the correct answer looks like: a number, a program, a canonical string, a structure, or a rule that must hold. Each maps to a verifier you can write in a few lines.

• Math or symbolic equivalence. Use when answers are numbers or expressions and 1/2, 0.5, and 0.50 must all count as correct.
• Code execution against tests. Use when the output is a program and passing the tests is the definition of right.
• Exact or normalized string match. Use when there is one canonical answer and light normalization handles case or whitespace.
• Format or schema validation. Use when structure matters, such as valid JSON or the presence of think and answer tags.
• Constraint checkers. Use when rules must hold, such as a length cap, banned tokens, or correct units.

Production RLVR rarely uses just one. A common pattern pairs a correctness verifier with a format check and sums their scores, so the model has to be both right and well-formed to earn full reward. The next two sections build the verifiers that carry the most weight.

When you combine verifiers, the weights matter. If a format check returns the same magnitude as a correctness check, the model can trade one for the other, so most setups keep format as a small bonus or a gate and let correctness dominate. Sum the components, but size them so the total still means what you think it means.

Building a math verifier with verifiable rewards

Math looks easy to check and is not. A model that answers 0.5 to a problem whose gold answer is written as one half is correct, but a naive string comparison marks it wrong. The same trap hits fractions against decimals, unsimplified expressions, and sets written in a different order. String equality is the wrong tool.

The fix is symbolic and numeric equivalence rather than string equality. Hugging Face’s Math-Verify library parses both answers into expressions and checks whether they are mathematically the same, handling LaTeX, fractions, and numeric tolerance so equivalent forms all score as correct. Note the naming split that trips people up: you pip install math-verify with a hyphen, then import math_verify with an underscore.

from math_verify import parse, verify

# parse gold and model answer, then check symbolic/numeric equivalence
gold = parse("$\\frac{1}{2}$")
model_answer = parse("0.5")

print(verify(gold, model_answer))   # True  -> verify(gold, target) order matters

The argument order matters: verify takes the gold answer first and the model answer second, because some checks are asymmetric. Parsing both sides is what lets a single decimal match a LaTeX fraction, which is exactly the robustness a math reward needs before it is allowed to drive training.

Numeric tolerance is doing quiet work here too. Floating-point answers rarely match to the last digit, so the parser compares within a small tolerance rather than demanding exact equality. The same machinery handles sets, matrices, and intervals, which is why a general math verifier beats a pile of hand-written special cases.

One caution: the verifier is only as good as the answer extraction in front of it. If your prompt lets the model bury the final answer in prose, you can parse the wrong span and score a correct solution as zero. Most recipes require a delimiter, such as a boxed answer, so extraction stays unambiguous.

Treat the verifier’s output as the reward directly. A correct answer returns True, which you map to 1.0, and everything else scores 0.0. That binary signal is enough for GRPO to separate the winning completions from the losing ones inside each sampled group.

Building a code-execution reward safely

A code reward runs the model’s program against unit tests and scores one if every test passes. The idea is simple; doing it safely is not. You are about to execute code that an unaligned model wrote, at scale, and some fraction of it will be hostile or simply destructive by accident.

Security is the whole section, so do not skim it. Untrusted output must run isolated: no network access, hard CPU, memory, and wall-clock limits, an unprivileged user, and an ephemeral filesystem that is discarded after each run. A subprocess timeout alone is not isolation; it stops an infinite loop, not a network call or a file write.

The verifier below shows the reward logic and marks the boundary clearly. The subprocess timeout guards against loops, but the comment is explicit that the whole function must sit inside a locked-down sandbox: a container with dropped capabilities, a microVM, or a hosted execution service.

The wider set of controls for running model-written code sits in the guardrails guide.

import os, subprocess, sys, tempfile

def code_reward(completions, tests, **kwargs):
    """1.0 if the generated code passes its test, else 0.0.
    Run this ONLY inside a locked-down sandbox (no network, cpu/mem/time
    caps, unprivileged user). A subprocess timeout alone is not isolation."""
    rewards = []
    for code, test in zip(completions, tests):
        with tempfile.NamedTemporaryFile("w", suffix=".py", delete=False) as f:
            f.write(code + "\n" + test)
            path = f.name
        try:
            proc = subprocess.run(
                [sys.executable, path], capture_output=True, timeout=10
            )
            rewards.append(1.0 if proc.returncode == 0 else 0.0)
        except (subprocess.TimeoutExpired, OSError):
            rewards.append(0.0)
        finally:
            os.unlink(path)
    return rewards

Two details in there are load-bearing. Use sys.executable, not the string "python", because plenty of images ship only python3 and a bare "python" raises FileNotFoundError that takes down the whole training step instead of scoring the completion zero. And catch OSError alongside the timeout for the same reason: a reward function that raises is far worse than one that returns 0.0.

Wiring the verifier into training is the easy part. TRL’s GRPOTrainer takes your reward function directly, and any extra columns in the dataset arrive as keyword arguments, so a tests column lands in the function as tests. The function returns one float per completion and the trainer handles the sampling and the update.

from trl import GRPOTrainer, GRPOConfig
from datasets import load_dataset

dataset = load_dataset("your/prompts-with-tests", split="train")  # has a `tests` column

trainer = GRPOTrainer(
    model="Qwen/Qwen2.5-0.5B-Instruct",
    reward_funcs=code_reward,          # dataset columns arrive as kwargs (tests=...)
    args=GRPOConfig(output_dir="rlvr-code", num_generations=8),
    train_dataset=dataset,
)
trainer.train()

That keyword-argument contract is worth remembering. It is how a single reward function reaches the per-example data it needs, and how you would add a second signal, like a format flag, without touching the trainer. Keep the function pure and fast, since it runs on every completion in every group.

Two refinements pay off once the basics work. A graded code reward that returns the fraction of tests passed gives a smoother signal than all-or-nothing, which helps early in training when few completions pass everything. And pin the test environment, since a flaky or nondeterministic test injects noise the optimizer will happily chase.

How Do RLVR Reward Pipelines Fail?

A sound-looking verifier can still be wrong, and RL will find the gap faster than you will. This is the section most build guides skip, and it is where real runs fail. Three failure modes recur, each documented in recent work and each with a concrete mitigation you can apply today.

The first is verifier gaming. Given enough optimization pressure, a model learns to satisfy the check without solving the task, exploiting whatever shortcut the verifier allows, documented by Helff et al. in “LLMs Gaming Verifiers: RLVR can Lead to Reward Hacking” (arXiv:2604.15149). A code test that only checks a return value invites hard-coded outputs, and a regex format reward invites answers shaped correctly but empty of content.

The second is false positives from a weak verifier. If the check wrongly accepts bad output, the model optimizes straight into that bug. Fuzzing the verifier with adversarial completions surfaces these accepted-but-wrong cases before training does, the approach Ray takes in “Before the Model Learns the Bug: Fuzzing RLVR Verifiers” (arXiv:2606.01066), turning a silent leak into a test you can actually fix.

The third is the strangest. On some base models, notably parts of the Qwen family, even random or spurious rewards raise benchmark scores, because the update amplifies behavior the model already had. Shao et al., in “Spurious Rewards: Rethinking Training Signals in RLVR” (arXiv:2506.10947), report that random rewards lift Qwen2.5-Math-7B on MATH-500 by 21.4 points against 29.1 from ground-truth rewards, and that the same spurious rewards fail to produce gains on Llama3 or OLMo2. A gain on a single model therefore does not prove your verifier is sound, only that something moved.

The mitigations stack. Test the verifier as its own artifact, add adversarial and false-positive checks, combine several verifiers so one covers another’s blind spot, validate gains on a model family that does not show the spurious-reward effect, and log reward distributions so a collapse or a runaway is visible early.

Logging deserves a specific look, since it is the cheapest guardrail you have. A healthy run shows a spread of rewards inside each group that slowly shifts upward; a run where every completion suddenly scores the maximum is usually gaming or a false positive, not success. Watch the shape of the distribution, not only its mean.

Table 2: RLVR failure modes and mitigations

Treating your verifier as an evaluation grader with Future AGI

Step back and look at what a verifier actually is. It is a rule that takes an output and returns a pass, a fail, or a score. That is the exact definition of an evaluation grader.

Evaluation as a grading discipline is a larger practice than training rewards alone, and the verifier you built for training is also the eval you need around it.

Future AGI custom evals are built on that primitive.

You write the grading rule once, choose a deterministic rule-based check or an LLM judge, point it at your dataset columns, and set a pass or fail threshold. The same logic that rewards a training completion can now grade a held-out set or gate a deploy.

The first payoff attacks the failure modes above directly. Run your verifier as an offline eval over known-good and known-bad answers and measure how often it fires a false positive. Name the evaluators for what they check, a correctness check and a format-adherence check, so a regression stays legible to whoever reads the report.

The second is keeping the signal honest over time. Observe attaches those scores to traces and alerts when a monitored metric crosses a threshold, and running the same graders as a CI gate blocks a training run when the reward pipeline regresses, the same way a failing test blocks a bad deploy. A verifier is a rule, so it does not decay the way a learned scorer does; if you are running RLHF with a trained reward model alongside this, the distribution tests for that failure are in reward model drift in LLMs, which is a different problem from anything on this page.

The third matters most for code rewards. Model-written code should be screened before it reaches the sandbox, not only contained inside it. The Apache-2.0 Agent Learning Kit (pip install ai-evaluation) ships scanners for code injection, secrets, malicious URLs, jailbreaks, and PII that run in-process in under 10ms with zero API calls, which is fast enough to sit on every completion in every group without changing the economics of the run. It also carries 72 local metrics, so the string-match, schema, and constraint verifiers from the catalogue above are largely already written.

For the serving side, Protect runs 28 guardrail checks inline, ten first-party and eighteen provider-backed, each configurable to a pre, post, or both stage and to block, warn, mask, or log. The tool-permissions and MCP-security checks are the relevant ones when the trained model later calls tools rather than just emitting an answer.

Shipping RLVR you can trust

You now have the pieces: five verifier shapes with the two hardest ones built, a safe way to run code rewards, the wiring into GRPO, and the failure modes that decide whether any of it holds up. The through-line is that the reward deserves as much engineering as the policy it trains, because a weak verifier caps everything downstream.

Build it in order. Start with one sound verifier, a math check or a sandboxed code reward, and prove it works. Validate that verifier as an offline eval so you trust its pass and fail before it ever shapes a policy. Only then combine verifiers for correctness plus format, where a second check covers the first one’s gaps.

It also helps to place RLVR in the wider tuning picture. It is one branch of the wider fine-tuning picture, and the models it produces still need evaluation as RL-tuned systems before they ship, since a strong training reward is not the same as strong behavior on live traffic.

The single highest-leverage habit is to treat the verifier as a first-class artifact and harden it like one. The evaluation docs are the place to turn a training reward into a grader you can test, gate on, and trust long after the run finishes.