How GRPO Trains DeepSeek's Reasoning LLMs

DeepSeek-R1-Zero learned to reason without a single supervised example. No demonstration data, no cold start. It began as a base model, and reinforcement learning alone taught it to work through hard math and hold a chain of thought together. The method behind it is GRPO, and how it works matters more than the headline.

The engine behind that result is GRPO, or Group Relative Policy Optimization, a reinforcement learning method for LLM training. It is built for the exact problem reasoning models create: very long outputs that make the usual critic network expensive and unstable. GRPO removes that critic and replaces it with a simpler idea that scales.

This guide walks the actual pipeline that produced the DeepSeek-R1 family, with benchmark numbers taken from the paper and runnable code. You will see why a standard RL setup struggled at reasoning scale, what the rule-based rewards were really doing, and how the four staged runs behind R1 differ from R1-Zero’s single one.

This is the case study, not the algorithm explainer. The advantage math, the clipped objective, and the choice between GRPO, PPO, and DPO are worked from first principles in What Is GRPO in LLM Reinforcement Learning?; the code below is the reward side that was specific to R1.

By the end you will understand the mechanics well enough to reproduce the core loop on a small model, and you will know where the method still has sharp edges. Reasoning training here comes down to a reward signal the model cannot easily cheat, applied with an optimizer that fits in memory.

TL;DR

• DeepSeek-R1-Zero reached contest-level math from a base model with no supervised fine-tuning, using GRPO and two rule-based rewards it could not learn to fake.
• The method was introduced in DeepSeekMath and then used to train the R1 reasoning family in the DeepSeek-R1 paper; it did not first appear in R1.
• DeepSeek-R1-Zero used pure reinforcement learning on the base model with rule-based accuracy and format rewards, no supervised cold start and no neural reward model.
• Verified results: R1-Zero hit 71.0% pass@1 on AIME 2024 and 95.9% on MATH-500; R1 reached 79.8% and 97.3% after a small SFT seed and multi-stage RL.
• GRPO carries two normalization biases; Dr. GRPO removes the length-normalization and standard-deviation terms to correct them.

Why Did DeepSeek Need a New RL Method for Reasoning?

The standard alignment recipe is PPO, which trains a separate value network, or critic, alongside the policy. The critic predicts how good a partial generation is so the algorithm can estimate advantage. That design works well on short responses, but reasoning changes the shape of the problem in a way the critic handles poorly.

Reasoning outputs are long. A model that thinks out loud can emit thousands of tokens before answering. Training a critic to score every step of a long chain of thought roughly doubles memory and adds a second model that is hard to fit and stabilize. At reasoning scale, the critic becomes the bottleneck rather than the help.

The memory cost is not abstract. The critic is typically the same size as the policy, so training it means holding a second full model, its gradients, and optimizer state. On long reasoning traces that second model competes for the memory the long sequences already strain, which is why teams hit a wall scaling PPO to chain-of-thought lengths.

GRPO answers this by removing the critic entirely. Instead of predicting a baseline with a learned value network, it samples a whole group of completions for the same prompt and uses the group itself as the baseline. Each completion is judged relative to its peers, which is where the name group relative comes from.

No critic is trained. GRPO still keeps a frozen reference model for the KL term, but it carries no gradients or optimizer state, so it costs far less than PPO’s trainable critic.

Getting the origin right matters, because the popular telling is often wrong. GRPO was introduced in the DeepSeekMath paper, well before the reasoning models. It was then applied to train the reasoning family described in the DeepSeek-R1 paper.

The benchmark jumps people credit to R1 came from applying an existing method to a new training recipe.

GRPO sits at the reinforcement learning stage of a larger pipeline. A base model is pretrained, often fine-tuned on demonstrations, and only then optimized against a preference or correctness signal. R1-Zero is notable precisely because it skipped the middle step, going straight from a base model to reinforcement learning without any supervised demonstrations in between.

How DeepSeek Ran the GRPO Loop

The loop is short once the critic is gone. For each prompt, sample a group of completions, usually eight to sixty-four. Score each with a reward function. Standardize those rewards within the group to get each completion’s advantage. Update the policy with PPO’s clipped objective. Apply a KL penalty in the loss. The explainer derives each of those steps; what follows is what DeepSeek put into them.

One detail is easy to get wrong. The KL penalty in GRPO lives in the loss function, not folded into the reward, which is how the DeepSeekMath paper specifies it. Keeping it in the loss separates the correctness signal from the stay-close-to-reference signal, which makes the reward easier to reason about and debug.

The part worth writing out is not the advantage math, it is R1-Zero’s two reward rules. Both are code, and both are short enough to read in full.

import re

THINK = re.compile(r"<think>.*?</think>", re.S)
BOXED = re.compile(r"\\boxed\{([^{}]*)\}")

def accuracy_reward(completions, gold, **kwargs):
    """R1-Zero's accuracy rule: the boxed final answer must match the gold one."""
    out = []
    for c, g in zip(completions, gold):
        m = BOXED.findall(c)
        out.append(1.0 if m and m[-1].strip() == g.strip() else 0.0)
    return out

def format_reward(completions, **kwargs):
    """R1-Zero's format rule: the reasoning must sit inside <think> tags."""
    return [0.5 if THINK.search(c) else 0.0 for c in completions]

comps = [
    "<think>2+3=5</think>\\boxed{5}",   # right answer, right format
    "<think>2+3=6</think>\\boxed{6}",   # wrong answer, right format
    "the answer is \\boxed{5}",          # right answer, no think tags
]
assert accuracy_reward(comps, ["5", "5", "5"]) == [1.0, 0.0, 1.0]
assert format_reward(comps) == [0.5, 0.5, 0.0]

Notice what neither function contains: a model. There is nothing to over-optimize, because a regex either matches or it does not. Notice too that format is weighted below correctness, so a well-formatted wrong answer still loses to a badly formatted right one.

The TRL library gives you a trainer that runs the full loop. You pass a policy model, the reward functions from above, and a config that sets the group size through num_generations. Your prompt set needs a gold column, which TRL forwards to accuracy_reward as a keyword argument.

from trl import GRPOTrainer, GRPOConfig
from datasets import load_dataset

# your own math prompts, with a `gold` column holding the reference answers
dataset = load_dataset("your/math-prompts-with-gold", split="train")

trainer = GRPOTrainer(
    model="Qwen/Qwen2.5-0.5B-Instruct",
    reward_funcs=[accuracy_reward, format_reward],   # summed by the trainer
    args=GRPOConfig(output_dir="grpo-r1-style", num_generations=8),
    train_dataset=dataset,
)
trainer.train()

Group size is the main knob, set here through num_generations. A larger group gives a steadier baseline, because the mean and standard deviation are estimated from more samples, but every extra completion is another full generation to run. The DeepSeekMath paper sampled 64 completions per prompt; TRL defaults to 8, and public recipes commonly run somewhere between.

One practical constraint comes from TRL: the effective training batch has to be divisible by num_generations, so pick the group size and the batch size together rather than tuning either in isolation.

That list of two reward functions is R1-Zero’s whole reward system. Each function scores the batch on its own and the trainer sums the results, so the accuracy check and the format check sit side by side with no neural reward model anywhere between them.

The DeepSeek-R1-Zero Recipe: Pure RL, No SFT

R1-Zero is the surprising part. DeepSeek applied reinforcement learning directly to the base model with no supervised fine-tuning cold start. There were no worked solutions to imitate. The model had to discover a way to reach correct answers using only the reward it received for getting them right and formatting them properly.

The rewards were rule-based, not learned. An accuracy reward checked whether the final answer was correct, verified programmatically against a known solution. A format reward checked whether the model wrapped its thinking and its answer in the required structure. There was no neural reward model scoring quality, which is the design choice that made the run stable.

The paper is explicit about why. “We do not apply the outcome or process neural reward model in developing DeepSeek-R1-Zero,” it says, “because we find that the neural reward model may suffer from reward hacking in the large-scale reinforcement learning process.” Retraining a reward model also costs resources and complicates the pipeline. Both rules are code, so a wrong answer or a broken format scores zero no matter how convincing the prose looks.

The reason to prefer rules over a learned model is reward hacking. At large RL scale, a neural reward model becomes a target the policy learns to exploit, finding high-scoring outputs that are not actually good. Deterministic checks resist that failure mode. A math answer either equals the gold answer or it does not, so there is little to game.

With only correctness and format signals, the model began producing longer chains of thought and started to re-check its own work. That self-verification was never supervised. The reward scored only correctness and format; longer, self-checking reasoning turned out to be an effective route to more correct answers, which is what later verifiable-reward pipelines built on.

Table 1: R1-Zero vs R1 pipeline

That second column expands into four stages. DeepSeek-R1 turns R1-Zero’s single RL run into a staged pipeline:

1. Cold-start SFT on thousands of long chain-of-thought examples, giving the base model a readable starting point.
2. Reasoning-oriented GRPO with an added language-consistency reward to curb the language mixing R1-Zero showed.
3. Rejection-sampling SFT on the RL checkpoint, building roughly 800k fresh examples: about 600k reasoning samples kept only where the response was correct, plus about 200k non-reasoning samples for writing, factual QA, and translation.
4. A final RL stage across all prompt types. Reasoning prompts keep the rule-based rewards; for general helpfulness and harmlessness data DeepSeek does “resort to reward models to capture human preferences.” R1, unlike R1-Zero, is therefore not reward-model-free end to end.

Verified DeepSeek-R1 Benchmark Results

Numbers here are limited to the ones reported in the DeepSeek-R1 paper, with no rounding up. DeepSeek-R1-Zero reached 71.0% pass@1 on AIME 2024 and 95.9% on MATH-500. That is a base model, trained with pure reinforcement learning and rule-based rewards, competing on contest math without ever seeing a supervised solution.

DeepSeek-R1 improved on that with 79.8% pass@1 on AIME 2024 and 97.3% on MATH-500. That is 8.8 points on AIME 2024 and 1.4 on MATH-500. The gain came from adding a small supervised seed for readability and running multiple stages of reinforcement learning, not from a different reasoning engine. The core capability was already present in R1-Zero.

The way to read the jump matters. The cold-start SFT and the staged RL bought readability, cleaner language, and single-digit gains in accuracy. They did not create the reasoning; that came from the reinforcement learning stage. The lesson for practitioners is that RL supplied the capability and the supervised polish made it usable.

One caution on reading any of these figures. A pass@1 score depends on the sampling temperature, the number of samples drawn, and the answer parser used to judge correctness. Reproducing the numbers means pinning that harness down, which is the practical side of evaluating DeepSeek models rather than trusting a single reported value.

Table 2: Verified reasoning benchmarks

Known Limitations and the Dr. GRPO Correction

GRPO is not without bias, and the honest version of this story names two. The first is a response-level length bias from dividing each completion’s loss by its own token count. The effect runs in opposite directions depending on the sign of the advantage: correct completions get larger updates when they are short, while incorrect completions are penalized less when they are long. The second half is what drives the drift, so the policy slowly learns that padding out a wrong answer is cheap.

The second is a difficulty bias from dividing by the group standard deviation. Questions where every sample scores almost the same get a tiny standard deviation, which inflates their advantages and over-weights them against questions with mixed outcomes. Both effects are easy to miss unless you track output length and per-question reward spread alongside the headline reward.

A 2025 refinement called Dr. GRPO, from Liu et al. at Sea AI Lab with the National University of Singapore and Singapore Management University (arXiv:2503.20783), targets exactly these. It removes the length-normalization term, which fixes the verbosity bias, and the standard-deviation term, which fixes the difficulty bias, keeping the group-relative baseline while dropping the two parts that skewed training. If you are building on GRPO today, it is the current adjustment worth knowing.

The deeper lesson sits underneath both. The rule-based reward is what kept R1-Zero honest through all of this. A weak or learnable verifier would have been gamed long before any length bias mattered. Trustworthy rewards are the precondition for stable reinforcement learning, which is the whole premise of building verifiable reward pipelines.

Evaluating a GRPO-Trained Reasoning Model

A trained reasoning model still needs evaluation beyond its training reward. The reward answered a narrow question, whether the final math answer was correct. Deployment asks broader ones: does the model hallucinate on open-ended prompts, is the visible chain of thought faithful to the actual computation, and does it complete real tasks rather than contest problems.

The methods here are the same ones used for any post-training checkpoint. Scoring a reasoning model against held-out prompts overlaps heavily with evaluating fine-tuned reasoning models, and once the model is wrapped in tools or a longer workflow it becomes a question of evaluating reasoning agents end to end.

That gap is why teams add an evaluation and observability layer after training. The training signal optimizes for verifiable correctness on a fixed distribution, while production traffic is wider and messier.

Scoring reasoning outputs and tracing runs are documented in the Future AGI evaluation docs and Observe docs. If you want the checks to run inside your own training loop rather than against a service, the Apache-2.0 Agent Learning Kit (pip install ai-evaluation) carries 72 local metrics that execute in-process with no API calls, and the tracing side is the Apache-2.0 traceAI OpenTelemetry instrumentation.

What DeepSeek Proved About RL and Reasoning

Back to the opening claim: a base model learned to reason with no supervised examples. The result holds because the reward was hard to fake. Pure reinforcement learning, paired with a rule-based signal the model could not easily cheat, was enough to induce chains of thought, self-checking, and contest-level math from scratch.

For practitioners, three transferable lessons survive the specifics. Group-relative advantage gives you a critic-free update that fits long outputs and cheaper hardware. Rule-based rewards resist hacking in a way learned reward models do not. Staged RL, with a light supervised seed, turns raw capability into readable output without diluting it.

None of these lessons depend on DeepSeek’s scale. The same critic-free update and rule-based reward run on a half-billion-parameter model on a single GPU, part of why the recipe spread so fast once the paper landed. A reward the model could not easily game did the heavy lifting, on the same modest hardware a single research box already has.

Two pages pick up where this case study ends, and neither repeats it. What Is GRPO in LLM Reinforcement Learning? derives the algorithm from first principles, with the advantage math, the clipped objective, and the GRPO-versus-PPO-versus-DPO decision. Verifiable reward pipelines goes deeper on the reward side than the two rules above, covering math equivalence checking, sandboxed code execution, and the ways a verifier gets gamed.