What Is GRPO in LLM Reinforcement Learning?

You have a model, a reward you can compute, and a GPU budget that will not fit a second network. That is the exact corner GRPO was built for. It throws out the critic that PPO leans on and reads quality from a handful of answers compared against each other, cheaper and more stable than that sounds.

GRPO, short for Group Relative Policy Optimization, is a critic-free reinforcement learning method for language models. In grpo reinforcement learning llm training, the model samples several answers to the same prompt, scores each one, and shifts probability toward the answers that beat the group average. There is no separate value network anywhere in the loop.

That one change is what makes it practical. PPO, the method GRPO descends from, trains a second network called a critic to predict how good a partial answer is. On long outputs that critic doubles the memory footprint and adds a model that is fussy to stabilize. GRPO removes it and reads the baseline off the group instead.

This explainer builds the method from first principles. You will see what the acronym means, the exact advantage math in runnable code, a clean comparison against PPO and DPO, and the cases where GRPO is the wrong tool. The goal is a working mental model you can act on, not a survey of the literature.

None of it requires a research background. If you can read a short Python function and follow a five-step loop, you can follow how GRPO turns a reward signal into a better policy. The applied history and the reward engineering each have their own guide, linked where they fit.

TL;DR

• GRPO stands for Group Relative Policy Optimization, a reinforcement learning method that drops PPO’s critic and scores a group of answers per prompt.
• Each answer’s advantage is its reward minus the group mean, divided by the group standard deviation, so above-average answers get reinforced.
• It needs no value network and no learned reward model, so any reward function, including a rule-based check, can drive training.
• Use GRPO for verifiable or reasoning rewards, DPO for static preference pairs, and PPO when you need a learned value estimate with full online control.
• GRPO carries a length bias and a difficulty bias; Dr. GRPO (arXiv:2503.20783) removes both by dropping the 1/|o_i| and standard-deviation normalization terms.

What Is GRPO and What Problem Does It Solve?

GRPO expands to Group Relative Policy Optimization, and each word carries weight. Group means it works with a batch of sampled answers, not one. Relative means each answer is judged against its siblings rather than an absolute target. Policy Optimization means it is a policy-gradient method, the same family as PPO, adjusted to remove the part that hurt at scale.

The part it removes is the critic. In PPO a value network learns to predict the expected reward of a partial generation, which gives the algorithm a baseline. That network is a second model to train, hold in memory, and keep stable. On long chain-of-thought outputs it becomes the most expensive and most fragile piece of the setup.

Why fragile is worth a sentence. On a long chain of thought the critic has to guess the final reward from a half-finished answer, spreading credit across hundreds of tokens when the real signal only lands at the end. A wrong guess feeds a noisy baseline into every update, which is much of why PPO runs are finicky to stabilize.

GRPO’s insight is that you do not need an absolute value to know whether an answer was good. Sample several answers to the same prompt and ask a cheaper question: did this one beat the others? The group supplies the baseline the critic used to provide, and it costs nothing beyond the samples you already drew.

The method first appeared in the DeepSeekMath paper (arXiv:2402.03300), which trained GRPO with a learned reward model. GRPO’s now-familiar rule-based, verifiable-reward setup arrived later, with DeepSeek-R1 (arXiv:2501.12948), whose R1-Zero run drove a base model using accuracy and format checks alone.

GRPO has since become a default for reasoning and code work, anywhere a correctness signal is cheap to compute.

One boundary before we go further. This page is the algorithm: the advantage math, the update, and the choice between GRPO, PPO, and DPO. It is not the DeepSeek story. The staged R1 pipeline, the R1-Zero recipe, and the published AIME and MATH-500 numbers live in the DeepSeek-R1 case study, and the reward functions that feed a run like that live in the verifiable-reward build guide. Nothing here repeats those.

How a GRPO Update Works, Step by Step

The loop has five steps, none complicated. First, sample a group of completions for a prompt, often eight to sixteen. Second, score each with a reward function. Third, turn those rewards into group-relative advantages. Fourth, update the policy with a clipped objective. Fifth, add a KL penalty in the loss to stay near a reference model.

The third step defines the method. For each answer, subtract the group mean from its reward, then divide by the group standard deviation. An answer that scored above the group gets a positive advantage and is reinforced. One that scored below gets a negative advantage and is pushed down. The whole idea fits in one line of code.

import numpy as np

def grpo_advantages(rewards):
    rewards = np.asarray(rewards, dtype=np.float32)
    return (rewards - rewards.mean()) / (rewards.std() + 1e-8)

# self-check: symmetric rewards give symmetric advantages
adv = grpo_advantages([1.0, 0.0, 1.0, 0.0])
assert np.allclose(adv, [1.0, -1.0, 1.0, -1.0])
print("ok", adv)

Dividing by the standard deviation does real work here. Subtracting the mean already removes the absolute level of a prompt’s rewards; dividing by the spread puts every prompt on the same scale on top of that. Without it, prompts whose rewards happen to spread wider would pull harder on the gradient than prompts where the answers sat close together, regardless of which prompt had more to teach.

That same rescaling carries a cost. Questions whose sampled answers all score alike get a tiny standard deviation, so dividing by it inflates their weight in the update. That hits prompts the model already aces and prompts it always fails, at the expense of the mixed-outcome prompts where the learning signal actually lives. The Dr. GRPO section below returns to it.

Two steps borrow from PPO. The clipped objective caps how far the policy can move on any update, which stops one lucky batch from destabilizing it. The KL penalty lives in the loss rather than the reward, and keeps the trained policy from drifting too far from the reference it started as. Both are guardrails around the group-relative core.

It is worth seeing why both matter here. Because GRPO reads its advantage off a small group, one noisy batch can shove the policy hard in the wrong direction. The clip caps that shove, and the KL penalty stops the model from wandering into gibberish that happens to score well. They are what let a critic-free method stay stable.

In practice you rarely write the loop by hand. The TRL library ships a GRPOTrainer that samples the group, applies your reward function, computes the advantages, and runs the update. You supply a model, one or more reward functions, and a config that sets the group size through num_generations.

from trl import GRPOTrainer, GRPOConfig
from datasets import load_dataset

dataset = load_dataset("trl-lib/tldr", split="train")

def reward_len(completions, **kwargs):
    # toy rule-based reward: prefer ~50-word answers
    return [-abs(50 - len(c.split())) for c in completions]

trainer = GRPOTrainer(
    model="Qwen/Qwen2.5-0.5B-Instruct",
    reward_funcs=reward_len,
    args=GRPOConfig(output_dir="grpo-demo", num_generations=8),
    train_dataset=dataset,
)
trainer.train()

The num_generations value is the group size. A larger group gives a smoother advantage estimate, since the mean and standard deviation come from more samples, but each extra completion is another generation to run per prompt. TRL also requires the effective training batch to divide evenly by num_generations, so you set the two together.

Cost-conscious recipes often stay in the eight-to-sixteen range, though the original DeepSeekMath work sampled 64 per question. This toy reward prefers answers near fifty words.

The reward function follows a small contract. It takes the list of completions and returns one number per completion, and **kwargs catches any extra dataset columns a reward needs. You can pass several reward functions at once and the trainer sums their scores, which is how a correctness check and a format check combine into one signal.

GRPO vs PPO vs DPO, and When to Use Each

The three methods get lumped together as ways to tune a model on preferences, but they make different assumptions. The cleanest way to choose is to ask two questions: is your training signal already static data, and do you need a learned value estimate?

Your answers point at one method without much ambiguity. The tradeoffs mirror the broader call between evaluation and RLHF-style feedback.

Reach for DPO when you already hold a pile of preference pairs, each a chosen and a rejected answer, and you want the cheapest stable path. DPO optimizes directly on those pairs offline, with no sampling loop and no reward model. It is the low-memory option, but it cannot react to new answers the model has not generated yet.

Reach for PPO when you need full online control and a learned value estimate, usually general RLHF with a neural reward model. The critic that makes PPO expensive also makes it flexible, giving a per-token baseline that GRPO only approximates. If your reward is a subtle learned model and you can afford the memory, PPO still earns its place.

Reach for GRPO when you can compute a cheap or verifiable reward and want online RL without a critic. Math, code, and format tasks fit this exactly, because correctness is a function call away. You get the online behavior of PPO, sampling fresh answers each step, without the memory cost of a second network.

Table 1: PPO vs GRPO vs DPO

These are not permanent commitments. Teams routinely prototype with DPO on a fixed set, then move to GRPO once they can generate a reward online, and the choice between evaluation and fine-tuning often decides whether you need reinforcement learning at all. The decision is about your data and signal today, not a permanent allegiance.

When Should You Use GRPO, and When Not?

GRPO is strongest wherever a reward is cheap and trustworthy. Math with a checkable final answer, code that either passes tests or does not, and outputs that must match a strict format all give a clean signal the model cannot easily fake. In these domains you skip the reward model entirely and let a deterministic check drive learning.

A deterministic reward also resists gaming. When the check is a unit test or an exact-match comparison, the model cannot flatter a learned scorer into a high mark; it has to actually satisfy the rule. That property is why rule-based rewards anchor so much reasoning work, where a confident wrong answer earns exactly nothing.

It also shines under a tight memory budget. Dropping the critic frees the space a second model would have taken, which is what let reasoning models train on long chains of thought without exotic hardware. For teams tuning and then evaluating RL-adjusted models on modest GPUs, that saving is often the deciding factor.

GRPO is a poor fit when quality is subjective and no cheap reward exists. If good means helpful, tactful, or well written, there is nothing to compute, and you are back to needing a reward model or an LLM judge. GRPO does not remove that need; it pays off once you have a signal worth optimizing against.

One caveat remains. GRPO carries two normalization biases, isolated by Liu et al. in “Understanding R1-Zero-Like Training: A Critical Perspective” (arXiv:2503.20783), the paper that introduced Dr. GRPO. The first is a response-level length bias from dividing each response’s loss by its own token count, the 1/|o_i| term. The paper’s finding is asymmetric: correct responses get larger updates when they are short, while incorrect responses are penalized less when they are long, so the policy drifts toward padding out its wrong answers.

The second is a question-level difficulty bias from dividing the advantage by the group’s reward standard deviation. Questions with a low standard deviation, meaning the sampled rewards are almost all 1 or almost all 0, receive higher weight during policy updates. Dr. GRPO drops both normalizers and keeps the group-relative baseline.

Both are worth knowing about, though neither usually changes whether GRPO is the right choice for a verifiable-reward task. Later variants such as DAPO, GSPO, and CISPO push on the same objective in other ways, but the group-relative core is unchanged.

Table 2: Is GRPO the right choice?

How to Tell If Your GRPO Run Is Working

While training runs, a few signals tell you whether GRPO is behaving. Watch the mean reward per group trend upward, the reward variance stay healthy rather than collapsing to zero, and the KL to the reference stay bounded. Track completion length too: a slow climb in length with flat reward is the classic sign of length bias.

The variance signal deserves a closer look. If reward variance inside a group collapses to zero, every sampled answer scores the same, the advantage goes flat, and learning stalls even while the loss looks calm. It usually means the prompts are too easy or too hard for the current policy, and a better-matched prompt mix is the fix.

Training curves are necessary but not sufficient. A rising reward only says the model is getting better at the reward, not that it is better on real prompts. You still need to score held-out outputs for correctness, hallucination, and format adherence, on inputs the training set never contained.

That held-out scoring is a standard evaluation and observability task. Write the grading rule as a Future AGI custom eval, pick an LLM judge or a deterministic check, and run it over the checkpoint’s outputs on prompts the training set never contained.

For the checks you want in the training loop itself, the Apache-2.0 Agent Learning Kit (pip install ai-evaluation) ships 72 local metrics that run in-process with no API calls, so a per-checkpoint eval costs no tokens and adds no network hop. Trace the runs through Observe, which is built on the Apache-2.0 traceAI OpenTelemetry instrumentation.

GRPO in One Page

The whole method fits in three sentences. GRPO samples a group of answers per prompt and scores them with any reward function. It converts those scores into group-relative advantages by subtracting the group mean and dividing by the standard deviation. It updates the policy toward the winners with a clipped objective and a KL penalty, and never trains a critic.

Where you go next depends on what you need.

For the applied story of GRPO producing a state-of-the-art reasoning model, read the DeepSeek case study.

To build the reward side that keeps this training honest, the guide to verifiable reward pipelines picks up there.

For where reinforcement learning sits among tuning options, start with the fine-tuning guide.

Whichever path you take, the last mile is measurement. A model that scores well on its training reward can still fail on live traffic, which is why evaluating reasoning agents end to end matters as much as the training method that produced them.