GRPO Explained

GRPO, or Group Relative Policy Optimization, is a reinforcement learning algorithm for language models that makes one substitution: it deletes PPO’s critic and estimates the baseline from a group of samples drawn for the same question instead.

That one substitution is where most of what people find distinctive about GRPO comes from: the memory saving, the group size you have to choose, and the failure mode that decides whether a training batch teaches anything at all.

It is not quite everything. Clipping, the KL coefficient and the way token losses are aggregated are separate design axes, and the last of those is where most of the algorithm’s recent history has happened.

It was introduced in DeepSeekMath as “a variant of Proximal Policy Optimization (PPO), that enhances mathematical reasoning abilities while concurrently optimizing the memory usage of PPO.”

The short version: score several answers to the same question, use their average as the bar, and push the above-average ones up. No second model required.

What Problem Does GRPO Solve?

The cost of the critic.

Policy gradient methods need a baseline. Raw reward tells you what an output scored, not whether that score was good for that particular question.

An answer worth 0.6 on a question the model usually aces is a bad answer. The same 0.6 on a question it usually fails is excellent. Without a baseline, the gradient cannot tell those apart.

PPO’s answer is to train a second model, a value function, to predict the expected reward. The paper is blunt about what that costs:

“As the value function employed in PPO is typically another model of comparable size as the policy model, it brings a substantial memory and computational burden.”

So you are training two large models to improve one. The critic never ships. It exists only to produce a number that the policy gradient consumes and discards.

GRPO asks whether that number can be obtained by measurement rather than prediction.

Two adjacent subjects are covered elsewhere and not repeated here. A survey of the wider family of reasoning strategies, alongside RLHF, PPO, DPO and search-based methods, is in our guide to LLM reasoning.

How evaluation scores become the reward signal that any of these algorithms consume is covered in LLM eval versus RLHF feedback loops.

How Does GRPO Estimate the Baseline Without a Critic?

By sampling the question several times and using what came back.

The paper states the substitution directly:

“GRPO foregoes the critic model, instead estimating the baseline from group scores, significantly reducing training resources compared to Proximal Policy Optimization (PPO).”

The baseline itself is “the average reward of multiple sampled outputs, produced in response to the same question”.

The procedure is four steps.

1. Take one question and sample G outputs from the current policy.
2. Score all G with the reward function.
3. Compute the mean and standard deviation of those G rewards.
4. Set each output’s advantage to its own reward, normalised against that group.

Written out, the advantage for output i under outcome supervision, which is the form the common implementations run, is:

A_i,t = (r_i - mean(r)) / std(r)

Two properties of that formula are worth pulling out, because they are what actually change when you switch from PPO.

The subscript t is doing nothing here. Every token in output i receives the same advantage. There is no per-token credit assignment, because the thing that used to provide it was the critic. The signal is per-response: this answer was better than typical, so make all of it more likely.

The subscript is not decoration, though. The paper also defines a process-supervised variant in which each token takes the sum of normalised step rewards from its position onward. That one does vary with t and does restore per-token credit, at the price of a reward model that scores individual reasoning steps.

Only relative differences inside the group survive. Subtracting the mean and dividing by the standard deviation discards the reward’s scale and offset. A reward returning 0 and 1 and one returning 0 and 100 produce identical gradients, because one is a rescaling of the other and nothing downstream reads the reward directly.

That invariance comes from the division, so it holds only while you keep it. The last section covers the variants that drop the std term, and with it the immunity to reward scale.

Ranking alone is not preserved, though. Rewards of 0, 1, 2 and rewards of 0, 1, 10 order their samples identically and still produce different advantages, because normalisation keeps the spacing between scores. If your reward is graded rather than binary, the gaps you choose are part of the signal.

The KL penalty moves too. Rather than folding it into the reward the way PPO implementations often do, GRPO adds “the KL divergence between the trained policy and the reference policy to the loss”, which the authors say avoids complicating the advantage calculation.

That keeps the advantage a clean statement about relative quality, and leaves staying close to the reference model as a separate term with its own coefficient.

What Does Dropping the Critic Actually Save?

“Significantly reducing training resources” is worth turning into a number.

Assume mixed-precision training with Adam. A trainable model costs roughly 16 bytes per parameter: bf16 weights at 2, an fp32 master copy at 4, two Adam moments at 4 each and a bf16 gradient at 2. A frozen model costs about 2.

Assume also the common four-model layout, in which PPO holds policy and critic trainable plus reference and reward frozen, all four at the policy’s size. GRPO holds three.

Under those assumptions the percentage does not move, because every term is linear in parameter count. Dropping the critic removes one of the two trainable models, and trainable models cost eight times what frozen ones do.

Four honest caveats. This counts model and optimiser state only, excluding activations, the KV cache and rollout buffers, which are substantial and which GRPO makes larger rather than smaller. So 44.4% is a saving on one line item, not on end-to-end training memory.

A critic is not always the same size as the policy, though the paper’s framing is that it typically is. The totals are aggregates across however many devices you shard over, not a per-GPU requirement.

And the table assumes a learned reward model on both sides. If your reward is a rule-based verifier with no weights, the common setting for maths and code, both columns shrink: 7B becomes 238 GB against 126 GB, and the saving rises to 47.1%.

Which points at the real trade. The saving is not free, it is moved. A critic gives you a baseline for one forward pass. A group gives you a baseline for G full generations of the same question, so GRPO spends in sampling what it saves in memory.

How Big Should the Group Be?

8 to 16 for most setups. TRL’s GRPOTrainer defaults num_generations to 8, and larger values are used where the rollout budget allows: DeepSeekMath itself sampled 64 outputs per question when training DeepSeekMath-RL.

The reason the number stays modest is that precision improves slowly and cost does not. The group mean is an average of G samples, so its noise shrinks as one over the square root of G. The rollout cost rises with G exactly.

Going from 4 samples to 16 halves the noise in your baseline and costs four times as much. Going from 16 to 64 halves it again, for another four times.

You are buying square-root improvements with linear money, which is why doubling G is rarely the best use of the next unit of compute.

Baseline precision is only half the argument, though, and it is the half that pushes G down. The next section is the half that pushes it back up.

When Does a Group Teach Nothing?

When every sample in it scores the same, which happens more often than the formula suggests.

Look again at the numerator. Every advantage in a group is a deviation from that group’s own mean, so if all G samples earn identical rewards, every deviation is zero and the group contributes nothing to the gradient. The compute was spent and no learning occurred.

The denominator is zero too, which is undefined rather than harmless. Implementations add a small epsilon so the whole group comes out at zero instead of producing NaNs.

With a binary reward and a per-question pass rate of p, a group is uniform whenever all G samples agree, which happens with probability p^G + (1-p)^G.

Read the top and bottom rows. At a group size of 8, a question the model already passes 95% of the time produces a useless group 66.3% of the time. A question it passes only 5% of the time is exactly as wasteful.

The curve is symmetric, and it is savage at both edges. Questions the model has mastered teach nothing. Questions it cannot touch teach nothing either. GRPO learns from the band in the middle, where the model sometimes succeeds and sometimes does not.

This is also the argument that pushes group size back up, and why the previous section landed on a band rather than a floor. At a pass rate of 95%, moving from G=8 to G=16 cuts wasted groups from 66.3% to 44.0%.

Baseline precision alone would tell you to keep G small. The odds of drawing a group that disagrees with itself are what stop you taking that advice to its conclusion. The two pressures meet at 8 to 16 for most training sets, and higher when the problems sit near the edges.

This has a consequence that is easy to miss when reading the algorithm on its own. Problem selection is not preprocessing, it is a first-order lever. A training set that is uniformly too easy burns full rollout budgets to produce zero gradient, and the loss curve alone will not tell you that is what happened.

The tooling has caught up with the problem. TRL logs frac_reward_zero_std, the fraction of a generation batch whose reward standard deviation is zero, meaning every sample for that prompt scored the same.

DAPO makes dynamic sampling a core technique, oversampling and discarding groups with an accuracy of exactly 0 or 1 so that training steps only run on prompts that produced disagreement.

The same logic applies to the reward function rather than the questions. A reward that returns the same value for a good attempt and a mediocre one manufactures uniform groups by construction, no matter how well chosen the problems were.

What Has Changed Since the Paper?

The substitution held. Several of the details around it did not. As of August 2026, three corrections are common enough that a current implementation is unlikely to run the 2024 formulation exactly as published.

Dividing by the standard deviation is now contested. Understanding R1-Zero-Like Training argues that the std(r) denominator introduces a question-level difficulty bias, because dividing by a small spread inflates the advantages of exactly the near-uniform questions the previous section identified as unproductive. Its Dr. GRPO variant removes the term.

TRL’s documentation cites that finding and turns it into a setting rather than a verdict. scale_rewards still defaults to per-group scaling, with batch-level scaling and no scaling as the alternatives, and the docs note that Dr. GRPO recommends not scaling.

Treat it as a knob you should reach for when your problems cluster at the easy or hard extremes, not as a fix everyone has adopted.

How token losses are aggregated turned out to matter more than expected. Normalising each sequence’s token losses by its own length biases the model toward shorter completions when the advantage is positive and longer ones when it is negative.

TRL labels that original aggregation "grpo" and calls it “not recommended due to length bias”. Its current default loss_type is "dapo", which normalises by the number of active tokens in the global accumulated batch instead. Dr. GRPO’s alternative normalises by a global constant.

Importance sampling moved up a level for some setups. GSPO computes the ratio once per sequence rather than per token, and the GSPO paper reports that this often yields more stable training and better alignment with sequence-level rewards, which matters most for mixture-of-experts models. TRL exposes it as importance_sampling_level and still defaults to per-token, so it is opt-in.

None of this touches the core claim. The critic is still gone, the baseline is still the group, and the reasoning about group size and uniform groups above applies to every one of these variants.

What changed is the arithmetic wrapped around the advantage, not the idea that a group can stand in for a value model.

Where Does Future AGI Fit?

Narrowly, at the point the last section landed on.

GRPO consumes a reward signal and is silent about where it comes from. Its one hard requirement is that the signal separates samples, and a reward that fails to separate them is a defect you can measure on sampled outputs before committing any training compute to it.

That check is an evaluation problem. Rules specific to your domain will not match a general template, which is what custom eval templates are for: they cover “any domain-specific, business, or regulatory rule you define”, expressed as criteria with placeholder variables, alongside 156 built-in templates across 14 groups as of August 2026.

The useful move is to score a group of sampled outputs the way GRPO would and look at the spread rather than the average. A batch of questions whose samples all score identically is a batch that will not train, and that is visible in an eval run in minutes rather than in a loss curve that flatlines for reasons nobody can attribute.

How eval scores get wired into a tuning loop as the reward itself is a longer story, and it is already told in our post on eval versus RLHF feedback loops rather than repeated here. For the adjacent question of whether the tuned model actually improved, see evaluating fine-tuned LLMs.

What Should You Take From GRPO?

One substitution, three consequences.

Deleting the critic saves roughly 44% of model and optimiser memory, under the assumptions set out above, and costs you G generations per question instead. The advantage becomes per-response rather than per-token, so credit assignment inside a long answer is something you no longer get by default. And because every advantage is a deviation from its own group’s mean, only disagreement between samples produces learning.

Check them in that order, from the bottom up. Look at whether your rewards actually spread across samples before you tune the group size, and settle the group size before you argue about which variant of the loss to run.

The memory, noise and uniform-group figures in this post are arithmetic, not citations. Each one follows from the assumptions stated beside it, and the derivations are reproducible from those assumptions alone.

Frequently Asked Questions About GRPO

What Does GRPO Stand For?

Group Relative Policy Optimization. It was introduced in the DeepSeekMath paper as a variant of Proximal Policy Optimization that, the authors write, enhances mathematical reasoning abilities while concurrently optimizing the memory usage of PPO. The name is the mechanism: an output is scored relative to other outputs sampled for the same question, rather than against a learned prediction.

How Is GRPO Different From PPO?

PPO trains a separate critic to predict the expected reward for a prompt and uses that as the baseline. GRPO removes the critic and estimates the baseline from the average reward of several outputs sampled for the same question. It also places the KL penalty in the loss rather than folding it into the reward, which the authors say avoids complicating the advantage calculation.

Why Does GRPO Not Need a Value Model?

Because a baseline only has to answer one question: was this output better or worse than typical for this prompt. A critic answers it with a learned prediction. GRPO answers it by sampling several outputs and taking their mean, a direct estimate. The paper notes the critic is typically comparable in size to the policy, so removing it drops one large trainable model from the loop.

What Group Size Should You Use for GRPO?

8 to 16 for most setups, and TRL defaults to 8. Returns diminish while cost does not: noise in a mean of G samples shrinks as one over the square root of G, so 4 to 16 halves the baseline noise at four times the rollouts. The competing constraint is spread, since smaller groups make uniform groups, which carry no gradient, considerably more likely. DeepSeekMath itself used 64.

Does GRPO Need Verifiable Rewards?

Not strictly, though it suits them, because it works from differences between samples inside a group rather than from absolute values. What it does require is a reward that separates samples. Every advantage is a deviation from the group’s own mean, so a question where every sample scores identically yields no gradient at all, whatever the reward’s source.