AI 101 — What is GRPO (group relative policy optimization)?

GRPO is the reinforcement-learning method behind most open reasoning models: instead of training a second neural network to judge how good each answer is, it has the model answer the same question many times and rewards whichever answers beat the group's own average.

The name is an initialism nobody says out loud — group relative policy optimization — but the idea fits in a sentence, and once you have it a lot of recent AI news stops sounding like alchemy. "Policy" is just the model being trained. "Group relative" is the trick.

Why it matters right now

Reasoning models — the ones that visibly think before answering — get their long chains of thought from a reinforcement-learning stage that runs after pre-training. That stage needs a training signal, and how you build it decides whether the run costs a rack of GPUs or a hall of them.

GRPO arrived in DeepSeek's DeepSeekMath paper in February 2024, described by its authors as a variant of PPO that improves reasoning "while concurrently optimizing the memory usage of PPO." A year later DeepSeek's R1 report made it famous: R1-Zero, trained with GRPO and simple rule-based rewards and no human-written reasoning examples at all, climbed from 15.6% to 77.9% pass@1 on the AIME 2024 math benchmark during training, per the R1 paper. Most open reasoning models released since have used GRPO or a close variant, and the name has spread into agent training too.

The mental model: drop the critic

Reinforcement learning needs to know which actions were better than expected. The standard tool, PPO, learns that with a second network called a value model or critic: a learned predictor of how much reward a situation is worth, so the model can be told "this went better than predicted." The catch is that the critic is roughly as large as the model you are training, with its own memory footprint and its own training instability. For a model with hundreds of billions of parameters, that is a second monster on the same GPUs.

GRPO deletes the critic. For each question it samples a group of answers — DeepSeek used 16 per question in R1's first RL stage — scores each one with a checker, then compares the answers with each other. The advantage for a single answer is its score minus the group's average, divided by the group's spread, as the R1 paper sets out. If one answer solved the problem and fifteen failed, that answer gets a strong positive push. If all sixteen failed, nothing gets pushed, because nothing stood out.

That is the whole trick: the group is its own baseline. No critic to train, no critic to store, and the memory saved goes back into a bigger batch or a bigger model.

The analogy: sixteen free throws

Picture a coach with sixteen players taking the same free throw. Judging a single shot properly would need a model of the perfect shot — a scout who predicts, from the stance and the arc, how likely this attempt is to go in. That scout is the critic, and he is expensive: he has to be right about basketball in general, not just about this drill.

The GRPO coach does something cheaper. He watches all sixteen shots, counts how many went in, and grades each player against the group's average. Player 7 swished while ten others missed: strong signal, change what she is doing. The whole group missed: no lesson today, because nobody did better than anyone else. It is grading on a curve, and the curve is made of the model's own attempts.

Where the reward comes from

A curve needs a score, and scores are where this method lives or dies. The cheap, reliable option is a checker: a math answer that matches the reference, code that passes the unit tests, a response that follows the required format. DeepSeek's R1 report calls these rule-based rewards and notes they exist partly to avoid depending on a reward model that can be gamed.

What gets rewarded is not knowledge. The model is not being taught new facts; it is being shown which of its existing behaviours pay off — trying a second approach, checking its own arithmetic, writing intermediate steps instead of guessing. That is how a model can jump from 15.6% to 77.9% on competition math without new textbooks.

Common misconceptions

"GRPO is PPO with a part removed, so it must be worse." It is a different estimator, not a crippled one. The R1 authors ran PPO on the same task and found it could come close — but only after tuning a sensitivity GRPO does not have, with an extra model in memory. Cheaper wins on a budget, and RL budgets are the constraint.

"It works for any task." It needs a score you can compute. Math and code have checkers. An essay does not, and the R1 report is explicit that when the reward comes from a model rather than rules, training can be gamed. That is the boundary of the technique, not a detail.

"The model learns to think like a human." It learns whichever habits the checker rewards. If the checker only reads the final answer, the model may find a shortcut that scores without reasoning well. Labs spend more effort on reward design than on the optimizer for this reason.

"More attempts always help." A group where every answer scores the same teaches nothing — the advantage is zero. The signal comes from questions the model can sometimes solve, not from ones it always gets right.

Where it's heading

The recipe is still moving. Alibaba's Qwen team published Group Sequence Policy Optimization in 2025, which compares whole answers rather than individual tokens, reports better training efficiency than GRPO, and credits it with improvements in the Qwen3 models. Agent training has pushed further, with delayed rewards for multi-step tool use. The pattern holds: cheaper estimates of "better than average" plus a reward you can trust.

Where to learn more

The DeepSeekMath paper is the original source and is readable if you skim the equations; the R1 report is the practical manual, full of sampling settings and failure modes. For the surrounding ideas, our explainer on what reinforcement learning is covers the family this belongs to.

Related reading: What are reasoning tokens? explains the output this kind of training is designed to produce, and What is LLM-as-a-judge? covers the scoring approach GRPO users usually avoid.

If a model can be trained by a checker, who should decide what the checker rewards? Write to us via our contact page.

AI 101 — What is model collapse?

Model collapse is what happens to a generative AI model that learns mostly from other AI models: it slowly stops representing the full range of real data, and the loss is not recoverable by training longer. That is the plain version. The precise version, from the 2024 study that named the effect, is that "indiscriminate use of model-generated content in training causes irreversible defects in the resulting models, in which tails of the original content distribution disappear" — published in…

Exponential Time Hypothesis is false, OpenAI and Anthropic say

Two labs now say their internal models overturned a 25-year-old assumption at the base of computational complexity. The people who invented that assumption are the ones checking the work. OpenAI published three research results on Monday that it says its internal models produced, and the sharpest one claims the Exponential Time Hypothesis is false.

Claude proved 123 open-source libraries can't crash on RISC-V

Two days after two PHP bugs put a slice of the internet into an unscheduled maintenance window, Anthropic published the closest thing to an antidote anyone has shipped: Claude wrote formal proofs that 123 open-source libraries cannot crash, at the assembly level, for any input up to 64 MB. The qualifier at the end of that sentence is where the real story is. What they actually proved Anthropic worked with Theorem, a verification startup, and pointed Claude at Google's OSS-Fuzz corpus — the…

AI 101 — What does GPT stand for?

GPT stands for Generative Pre-trained Transformer — the three-word recipe behind OpenAI's language models, from GPT-2 to today's GPT-6. It is also the most-abbreviated phrase in AI: most people who type "GPT" into a search box are really asking what the letters mean and why they keep appearing in the news. Why it matters right now Every model name in the headlines is a small riddle.

AI 101 — What is AGI?

AGI — short for artificial general intelligence — is a hypothetical AI system that could do any intellectual job a person can do, instead of being good at one narrow task. Every AI you can actually use today is narrow: it writes, translates, spots patterns in scans, plays games — each system built for its lane. AGI names the destination where one system covers all the lanes.

AI 101 — What is reinforcement learning?

Reinforcement learning is how a machine learns by trial and error: it tries things, notices which attempts earned a reward, and drifts toward the behaviour that earns more of it. There is no answer key involved. In the supervised learning most people picture, someone hands the model millions of labelled examples and it learns to copy them. In reinforcement learning, nobody says what the right move is — the machine acts, gets a number back telling it how well that went, and adjusts.

AI 101 — What is LLM-as-a-judge?

If you read a week of AI news, you'll notice almost nobody grades models by hand any more. When a lab says its new model is better at writing, summarising or handling customers, the number usually came from another AI. An LLM judge is a language model used to grade the work of another model — or of your own AI feature — when no piece of code can decide whether the answer was good.

AI 101 — What is multimodal AI?

Multimodal AI is a model that handles more than one kind of input — text, images, audio, video — inside the same system, rather than one model per sense stitched together. "Modality" is just the academic word for a channel of information. Text is a modality. A photograph is a modality. A voice recording is a modality. Multimodal AI reads more than one of them at once and reasons across them. That last part is the whole trick.