GRPO

PPO works, but the last two chapters left it carrying an expensive passenger: the value model. On a 16GB card that critic is the difference between fitting and overflowing, and it exists for one job, turning a single end-of-sequence reward into a dense per-token advantage via GAE. Group Relative Policy Optimization (GRPO), introduced by the DeepSeek team and the workhorse behind DeepSeek-R1, asks a sharp question: what if I get the advantage a different way, from comparison within a group of samples, and delete the critic entirely? This chapter derives GRPO straight from the PPO objective, shows precisely which term the group-relative advantage replaces, works through the KL-to-reference regularizer and why the specific "k3" estimator is the right choice, disentangles the token-versus-sequence aggregation question that causes most of the confusion in real implementations, and catalogues the pathologies (length bias, standard-deviation collapse) along with the DAPO-style fixes that patch them. GRPO is the algorithm Part VII actually runs on the baseline machine, so this is the one to get exactly right.

Theory

The idea: replace the critic with a group baseline

Go back to the advantage, $A(s,a)=Q(s,a)-V(s)$. The value $V(s)$ is a baseline, an estimate of "how good is this state on average," and PPO learns it with a whole second network. But there is an older, cheaper way to get a baseline that the REINFORCE chapter already flagged: sample several actions from the same state and use their average reward as the baseline. If from prompt $q$ I draw a group of $G$ complete responses $\{o_1,\dots ,o_G\}$ and score each with reward $r_i$, then the group mean $\bar{r}=\frac{1}{G}{\sum }_i{r}_i$ is an empirical, per-prompt estimate of the state value, and $r_i-\bar{r}$ is an empirical advantage: how much better response $i$ was than its peers on the same prompt. No critic, no bootstrap, no GAE. This is the entire conceptual move of GRPO. The critic $V_{\varphi }(s)$ is replaced by a Monte Carlo baseline computed fresh for each prompt from its own group of samples.

GRPO goes one step further and normalizes by the group's spread, giving the group-relative advantage

$${\hat{A}}_i=\frac{r_i-\text{mean}(\{r_1,\dots ,r_G\})}{\text{std}(\{r_1,\dots ,r_G\})},\text{\hspace{1em}(8.1)}$$

which is assigned identically to every token in response $i$ (every token in a sequence shares that sequence's advantage). The mean-subtraction is the variance-reducing baseline; the standard-deviation division is a whitening step that keeps the advantage scale roughly constant across prompts of wildly different difficulty, so the learning rate does not have to absorb the fact that one prompt's rewards live in $[0,1]$ and another's cluster at $0.9$. I will return to that division, because it is also the source of one of GRPO's pathologies.

Deriving the GRPO objective from PPO

Now put equation (8.1) into the PPO machinery. GRPO keeps PPO's clipped surrogate exactly (chapter 7's equation 7.7) and adds an explicit KL penalty to a frozen reference policy ${\pi }_{\text{ref}}$, optimizing

$$\begin{array}{rl}{\mathcal{J}}_{\text{GRPO}}(\theta )={\mathbb{E}}_{q,\{o_i\}\sim {\pi }_{\text{old}}}\left[\frac{1}{G}\sum _{i=1}^G\frac{1}{|o_i|}\sum _{t=1}^{|o_i|}\right(& \min (r_{i,t}(\theta ){\hat{A}}_i,\text{clip}(r_{i,t}(\theta ),1-ϵ,1+ϵ){\hat{A}}_i)\\ & -\beta D_{\text{KL}}[{\pi }_{\theta }\Vert {\pi }_{\text{ref}}{]}_{i,t})],\end{array}\text{\hspace{1em}(8.2)}$$

where $r_{i,t}(\theta )=\frac{{\pi }_{\theta }(o_{i,t}\mid q,o_{i,<t})}{{\pi }_{\text{old}}(o_{i,t}\mid q,o_{i,<t})}$ is the same per-token importance ratio as PPO, and $\beta$ weights the KL regularizer.

The payoff is immediate and is the reason GRPO exists for people on small hardware: no value model. The critic's weights, gradients, and optimizer state, the single most expensive optional component of PPO (chapter 6's vram-budget), are simply gone. What remains is the policy (trained), a frozen reference (forward-only), and, in the RLVR setting, a verifier that is a Python function rather than a reward network. That is the configuration that fits a 1.5-3B policy comfortably on the RTX 5080.

The KL estimators: k1, k2, k3

The KL term needs care because I estimate it from samples, not in closed form, and the naive estimator is bad. I am estimating $D_{\text{KL}}[{\pi }_{\theta }\Vert {\pi }_{\text{ref}}]={\mathbb{E}}_{x\sim {\pi }_{\theta }}[\log \frac{{\pi }_{\theta }(x)}{{\pi }_{\text{ref}}(x)}]$ using samples $x$ drawn from ${\pi }_{\theta }$ (the tokens I actually generated). Define the likelihood ratio $\rho =\frac{{\pi }_{\text{ref}}(x)}{{\pi }_{\theta }(x)}$ (reference over current, note the direction). Schulman's three estimators are:

$$k_1=-\log \rho =\log \frac{{\pi }_{\theta }}{{\pi }_{\text{ref}}},k_2=\frac{1}{2}(\log \rho {)}^2,k_3=\rho -1-\log \rho .\text{\hspace{1em}(8.3)}$$

A useful sanity check that the k3 gradient behaves: even though k3 is a nonlinear function of $\rho$, its gradient with respect to $\theta$ points in the KL-reducing direction, and near $\rho =1$ it reduces to the familiar penalty. The practical note is just that you must get the direction of $\rho$ right (${\pi }_{\text{ref}}$ over ${\pi }_{\theta }$); flip it and the estimator is no longer the KL you want.

Token versus sequence aggregation

Equation (8.2) as written normalizes each response by its own length $|o_i|$ (the inner $\frac{1}{|o_i|}{\sum }_t$) and then averages over the $G$ responses. This is a choice, and it is one of the most consequential and most confused details in GRPO. There are two axes: how to reduce over tokens within a response, and how to weight responses of different lengths against each other.

The one-line rule I use: per-sequence normalization treats responses as the unit and quietly rewards length; per-token (batch-level) normalization treats tokens as the unit and removes that particular bias. For reasoning-model training the per-token variant is now the common default, and I will use it in the Part VII lab.

Pathologies and the DAPO-style fixes

Two failure modes recur, both traceable to specific terms above.

Length bias, just derived, from per-sequence normalization. Fix: per-token (batch-level) loss normalization, and DAPO additionally adds an explicit "overlong" soft penalty that shapes the reward down for responses that run past a length budget, so the model is discouraged from rambling to game the normalizer.

Standard-deviation collapse, from the division in equation (8.1). When every response in a group gets the same reward (all correct, or all wrong, which is extremely common with binary verifiable rewards on easy or impossible prompts), the group std is zero and equation (8.1) is $0/0$. Implementations add an $ϵ$ to the denominator, so the advantage becomes zero and the prompt contributes no gradient at all. That is a silent waste: a large fraction of your batch can be prompts the model already fully solves or cannot touch, contributing nothing, and it also introduces a subtle scale artifact because prompts that happen to have small nonzero std get amplified advantages (dividing by a tiny number), overweighting near-degenerate prompts. Two fixes: Dr. GRPO argues for dropping the std division entirely (use only $r_i-\bar{r}$, an unnormalized advantage, removing the amplification artifact), and DAPO's dynamic sampling filters out prompts whose group is all-correct or all-wrong and resamples until the batch is full of prompts with reward variance, so every gradient step spends its compute on prompts that actually carry signal.

Clip-higher, DAPO's third tweak, addresses entropy collapse. Recall from chapter 7 that PPO's clip is symmetric, $[1-ϵ,1+ϵ]$. DAPO decouples the two sides, $[1-{ϵ}_{\text{low}},1+{ϵ}_{\text{high}}]$ with ${ϵ}_{\text{high}}>{ϵ}_{\text{low}}$ (for example $0.28$ versus $0.20$). The rationale falls straight out of the chapter-7 piecewise analysis: for a good, currently-low-probability token, the upper clip at $1+ϵ$ caps how fast its probability can rise, which throttles exploration of promising-but-rare tokens and lets the policy collapse onto a few high-probability moves. Raising the upper bound alone lets rare good tokens climb faster (more exploration) without loosening the lower bound that guards against over-suppression. It is a targeted widening of exactly one arm of the clip geometry.

Tooling

The tool is TRL's GRPOTrainer (and its close cousins in verl and OpenRLHF). It embodies equation (8.2) directly: for each prompt it generates $G$ completions (the num_generations argument, typically 4 to 16), scores them with a user-supplied reward_funcs list (which for RLVR is just Python functions returning floats, no reward model in VRAM), computes the group mean and std to get ${\hat{A}}_i$ per equation (8.1), and optimizes the clipped-plus-KL loss. The knobs map one-to-one onto the theory: beta is the KL coefficient $\beta$, epsilon and (in newer versions) epsilon_high are the clip bounds including DAPO's clip-higher, loss_type selects the aggregation ("grpo" for per-sequence, "dr_grpo" or "bnpo"-style for the per-token batch normalization that removes length bias), and scale_rewards toggles the std division from equation (8.1) so you can run the Dr. GRPO unnormalized variant.

Lab

The goal of this lab is to compute the two things GRPO adds to PPO, the group-relative advantage (equation 8.1) and the three KL estimators (equation 8.3), on synthetic data where I know the answer, and to watch k3 beat k1 and k2 on the bias-and-variance criterion the derivation claimed. This isolates the two novel pieces so that when the full GRPO run in Part VII uses them, they are already understood in miniature. Pure arithmetic, no GPU, no model.

This is a uv project. From the repo root:

uv init labs/grpo-pieces
cd labs/grpo-pieces
uv add numpy matplotlib

"""The two novel pieces of GRPO, in isolation:
  (1) group-relative advantage (eq 8.1) over G samples per prompt, and
  (2) the k1/k2/k3 KL estimators (eq 8.3), comparing bias and variance
      against the exact KL between two known categorical distributions.
"""
from pathlib import Path
import csv
import numpy as np

OUT = Path("artifacts"); OUT.mkdir(exist_ok=True)
rng = np.random.default_rng(0)

# ---- Piece 1: group-relative advantage --------------------------------------
def group_advantage(rewards, scale_by_std=True, eps=1e-6):
    mean = rewards.mean()
    if scale_by_std:
        return (rewards - mean) / (rewards.std() + eps)
    return rewards - mean            # Dr. GRPO variant (no std division)

# ---- Piece 2: KL estimators over a known pair of distributions --------------
def kl_exact(p, q):
    return float(np.sum(p * np.log(p / q)))

def kl_estimators(p, q, n_samples):
    """Sample x ~ p (= pi_theta), form rho = q(x)/p(x) = pi_ref/pi_theta."""
    idx = rng.choice(len(p), size=n_samples, p=p)
    rho = q[idx] / p[idx]
    logrho = np.log(rho)
    k1 = -logrho                      # unbiased, high variance, can be < 0
    k2 = 0.5 * logrho**2             # biased, nonnegative
    k3 = rho - 1.0 - logrho          # unbiased, nonnegative
    return k1, k2, k3

def main():
    # --- advantage demo: a group where 2 of 6 responses are correct ---
    rewards = np.array([1.0, 0.0, 0.0, 1.0, 0.0, 0.0])
    print("rewards:", rewards.tolist())
    print("A (std-normalized):", np.round(group_advantage(rewards), 3).tolist())
    print("A (Dr.GRPO, no std):", np.round(group_advantage(rewards, False), 3).tolist())
    # std-collapse case: every response identical
    flat = np.ones(6)
    print("A when all-correct (std collapse):",
          np.round(group_advantage(flat), 3).tolist(), "-> zero gradient")

    # --- KL estimator comparison ---
    p = np.array([0.4, 0.3, 0.2, 0.1])          # pi_theta
    q = np.array([0.1, 0.2, 0.3, 0.4])          # pi_ref (deliberately far)
    true_kl = kl_exact(p, q)
    print(f"\nExact KL[pi_theta || pi_ref] = {true_kl:.4f}")

    rows = []
    for n in (1, 4, 16, 64, 256):
        trials = 20000 // n
        m1 = np.array([kl_estimators(p, q, n)[0].mean() for _ in range(trials)])
        m2 = np.array([kl_estimators(p, q, n)[1].mean() for _ in range(trials)])
        m3 = np.array([kl_estimators(p, q, n)[2].mean() for _ in range(trials)])
        for name, m in (("k1", m1), ("k2", m2), ("k3", m3)):
            rows.append({"n": n, "estimator": name,
                         "mean": m.mean(), "bias": m.mean() - true_kl,
                         "std": m.std(),
                         "frac_negative": float((m < 0).mean())})

    csv_path = OUT / "kl_estimators.csv"
    with csv_path.open("w", newline="") as f:
        w = csv.DictWriter(f, fieldnames=["n","estimator","mean","bias","std","frac_negative"])
        w.writeheader(); w.writerows(rows)

    print("\n n  est   bias      std     frac<0")
    for r in rows:
        print(f"{r['n']:>3} {r['estimator']}  {r['bias']:+.4f}  "
              f"{r['std']:.4f}  {r['frac_negative']:.2f}")
    print(f"\nArtifact: {csv_path.resolve()}")

if __name__ == "__main__":
    main()

Run it:

uv run python grpo_pieces.py

What you should see. For the advantage: the std-normalized advantages for [1,0,0,1,0,0] come out symmetric around zero with the two correct responses getting equal positive values and the four wrong ones equal negative values, the Dr. GRPO variant gives the same signs at a different (unnormalized) scale, and the all-correct group prints all-zeros, the std-collapse case that contributes no gradient. For the KL estimators: k3's bias column hovers at essentially zero for every $n$ while its std shrinks as $n$ grows (unbiased, consistent), k1 shares the near-zero bias but with larger std and a nonzero frac_negative at small $n$, and k2 shows a persistent nonzero bias that more samples never remove. The CSV lets you plot bias-versus-$n$ for all three and see k2's flat bias floor against k1 and k3 converging to the true KL. This is the miniature of the two pieces the Part VII GRPO run assembles into a full training step.