grpo

GRPO Algorithm Reference

Overview

Group Relative Policy Optimization (GRPO) is a policy gradient method that eliminates the need for a learned critic by computing advantages from group statistics. For each prompt, multiple completions are sampled and their rewards are normalized within the group to produce advantages.

Key insight: Instead of training a value function to estimate V(s), GRPO uses the mean reward of the group as the baseline. This removes the critic entirely, reducing memory and avoiding value function approximation errors.

Training Loop

Each step follows this pipeline:

1. Sample G completions per prompt from current policy
2. Decode completions to text
3. Score completions with reward function
4. Compute group-relative advantages
5. Compute per-token log-probs (current policy + reference policy)
6. Compute clipped surrogate loss with KL penalty
7. Backpropagate and update

Advantage Estimation

Rewards are normalized within each group of G completions for the same prompt:

mu   = mean(r_1, ..., r_G)
sigma = std(r_1, ..., r_G)
A_i  = (r_i - mu) / (sigma + epsilon)

epsilon is a small constant (typically 1e-4 to 1e-8) for numerical stability only. It prevents division by zero when all rewards in a group are identical.

Properties:

• Advantages within each group sum to approximately zero;
• High-reward completions get positive advantages, low-reward get negative;
• The learning signal vanishes if epsilon is too large (dominates denominator) or if rewards are constant. For example, if all rewards in a group are identical (or nearly identical), then causing all advantages to collapse to 0;

Log-Probability Computation

Per-token log probabilities via log-softmax:

log_prob(token_i) = logit(token_i) - logsumexp(logits)

Critical invariant: log_prob <= 0 always, since it's the log of a probability in (0, 1].

Sequence-level log probability: log_pi(y|x) = sum_t log_pi(t_j | x, t_<j)

Loss Function