GRPO Advantages & Clipped Loss

The problem, from first principles

Policy-gradient methods need an advantage: how much better than expected a response was. PPO trains a separate value network to supply the baseline, which doubles the memory of the training job. GRPO removes it with a trick that suits language models: sample a group of G responses to the same prompt, score them, and use the group itself as the baseline. A response's advantage is its reward minus the group mean, divided by the group standard deviation. If every response in a group gets the same reward there is nothing to learn from that prompt and the advantage is zero. The advantages then feed the same clipped surrogate used by PPO, which limits how much one update can change the policy.

From theory to code

Implement group_advantages and clipped_surrogate_loss.

Constraints

• rewards has shape (P, G): P prompts with G sampled responses each. group_advantages(rewards, eps) returns (rewards - mean) / (std + eps) computed per row, with the population standard deviation (ddof=0). Use eps = 1e-6 by default.
• clipped_surrogate_loss(ratio, advantage, clip_eps) takes arrays of the same shape, where ratio = pi_new / pi_old for each sampled token or response, and returns the mean of -min(ratio * advantage, clip(ratio, 1 - clip_eps, 1 + clip_eps) * advantage).
• A group whose rewards are all equal has advantage exactly 0 for every response.
• Return a (P, G) float array and a Python float respectively.

Hints