InstructGPT: The PPO Clipped Objective
Advanced
The problem, from first principles
InstructGPT was trained with PPO (Schulman et al., 2017). PPO limits how far one update can move the policy, which keeps training stable. The clipped surrogate does this by refusing to reward ratios that leave a small band around 1.
From theory to code
Implement clipped_surrogate(ratio, advantage, eps), returning the element-wise minimum of the raw and clipped objectives.
Constraints
epsdefaults to 0.2.
Hints
Hint 1
Compute ratio * advantage and clip(ratio, 1 - eps, 1 + eps) * advantage, then take np.minimum.
Keywords
research-paperstransformersllmalignmentrlhf
Python
Run your code to see results
No results yet
Run your code to check it against the examples, or submit to run every test.