PPO: The Clipped Objective Over a Batch
Advanced
The problem, from first principles
The clipped surrogate is the heart of PPO. Each sample contributes the smaller of the raw and clipped weighted advantage, and the batch objective is their average.
From theory to code
Implement ppo_objective(ratios, advantages, eps), the batch mean of the element-wise clipped minimum.
Constraints
epsis the clip range, typically 0.2.
Hints
Hint 1
Compute ratios * advantages and clip(ratios) * advantages, take the element-wise minimum, then average.
Keywords
research-papersreinforcement-learningpolicy-gradientppo
Python
Run your code to see results
No results yet
Run your code to check it against the examples, or submit to run every test.