PPO: The Combined Training Loss
Intermediate
The problem, from first principles
PPO optimizes three things together: the clipped policy objective, the value function fit, and an entropy bonus that keeps the policy from collapsing early. The paper combines them with two coefficients into one scalar objective.
From theory to code
Implement ppo_total_loss(clip_obj, vf_loss, entropy, c1, c2), returning the negated combined objective.
Constraints
- Optimizers minimize, so the objective is negated.
Hints
Hint 1
Compute clip_obj - c1 * vf_loss + c2 * entropy, then return its negative.
Keywords
research-papersreinforcement-learningpolicy-gradientppo
Python
Run your code to see results
No results yet
Run your code to check it against the examples, or submit to run every test.