TRPO: The Surrogate Objective
Intermediate
The problem, from first principles
TRPO (Schulman et al., 2015) improves a policy by maximizing the advantage of new actions, weighted by how much more likely the new policy makes them. The ratio is an importance weight that corrects for sampling from the old policy.
From theory to code
Implement surrogate_objective(ratio, advantage), returning the mean of ratio * advantage.
Constraints
- The ratio of 1 means the new policy equals the old one.
Hints
Hint 1
Multiply element-wise and take the mean.
Keywords
research-papersreinforcement-learningpolicy-gradienttrust-region
Python
Run your code to see results
No results yet
Run your code to check it against the examples, or submit to run every test.