TRPO: The Trust-Region Step Size
Advanced
The problem, from first principles
TRPO takes the largest step whose KL stays within the limit. Approximating the KL by a quadratic around the current policy turns that into a closed-form step length, which is the scaling the paper applies to its natural gradient direction.
From theory to code
Implement trust_region_step_size(delta, quad), returning sqrt(2 delta / quad).
Constraints
quadis positive.
Hints
Hint 1
Solve the quadratic constraint 0.5 * beta^2 * quad = delta for beta.
Keywords
research-papersreinforcement-learningpolicy-gradienttrust-region
Python
Run your code to see results
No results yet
Run your code to check it against the examples, or submit to run every test.