PPO: The Probability Ratio
Beginner
The problem, from first principles
PPO (Schulman et al., 2017) reuses each batch of experience for several updates. To do that correctly, each sample is reweighted by how much more or less likely the current policy is to take its action than the policy that collected it.
From theory to code
Implement probability_ratio(logp_new, logp_old), returning exp(logp_new - logp_old).
Constraints
- Work in log space; exponentiate the difference.
Hints
Hint 1
Subtract the log-probabilities, then take exp.
Keywords
research-papersreinforcement-learningpolicy-gradientppo
Python
Run your code to see results
No results yet
Run your code to check it against the examples, or submit to run every test.