GAE: Combining Residuals With Lambda
Advanced
The problem, from first principles
GAE (Schulman et al., 2016) blends many one-step advantage estimates with an exponential weight. The parameter lambda trades bias against variance: lambda 0 uses only one step, and lambda 1 uses the full Monte Carlo return.
From theory to code
Implement gae_advantages(deltas, gamma, lam), computing the exponentially weighted sum of future residuals.
Constraints
lambetween 0 and 1.
Hints
Hint 1
Walk the residuals from last to first, keeping a running sum scaled by gamma * lam.
Keywords
research-papersreinforcement-learningpolicy-gradientadvantage
Python
Run your code to see results
No results yet
Run your code to check it against the examples, or submit to run every test.