Generalized Advantage Estimation (GAE)
Advanced
GAE: exponentially-weighted average of TD errors. Trades off variance (low λ) and bias (high λ) in advantage estimates.
The problem, from first principles
TD(0) has low variance but high bias. MC has zero bias but high variance. GAE interpolates via λ: combines 1-step, 2-step, ..., n-step returns with exponential weights.
From theory to code
Implement gae(values, rewards, next_values, gamma, lambda_) which:
- Computes TD residuals: δ = r + γV(s') - V(s)
- Exponentially weighted sum: A = Σ(γλ)^t δ_t
- Returns advantages array matching trajectory length
Constraints
- λ in [0, 1]: low = variance, high = bias
- Typically λ = 0.95
- Must handle trajectory endings (done flags)
Hints
Hint 1: TD residuals
δ_t = r_t + γV(s_{t+1}) - V(s_t)Hint 2: Exponential weighting
gae_t = δ_t + (γλ) δ_{t+1} + (γλ)^2 δ_{t+2} + ...Hint 3: Backward iteration
Compute from end of trajectory backward, accumulating exponentiallyKeywords
rlgaeadvantage-estimationvariance-reduction
Python
Run your code to see results
No results yet
Run your code to check it against the examples, or submit to run every test.