GAE: One-Step TD Residuals
Beginner
The problem, from first principles
The advantage of an action is how much better it turned out than the value function expected. A single step of that comparison is the TD residual: the observed reward plus the discounted next value, minus the current value.
From theory to code
Implement td_residuals(rewards, values, gamma, last_value), returning the one-step residual at each time step.
Constraints
- The value after the last step is
last_value.
Hints
Hint 1
Shift the values by one step to get each next value, then compute r + gamma * next - value.
Keywords
research-papersreinforcement-learningpolicy-gradientadvantage
Python
Run your code to see results
No results yet
Run your code to check it against the examples, or submit to run every test.