Prioritized Experience Replay
Advanced
Prioritized Experience Replay (PER) samples high-TD-error transitions more often during learning, focusing effort on surprising/informative experiences.
The problem, from first principles
Standard experience replay samples uniformly. But some transitions are more informative: high TD-error means the value estimate is wrong. PER uses a priority weight proportional to |TD-error| to oversample these transitions.
From theory to code
Implement prioritized_replay_buffer(transitions, td_errors, batch_size, alpha=0.6) which:
- Takes transitions (list of (s,a,r,s',done) tuples)
- Takes td_errors (list of TD-error magnitudes)
- Returns a batch of indices sampled with probability proportional to (priority + epsilon)^alpha
Constraints
- alpha in [0,1]: interpolates between uniform (0) and max priority (1)
- Priorities are |TD-error| + epsilon
- Batch size is requested size
- Return indices, not transitions
Hints
Hint 1: Normalize priorities
Priority_i = (|TD-error_i| + epsilon)^alpha. Normalize to sum to 1.Hint 2: Sample with replacement
Use np.random.choice with probabilities.Hint 3: Handle edge cases
Zero errors should have small nonzero priority.Keywords
rlexperience-replayimportance-samplingoptimization
Python
Run your code to see results
No results yet
Run your code to check it against the examples, or submit to run every test.