Double Q-Learning
Advanced
Double Q-Learning addresses overestimation bias in standard Q-learning by maintaining two Q estimates and decoupling selection from evaluation.
The problem, from first principles
Standard Q-learning uses max Q(s',a') which can overestimate values (max of noisy estimates is biased high). Double Q-learning uses one Q for selection and another for evaluation, reducing bias.
From theory to code
Implement double_q_learning(env_step, num_episodes, epsilon, gamma, alpha, seed=None) which:
- Maintains two Q estimates: Q1 and Q2
- Updates both alternately
- Uses Q1 to select best action, Q2 to evaluate (and vice versa)
- Returns (Q_avg, policy)
Constraints
- Two separate Q dictionaries
- Alternate which Q is used for selection vs evaluation
- Random choice of which Q to update each step
- Return averaged final Q values
Hints
Hint 1: Two Q values
Keep Q1 and Q2 both as dicts.Hint 2: Decoupling
Use one Q to pick best action, other Q to evaluate value.Hint 3: Alternation
Each step, randomly choose which Q to update.Keywords
rltemporal-differencecontroloff-policy
Python
Run your code to see results
No results yet
Run your code to check it against the examples, or submit to run every test.