Advantage Actor-Critic (A3C)
Advanced
Advantage Actor-Critic (A3C): parallel actors, shared policy and value networks. Each actor collects rollouts, computes advantages, and pushes updates asynchronously.
The problem, from first principles
Actor-Critic is good but samples slowly. A3C parallelizes: N workers collect N independent episodes concurrently, all update the same shared network without locks. Massive speedup, no batching needed.
From theory to code
Implement a3c_worker_update(actor_network, critic_network, trajectory, gamma, entropy_coeff) which:
- Takes one rollout trajectory
- Computes GAE advantages over trajectory
- Computes actor loss: -log π(a|s) * advantage - entropy_coeff * H(π)
- Computes critic loss: MSE(V, target)
- Returns gradient updates for actor and critic
Constraints
- Entropy regularization prevents premature policy collapse
- Asynchronous updates: no batch averaging
- One worker processes one episode independently
Hints
Hint 1: Entropy regularization
H(π) = -π log π, added to actor loss to encourage explorationHint 2: Per-worker updates
Each worker maintains local copy, pushes gradients to shared weightsHint 3: Trajectory processing
Process entire episode from trajectory, compute advantages via bootstrapped valuesKeywords
rla3cactor-criticparallel
Python
Run your code to see results
No results yet
Run your code to check it against the examples, or submit to run every test.