Dueling Networks: Combining Value and Advantage

The problem, from first principles

Wang et al. (2016) split the Q-network into a state value V(s) and an advantage A(s, a). Many states have similar value regardless of the action, so learning V directly is efficient. Subtracting the mean advantage makes the split identifiable.

From theory to code

Implement dueling_q(V, A), returning V + A - mean(A) for each action.

Constraints

• V is a scalar and A is a vector over actions.

Hints