AdamW: The Decoupled Update
Intermediate
The problem, from first principles
Adam's L2 regularization adds the decay term to the gradient, so it is rescaled by the adaptive denominator. AdamW (Loshchilov & Hutter, 2017) applies decay directly to the weights, which restores the regularization meaning.
From theory to code
Implement adamw_update(w, m_hat, v_hat, lr, wd, eps), the Adam step plus a decoupled decay term.
Constraints
- Decay is scaled by the learning rate and not by the second moment.
Hints
Hint 1
Compute the Adam direction, then subtract lr times wd times the current weights.
Keywords
research-papersoptimizationweight-decay
Python
Run your code to see results
No results yet
Run your code to check it against the examples, or submit to run every test.