AdamW: The Effective Decay per Step
Beginner
The problem, from first principles
With decoupled decay, each step multiplies the weights by a fixed factor. Over many steps that factor compounds, so the learning rate and decay together set how fast old weights are forgotten.
From theory to code
Implement effective_decay(lr, wd, steps), the fraction of weights remaining after pure decay.
Constraints
- The shrink factor per step is one minus lr times wd.
Hints
Hint 1
Raise one minus lr times wd to the number of steps.
Keywords
research-papersoptimizationweight-decay
Python
Run your code to see results
No results yet
Run your code to check it against the examples, or submit to run every test.