SGDR: The Cosine Learning Rate
Beginner
The problem, from first principles
SGDR (Loshchilov & Hutter, 2016) anneals the learning rate along half a cosine curve within each cycle. Restarting it to the maximum at the start of each cycle lets the optimizer re-explore.
From theory to code
Implement cosine_lr(t_cur, T_i, eta_min, eta_max), the cosine-annealed learning rate.
Constraints
- Use the position within the current cycle.
Hints
Hint 1
Take the cosine of pi times the fraction of the cycle completed, map it from minus one to one onto the bounds.
Keywords
research-papersoptimizationlearning-rate-schedulerestarts
Python
Run your code to see results
No results yet
Run your code to check it against the examples, or submit to run every test.