Large Minibatch SGD: Gradual Warmup
Beginner
The problem, from first principles
Goyal et al. also warm up the large learning rate gradually. Starting from a small rate and ramping up over the first few epochs avoids early divergence when the scaled rate is high.
From theory to code
Implement gradual_warmup_lr(target, t, warmup), the linear ramp to the target then hold.
Constraints
- Steps are numbered from one.
Hints
Hint 1
Multiply the target by the smaller of one and the step over the warmup length.
Keywords
research-papersoptimizationlarge-batch
Python
Run your code to see results
No results yet
Run your code to check it against the examples, or submit to run every test.