Large Minibatch SGD: The Linear Scaling Rule
Beginner
The problem, from first principles
Goyal et al. (2017) train with very large minibatches by scaling the learning rate linearly with the batch size. Doubling the batch doubles the step, since the gradient estimate is averaged over twice as many examples.
From theory to code
Implement scaled_lr(base_lr, batch, base_batch), the linearly scaled learning rate.
Constraints
- Return the scaled rate as a number.
Hints
Hint 1
Multiply the base learning rate by the ratio of the new batch size to the base batch size.
Keywords
research-papersoptimizationlarge-batch
Python
Run your code to see results
No results yet
Run your code to check it against the examples, or submit to run every test.