LARS: The Layer-wise Update
Intermediate
The problem, from first principles
After computing a layer's trust ratio, LARS scales the global learning rate by it and applies the decayed gradient. Every layer moves a comparable relative amount, even when their gradient magnitudes differ widely.
From theory to code
Implement lars_update(w, g, lr, eta, wd), one LARS step for a single layer.
Constraints
- Use the trust ratio from the previous question.
Hints
Hint 1
Compute the trust ratio, multiply it with lr, then subtract that times the decayed gradient from the weights.
Keywords
research-papersoptimizationlarge-batch
Python
Run your code to see results
No results yet
Run your code to check it against the examples, or submit to run every test.