LARS: The Layer Trust Ratio
Intermediate
The problem, from first principles
LARS (You et al., 2017) lets each layer take a step proportional to how large its weights are relative to its gradient. The layer's learning rate is scaled by this trust ratio, so large-batch training stays stable across layers.
From theory to code
Implement trust_ratio(w, g, eta, wd), the layer-wise local learning-rate multiplier.
Constraints
- The denominator includes the decay term.
Hints
Hint 1
Take the weight norm, multiply by eta, and divide by the gradient norm plus wd times the weight norm.
Keywords
research-papersoptimizationlarge-batch
Python
Run your code to see results
No results yet
Run your code to check it against the examples, or submit to run every test.