LAMB: The Trust Ratio
Intermediate
The problem, from first principles
LAMB (You et al., 2019) combines Adam's per-parameter scaling with a LARS-style layer trust ratio. The ratio compares the weight norm to the norm of the Adam direction, so each layer's step is sized relative to its weights.
From theory to code
Implement lamb_trust_ratio(w, r), the ratio of the weight norm to the update norm.
Constraints
- Return a Python float.
Hints
Hint 1
Divide the norm of the weights by the norm of the update direction.
Keywords
research-papersoptimizationlarge-batch
Python
Run your code to see results
No results yet
Run your code to check it against the examples, or submit to run every test.