LAMB: The Layer-wise Update
Intermediate
The problem, from first principles
The LAMB step is the Adam direction, multiplied by the layer's trust ratio and the learning rate. Each layer moves a fixed fraction of its own weight norm per step.
From theory to code
Implement lamb_update(w, r, lr), one LAMB step for a single layer.
Constraints
- Reuse the trust ratio definition from the previous question.
Hints
Hint 1
Multiply the direction by lr and by the trust ratio, then subtract from the weights.
Keywords
research-papersoptimizationlarge-batch
Python
Run your code to see results
No results yet
Run your code to check it against the examples, or submit to run every test.