LAMB: The Adam Direction
Beginner
The problem, from first principles
LAMB's direction is Adam's normalized moment ratio, plus decoupled weight decay on the weights. It is the quantity whose norm the trust ratio compares against the weights.
From theory to code
Implement lamb_direction(m_hat, v_hat, w, wd, eps), the Adam direction with decay.
Constraints
- The decay term uses the weights themselves.
Hints
Hint 1
Divide the first moment by the square root of the second plus eps, then add wd times the weights.
Keywords
research-papersoptimizationlarge-batchadam
Python
Run your code to see results
No results yet
Run your code to check it against the examples, or submit to run every test.