Shampoo: Accumulating Preconditioners
Intermediate
The problem, from first principles
Shampoo (Gupta et al., 2018) preconditions a matrix gradient on both sides, using a running sum of G G^T on the left and G^T G on the right. Those sums capture the curvature along each axis.
From theory to code
Implement accumulate_preconditioners(L, R, G), adding the two gram matrices of the gradient.
Constraints
- Return the pair (L, R).
Hints
Hint 1
Add G times G transposed to L, and G transposed times G to R.
Keywords
research-papersoptimizationpreconditioning
Python
Run your code to see results
No results yet
Run your code to check it against the examples, or submit to run every test.