Adafactor: Memory Saved by Factoring
Beginner
The problem, from first principles
A full second-moment matrix for an n by m weight stores n times m numbers. Adafactor stores one vector of length n and one of length m, which is far smaller for large layers.
From theory to code
Implement factored_memory(n, m), the number of stored entries for a factored second moment.
Constraints
- Return an integer.
Hints
Hint 1
Add the two dimensions together.
Keywords
research-papersoptimizationmemory
Python
Run your code to see results
No results yet
Run your code to check it against the examples, or submit to run every test.