Adafactor: The Relative Step Size
Beginner
The problem, from first principles
Adafactor can run without a learning rate by using a relative step that decays with the square root of the step count, capped at a fixed maximum.
From theory to code
Implement relative_step(t), the capped inverse-square-root schedule.
Constraints
- The cap is 0.01.
Hints
Hint 1
Take the minimum of 0.01 and one over the square root of the step.
Keywords
research-papersoptimizationmemory
Python
Run your code to see results
No results yet
Run your code to check it against the examples, or submit to run every test.