Lookahead: The Slow Weight Sync
Beginner
The problem, from first principles
Lookahead (Zhang et al., 2019) wraps any inner optimizer. It keeps a slow copy of the weights, runs k fast steps, then moves the slow weights part of the way toward the result.
From theory to code
Implement lookahead_sync(slow, fast, alpha), the interpolation of slow weights toward fast ones.
Constraints
- alpha is between 0 and 1.
Hints
Hint 1
Add alpha times the difference between the fast and slow weights to the slow weights.
Keywords
research-papersoptimizationmeta-optimizer
Python
Run your code to see results
No results yet
Run your code to check it against the examples, or submit to run every test.