Sophia: The Clipped Step
Intermediate
The problem, from first principles
Sophia (Liu et al., 2023) uses a diagonal Hessian estimate to set a per-coordinate step size, then clips each coordinate's step to a fixed range. The clip bounds the worst-case step even where the curvature estimate is poor.
From theory to code
Implement sophia_step(m, h, gamma, eps), the Hessian-scaled momentum step with per-entry clipping.
Constraints
- The denominator is floored at eps.
Hints
Hint 1
Divide the momentum by the larger of gamma times the Hessian and eps, then clip to minus one and one.
Keywords
research-papersoptimizationsecond-order
Python
Run your code to see results
No results yet
Run your code to check it against the examples, or submit to run every test.