Gradient Clipping: Clipping by Norm
Beginner
The problem, from first principles
Pascanu et al. (2013) observed that recurrent gradients can explode. Clipping the gradient norm to a threshold bounds the step while keeping its direction.
From theory to code
Implement clip_by_norm(g, threshold), rescaling the gradient only when it is too large.
Constraints
- Gradients under the threshold pass through unchanged.
Hints
Hint 1
If the norm exceeds the threshold, multiply the gradient by threshold over the norm.
Keywords
research-papersoptimizationstability
Python
Run your code to see results
No results yet
Run your code to check it against the examples, or submit to run every test.