Gradient clipping (global norm)

The problem, from first principles

01-outlier-detection (Math & Statistics) already warned that a single extreme value can badly distort a mean or a standard deviation. The same failure mode threatens gradient descent directly: one badly-scaled batch, or one region of an especially steep loss surface, can produce a gradient whose magnitude is enormous, and taking a full-size step in that direction can throw a model's weights somewhere far worse than where they started, sometimes badly enough that training never fully recovers, a real, common cause of a loss curve that suddenly spikes to nan partway through training.

Gradient clipping is the direct fix: measure how large the gradient is overall, and if it's larger than some chosen threshold, shrink it back down before it's ever used to update anything, while carefully preserving its DIRECTION, only the magnitude gets capped.

From theory to code

Theory computes one GLOBAL norm across every gradient array at once (treating them as if concatenated into a single vector), and, if that norm exceeds a threshold, rescales every gradient by the same factor so the new global norm is exactly the threshold.

Implement compute_global_norm(grads) first, then clip_grad_norm(grads, max_norm) on top of it.

Constraints

• compute_global_norm computes ONE number across ALL gradient arrays combined, not a separate norm per array.
• clip_grad_norm returns fresh arrays (copies), whether or not clipping was actually needed.
• Rescaling must use the SAME factor for every gradient array, preserving their relative proportions.

Hints

Open one at a time. Each gives away a little more than the last.