FP16/BF16 Representable Range vs FP32, Why Naive FP16 Training Underflows

The problem, from first principles

03-memory-footprint-estimation and 01-quantize-float32-to-int8 both treated reduced precision as purely a storage question. Training in reduced precision is a different, sharper problem: gradients during real training routinely take on very small values (a well-trained network's gradients often shrink to 1e-6 or smaller as the loss flattens out) — and unlike int8's wide, evenly-spaced quantization range, float16 has a hard floor below which numbers don't round imprecisely, they vanish to exactly zero.

From theory to code

Implement cast_to_dtype(x, dtype), a real (not simulated) cast using NumPy's own dtype system, detect_underflow(original, casted), which flags every value that vanished entirely, and detect_overflow(casted), which flags every value that blew past the narrower dtype's representable maximum.

Constraints

• cast_to_dtype(x, dtype): dtype is "float16" or "float32".
• detect_underflow(original, casted): True only where original was genuinely nonzero AND casted rounded all the way to exactly 0.0 — a value that was already 0.0 doesn't count as "lost."
• detect_overflow(casted): True wherever a value is +inf or -inf (only possible after casting down to a narrower dtype).
• Both mask functions return boolean arrays matching their input's shape.

Hints

Open one at a time. Each gives away a little more than the last.