Backprop through time (BPTT): vanishing and exploding gradient intuition

The problem, from first principles

[02-rnn-cell-backward]'s Theory section named the concern directly: grad_h_prev = grad_z @ weight_hh means that walking a gradient backward through N time steps of an RNN multiplies it by weight_hh a total of N times, once per step. [03-dl-training/05-why-deep-networks-work/03-vanishing-exploding-gradients] demonstrated this SAME compounding-multiplication effect for DEPTH, a fresh random matrix per layer, and found that even a modest per-layer scale factor compounds catastrophically across enough layers. Backpropagation Through Time (BPTT), the standard algorithm for training an RNN, has this EXACT same structural vulnerability, but with one crucial difference worth making precise: it's not N DIFFERENT random matrices multiplied together, it's the exact SAME weight_hh matrix, multiplied by itself N times in a row (weight_hh^N, in matrix-power notation).

This distinction matters mathematically: repeatedly multiplying by the SAME matrix is exactly the setup behind the "power iteration" method from linear algebra, and it has a sharp, well-understood consequence: as N grows large, weight_hh^N becomes increasingly DOMINATED by weight_hh's single largest-magnitude eigenvalue (its "spectral radius"). If that spectral radius is less than 1, weight_hh^N shrinks toward the zero matrix, gradients vanish. If it's greater than 1, weight_hh^N grows without bound, gradients explode. This is WHY vanilla RNNs are notoriously difficult to train on long sequences, and it's the exact motivation behind LSTM cell, forward (gating mechanism) and GRU cell, forward (simplified gating), both immediately following this question: both architectures were specifically designed with gated pathways that let gradients flow backward WITHOUT necessarily being repeatedly multiplied by the same weight matrix at every single step.

From theory to code

Implement bptt_gradient_norms(seq_len, weight_hh, grad_h_final), starting from the LAST time step's gradient and repeatedly multiplying by the SAME weight_hh matrix seq_len times (simplifying away the tanh derivative term from [02-rnn-cell-backward]'s full formula, to isolate weight_hh's own compounding effect specifically), recording the gradient's norm after every step.

Constraints

• weight_hh must be reused, UNCHANGED, at every single step (unlike [03-dl-training/05-why-deep-networks-work/03-vanishing-exploding-gradients]'s matrix_gradient_norms, which deliberately drew a FRESH random matrix per layer).
• The returned list must have length seq_len + 1: the starting norm, followed by one entry per step.
• Must demonstrate BOTH failure modes for an appropriately scaled weight_hh: a small-magnitude matrix causes the norm to vanish toward zero across enough steps; a large-magnitude matrix causes it to explode.

Hints