Reading a loss curve: spotting training instability before it diverges

The problem, from first principles

[04-seq-modeling/03-recurrent-neural-networks/03-bptt-vanishing-exploding] demonstrated exploding gradients numerically, in isolation. In a REAL training run, the first visible symptom of exactly that kind of instability is usually a sudden, sharp SPIKE in the loss value, sometimes recovering on its own a few steps later, sometimes marking the start of a full, irrecoverable divergence (loss climbing toward infinity, or the model's weights becoming NaN). Catching an EARLY spike, before it turns into full divergence, is one of the most practically useful skills in actually training large models: catching it early might mean simply resuming from [08-resume-from-checkpoint]'s last good checkpoint with a lower learning rate, while catching it LATE can mean losing days of wasted compute to a run that silently diverged hours ago.

This question builds two concrete diagnostics directly from a raw loss history: a SPIKE detector (comparing each individual loss value against its own recent local baseline) and a DIVERGENCE detector (comparing a longer recent trend against the trend just before it), the same two questions a human staring at a live loss curve would actually be asking.

From theory to code

Implement detect_loss_spikes(loss_history, window, spike_ratio) (flagging any step whose loss exceeds spike_ratio times the average of the preceding window steps) and is_diverging(loss_history, window) (comparing the average of the most recent window steps against the window steps before that).

Constraints

• A spike at step i is compared against the average of steps i - window through i - 1 (the LOCAL baseline immediately preceding it), never against the very start of training or the whole history.
• No spike can be flagged before at least window steps of history exist to compute a baseline from.
• is_diverging compares the MOST RECENT window-step average against the window-step average immediately BEFORE it, returning False (not an error) if fewer than 2 * window total steps exist yet.

Hints

spikes = []
for i in range(window, len(loss_history)):
    baseline = sum(loss_history[i-window:i]) / window
    if loss_history[i] > spike_ratio * baseline:
        spikes.append(i)
return spikes

if len(loss_history) < 2 * window:
    return False
recent_mean = sum(loss_history[-window:]) / window
earlier_mean = sum(loss_history[-2*window:-window]) / window
return recent_mean > earlier_mean