Early stopping: halting training at the best validation checkpoint

The problem, from first principles

04-full-boosting-loop's n_trees and every other training loop in this curriculum runs for a fixed, predetermined number of iterations. In practice, training longer doesn't always help: past some point, a model starts fitting the training set's specific noise rather than the real pattern, and validation loss (measured on data the model hasn't fit to) starts getting worse even as training loss keeps improving — the same overfitting 03-bias-variance-tradeoff's high-variance regime describes.

From theory to code

Implement best_epoch_with_min_delta, early_stopping_should_stop, and train_with_early_stopping(step_fn, max_epochs, patience, min_delta=0.0). step_fn(epoch) -> (state, val_loss) is a black box: one epoch of training plus a validation-loss check, for whatever model. This question doesn't care what model it is, just when to stop training it.

Constraints

• best_epoch_with_min_delta(history, min_delta=0.0) returns the index of the best (lowest) loss, where "best" requires beating the previous best by more than min_delta.
• early_stopping_should_stop(history, patience, min_delta=0.0) returns True once patience epochs have passed with no qualifying improvement; False for an empty history.
• train_with_early_stopping returns a dict with best_state, best_loss, history, and stopped_epoch.
• The returned model is whichever epoch's state had the best validation loss, never necessarily the last epoch's.

Hints

Open one at a time. Each gives away a little more than the last.