Validation curves: training/validation error vs. one hyperparameter
The problem, from first principles
08-learning-curves fixed the model and swept the amount of data to ask "would more data help?" There's a second, equally natural question: fixing the amount of data, how does one hyperparameter, polynomial degree, tree depth, number of estimators, actually trade off underfitting against overfitting for this dataset? 03-bias-variance-tradeoff explained that tradeoff in the abstract; a validation curve makes it visible for one real, tunable number at a time.
This matters beyond curiosity: 04-grid-search and 05-random-search already search over hyperparameters automatically, but they hand back a single winning value with no visibility into why it won, or how sensitive the result is to picking a slightly different one. A validation curve is the same search made visible, one hyperparameter, plotted.
From theory to code
Theory describes the expected shape: training error trending down as complexity increases, validation error tracing a U, dropping, bottoming out, then climbing again as the model starts overfitting the fixed training set.
Producing that curve needs the same mechanical step at every candidate value: train on the same fixed data with that one hyperparameter setting, measure both errors, record them. Implement validation_curve(fit_and_evaluate_fn, param_values), which owns exactly that sweep; the caller-supplied fit_and_evaluate_fn does the actual training and measuring.
Constraints
fit_and_evaluate_fn(param_value) -> (train_error, val_error): trains a model with this one hyperparameter value, against the same, fixed training set every call, and returns both errors.param_values: a list of values (numbers, or anythingfit_and_evaluate_fnaccepts), not necessarily sorted.- Call
fit_and_evaluate_fnexactly once per entry inparam_values, in the order given. - Returns
(train_errors, val_errors), each shape(len(param_values),), in the same order asparam_values. - Do not swap which returned value goes into which output array.
Hints
Open one at a time. Each gives away a little more than the last.
Hint 1
Structurally this is the exact same sweep as 08-learning-curves's learning_curve, only what varies between calls is different, a hyperparameter value here instead of a training-set size.
Hint 2
Keep two plain Python lists while looping over param_values, appending the unpacked (train_error, val_error) tuple each time, then wrap both in np.array right before returning.
Keywords
No results yet
Run your code to check it against the examples, or submit to run every test.