Learning curves: training/validation error vs. dataset size

The problem, from first principles

Say a model isn't doing as well as you'd like. There's an obvious, expensive fix: go collect more training data. But is it actually going to help? 03-bias-variance-tradeoff diagnosed underfitting versus overfitting by comparing several differently-trained models against the truth. That's useful, but it doesn't directly answer the question a manager or a budget actually cares about: "if I spend the next month labeling 10x more examples, will this specific model get meaningfully better?"

A learning curve answers exactly that, and cheaply: train the same model (same hyperparameters, same complexity) on progressively larger slices of the data you already have, and watch how its training error and validation error move as the slice grows. The shape of those two curves, not their absolute values, tells you whether more data is worth collecting at all.

From theory to code

Theory describes two curve shapes: both errors plateauing together at a high, similar value (more data won't help, the model itself is too simple), or validation error visibly dropping and closing in on training error as the training set grows (more data is helping, and might keep helping).

Producing either curve needs the same mechanical step repeated at every size: train on that many points, measure both errors, record them. Implement learning_curve(fit_and_evaluate_fn, train_sizes), which owns exactly that sweep, the caller-supplied fit_and_evaluate_fn does the actual training and measuring.

Constraints

• fit_and_evaluate_fn(n_samples) -> (train_error, val_error): trains a model on the first n_samples training points and returns both its training error and its (fixed, held-out) validation error.
• train_sizes: a list of integers, not necessarily sorted.
• Call fit_and_evaluate_fn exactly once per entry in train_sizes, in the order given, never sorted or deduplicated.
• Returns (train_errors, val_errors), each shape (len(train_sizes),), one entry per size, in the same order as train_sizes.
• Do not swap which returned value goes into which output array.

Hints

Open one at a time. Each gives away a little more than the last.