Note: Bayesian optimization for hyperparameter search

The problem, from first principles

04-grid-search and 05-random-search both pick their next hyperparameter configuration blindly — a fixed grid or a random draw — with no memory of how earlier configurations actually scored. When each evaluation means training a real model (expensive), that blindness wastes evaluations on configurations a smarter search would have skipped entirely.

From theory to code

Implement expected_improvement(mean, std, best_so_far, xi=0.01) and propose_next_point(input_train, targets_train, candidates, length_scale, variance, noise, xi=0.01). Reuse 05-gaussian-processes's gp_predict, treating "hyperparameters in, validation score out" as the unknown function a Gaussian Process learns.

Constraints

• expected_improvement returns one score per candidate, always >= 0, and exactly 0 wherever std == 0.
• propose_next_point returns a single integer index into candidates: whichever one maximizes expected improvement.
• best_so_far is the best value actually observed among evaluated points, not the GP's own prediction.

Hints

Open one at a time. Each gives away a little more than the last.