Note: Gaussian Processes, a distribution over functions instead of over parameters

The problem, from first principles

Every model in this curriculum so far learns a fixed, finite set of parameters — weight/bias, or a tree's structure — from the training data, then uses only those parameters to predict, with no built-in notion of "how sure am I." A Gaussian Process (GP) is a genuinely different kind of model: instead of committing to one function described by a handful of parameters, it maintains a distribution over every function consistent with the training data, and predicts by averaging over all of them, weighted by how well each one fits — which gives it a real, calibrated uncertainty estimate for free.

From theory to code

Implement rbf_kernel(input_a, input_b, length_scale, variance) and gp_predict(input_train, targets_train, input_test, length_scale, variance, noise). gp_predict returns both a mean prediction and a variance (uncertainty) at every test point — this is the whole point of a Gaussian Process, not an add-on.

Constraints

• rbf_kernel(input_a, input_b, ...) returns shape (n_a, n_b), the pairwise kernel similarity between every row of input_a and every row of input_b.
• gp_predict returns (mean, variance_out): mean shape (n_test,), variance_out shape (n_test,), and variance_out must never be negative even where floating-point error could push it slightly below 0.
• noise is added only to the training-training kernel's diagonal, never to train-test or test-test kernels.
• No Python loops over samples; use matrix operations throughout.

Hints

Open one at a time. Each gives away a little more than the last.