Calibration & ECE

The problem, from first principles

Accuracy says how often a model is right. Calibration says whether its confidence means anything: when a model says "90% sure", is it right about 90% of the time? An overconfident model is dangerous because users trust the stated confidence. To measure it, group predictions into confidence bins and compare, in each bin, the average confidence with the actual accuracy. The expected calibration error (ECE) is the average gap, weighted by how many predictions fall in each bin. Language models are famously well calibrated before fine-tuning and often overconfident afterwards, which makes this a standard post-training diagnostic.

From theory to code

Implement expected_calibration_error and brier_score.

Constraints

• confidences is an array in (0, 1] holding the model's probability for its predicted answer. correct is a 0/1 array of the same length.
• Use n_bins equal-width bins over [0, 1]: bin b covers (b / n_bins, (b + 1) / n_bins] (upper edge included). A confidence exactly 0 goes in the first bin.
• expected_calibration_error(confidences, correct, n_bins=10) returns sum over non-empty bins of (bin size / n) * |accuracy - mean confidence|.
• brier_score(probs, outcomes) is the mean of (probs - outcomes) ** 2 for binary outcomes.
• Return Python floats.

Hints