02-classification-metrics's AUC only ever asks about ranking: does class 1 tend to score higher than class 0. It's silent on whether a specific predicted number, 0.8, means what it claims to mean. That question, whether a stated probability is a trustworthy frequency, is its own separate property called calibration, and a model can be excellent by AUC while being badly calibrated: its 0.8 might really behave like 0.95, or like 0.6, when you look at what actually happens to everyone it assigned that number to.
This is worth checking whenever the probability itself gets used downstream, not just the predicted class, sizing a bet, setting an insurance premium, deciding how urgently to flag a medical result. All of those need 0.8 to genuinely mean "about 80% of the time," not just "more likely than not."
Theory groups predictions by their predicted probability and compares, within each group, the average predicted probability against the actual fraction that were class 1. Implement reliability_diagram(labels, probabilities, n_bins), which builds exactly those per-bin numbers, and expected_calibration_error(labels, probabilities, n_bins), which compresses the whole diagram into the single weighted-average gap Theory describes.
labels: {0, 1}-valued, same shape as probabilities.probabilities: predicted P(class=1), values in [0, 1].reliability_diagram returns (bin_confidences, bin_accuracies, bin_counts), each shape (n_bins,).[0, 1]; a sample with probability p falls in bin i if edges[i] <= p < edges[i+1], except the last bin, which also includes p == 1.0 exactly.bin_counts[i] == 0) leaves bin_confidences[i] and bin_accuracies[i] at 0, not nan and not skipped.expected_calibration_error returns a single float: the sample-count-weighted average of |accuracy - confidence| across bins, 0 for perfect calibration.Open one at a time. Each gives away a little more than the last.
The mean of a {0, 1}-valued array is exactly the fraction of entries that are 1. That's not a coincidence you need a separate formula for, it's exactly what "actual fraction of class 1 in this bin" already is.
np.linspace(0, 1, n_bins + 1) gives the bin edges. Every bin except the last uses edges[i] <= p < edges[i+1]; the last bin needs <= on both ends, or a probability of exactly 1.0 falls into no bin at all.
Guard the per-bin mean behind if bin_counts[i] > 0. Since expected_calibration_error weights each bin by bin_count / n_total, an empty bin's leftover 0 confidence/accuracy contributes weight 0 regardless, no special-casing needed downstream.
Click "Run Tests" to test your implementation