Calibration Bins

Group predictions into equal-width probability bins on $[0,1][0,1]$ and compare, per bin, the average predicted probability with the observed rate of positives. y holds 0/1 labels and p the predicted probabilities. The last bin includes the value $1.01.0$; empty bins are skipped.

Implement solve(y, p, bins=10).

Returns. Return a list of (mean_confidence, observed_rate, count) tuples, one for each non-empty bin in increasing order. y must be 0/1, p must lie in $[0,1][0,1]$ and both must have the same length; otherwise ValueError is raised.

Examples

Example 1

Input:

solve([0, 0, 1, 1], [0.1, 0.2, 0.8, 0.9], 2)

Output:

[(0.15, 0.0, 2), (0.85, 1.0, 2)]

Example 2

Input:

solve([0, 1, 0, 1], [0.1, 0.2, 0.8, 0.9], 2)

Output:

[(0.15, 0.5, 2), (0.85, 0.5, 2)]