Success Rate & Confidence Intervals

The problem, from first principles

The headline number for an agent on a benchmark is its success rate: tasks solved divided by tasks attempted. But a rate from 20 tasks is far less trustworthy than one from 2,000, and comparing "62% versus 58%" without error bars is how teams fool themselves. A confidence interval shows the uncertainty. The simple normal approximation misbehaves for small samples and for rates near 0 or 1 (it can produce intervals outside [0, 1]). The Wilson score interval fixes this and is the standard recommendation for proportions.

From theory to code

Implement success_rate and wilson_interval.

Constraints

• success_rate(successes, n) returns successes / n (0.0 when n == 0).
• wilson_interval(successes, n, z=1.96) returns (low, high): with p = successes / n, centre (p + z**2 / (2n)) / (1 + z**2 / n) and half-width z * sqrt(p(1 - p)/n + z**2 / (4n**2)) / (1 + z**2 / n); the interval is centre -+ half-width.
• For n == 0 return (0.0, 1.0): no data means no information.

Hints