Reliability with pass^k

The problem, from first principles

For code generation, pass@k asks "if I try k times, does at least one attempt succeed?", which suits a user who can pick the best answer. An agent that acts on behalf of a customer has the opposite requirement: it must succeed every time, because an agent that cancels the wrong order one time in five is not usable. pass^k measures this: the probability that all k independent attempts at a task succeed. Estimated from n trials with c successes, it is the chance that a random size-k subset of those trials is all successes, C(c, k) / C(n, k). It drops quickly as k grows for an agent that is merely "usually right", which is exactly why it reveals unreliability that pass@1 hides.

From theory to code

Implement pass_hat_k and mean_pass_hat_k.

Constraints

• pass_hat_k(n, c, k) is C(c, k) / C(n, k) for 1 <= k <= n and 0 <= c <= n. Compute it as the product prod_{i=0}^{k-1} (c - i) / (n - i); return exactly 0.0 when c < k.
• mean_pass_hat_k(ns, cs, k) averages over tasks (each with its own n and c) and returns a float.
• For k = 1 the result is the ordinary success rate c / n.

Hints