Pass@k Estimation

The problem, from first principles

Code and math benchmarks have a checker: a response is simply right or wrong. Since models sample, one attempt per problem is a noisy measurement, and a useful question is "if I let the model try k times, how often does at least one attempt pass". The obvious estimate, draw exactly k samples per problem and count successes, has high variance. Better: draw n >= k samples, observe that c of them passed, and compute the probability that a random subset of size k of those n contains at least one success. That estimate is unbiased and uses all n samples.

From theory to code

Implement pass_at_k and mean_pass_at_k.

Constraints

• pass_at_k(n, c, k) is 1 - C(n - c, k) / C(n, k) for 1 <= k <= n and 0 <= c <= n. Return exactly 1.0 when n - c < k.
• Compute it as 1 - prod_{i = n - c + 1}^{n} (1 - k / i). Do not build binomial coefficients: they overflow for large n.
• mean_pass_at_k(ns, cs, k) averages pass_at_k over problems, where each problem can have a different number of samples. Return a Python float.
• k = 1 must give c / n.

Hints