Multiple-Choice Likelihood Scoring

The problem, from first principles

Many benchmarks ask a model to pick one of several answers. A language model does not output "B"; it assigns probabilities to text. The usual trick is to score each candidate continuation by the log-probability the model gives it and choose the highest. That immediately raises a fairness problem: a longer answer has more tokens and each token multiplies in another probability below one, so long answers are penalized for length alone. Normalizing by length removes most of that bias but introduces a different preference, so the choice of normalization is part of the benchmark definition and can change which model wins.

From theory to code

Implement choice_scores, pick_choice and multiple_choice_accuracy.

Constraints

• logprobs is a 1-D array holding the summed token log-probability of each candidate. lengths holds each candidate's length (tokens or characters) as positive numbers.
• choice_scores(logprobs, lengths, mode) returns logprobs for mode='sum' and logprobs / lengths for mode='mean'. Any other mode raises ValueError.
• pick_choice(logprobs, lengths, mode) returns the index of the highest score. Ties go to the lowest index (np.argmax behaviour).
• multiple_choice_accuracy(all_logprobs, all_lengths, labels, mode) takes lists with one entry per question and returns the fraction picked correctly as a float.

Hints