Exact Match & Token F1

The problem, from first principles

For short-answer tasks the simplest metric is whether the model's answer equals the reference. Taken literally it is brutal: "The Eiffel Tower." and "eiffel tower" are the same answer to a human and different strings to a computer. Benchmarks therefore normalize both sides first (lowercase, strip punctuation and articles, collapse whitespace) and then compare. Exact match is all-or-nothing, so a partially correct answer scores zero. Token F1 gives partial credit by measuring the overlap of words between prediction and reference, which is why question-answering leaderboards usually report both.

From theory to code

Implement normalize_answer, exact_match and token_f1.

Constraints

• normalize_answer(s): lowercase, delete every character in string.punctuation, remove the standalone words a, an and the, then collapse runs of whitespace to single spaces and strip.
• exact_match(prediction, golds) is 1.0 if the normalized prediction equals the normalized form of any reference in golds, else 0.0.
• token_f1(prediction, gold) splits both normalized strings on whitespace and counts overlapping tokens as a multiset intersection. If either side is empty the score is 1.0 when both are empty and 0.0 otherwise. Otherwise precision is overlap / len(pred tokens), recall is overlap / len(gold tokens) and F1 is their harmonic mean, or 0.0 if the overlap is 0.
• Use only the standard library plus NumPy.

Hints