Tool-Call Accuracy

The problem, from first principles

A strong signal of whether an agent behaved correctly is whether it made the right tool calls. Given a reference list of expected calls and the list the agent actually made, you can score them like retrieved items. A call counts as correct only if both the tool name and the arguments match exactly. Precision is the fraction of the agent's calls that were expected (penalizing unnecessary and wrong calls), recall is the fraction of expected calls the agent made (penalizing missing ones) and F1 combines them. Calls are compared as a multiset: making the same correct call twice when it was expected once earns credit for one of them and counts the other as extra. A second, more forgiving score looks at tool names only, to separate "chose the wrong tool" from "right tool, wrong arguments".

From theory to code

Implement tool_call_metrics.

Constraints

• Each call is a pair (name, args) with args a dict. Two calls are equal iff the names are equal and the argument dicts are equal (nested values included; key order does not matter).
• tool_call_metrics(predicted, expected) returns a dict with the float keys precision, recall, f1 (exact match on name and args) and name_f1 (match on names only), all computed from multiset intersections.
• precision = matches / len(predicted) and recall = matches / len(expected), with 0.0 when the denominator is zero. f1 = 2PR / (P + R) or 0.0 if P + R == 0. If both lists are empty every metric is 1.0.

Hints