Compare an Agent With and Without Experience
The problem, from first principles
Building a whole experience-retrieval system (this track's earlier questions) is only worth it if it actually helps. Proving that requires a real A/B comparison: run the same set of tasks twice, once with the agent allowed to draw on past experience and once without, and measure the difference — not just "did it get better on average," but specifically which tasks experience actually rescued.
From theory to code
You're given two same-length lists of per-task success (True/False), for the same tasks in the same order: with_experience and without_experience. Implement compare_experience_impact(with_experience, without_experience). Compute:
"with_success_rate": fraction of tasks the with-experience agent succeeded on."without_success_rate": same, for the without-experience agent."tasks_only_with_helped": count of tasks where the with-experience agent succeeded and the without-experience agent failed on that same task."delta":with_success_rate - without_success_rate.
Return the dict with exactly these four keys.
Constraints
- 1 to 1000 tasks; both lists always the same length.
Hints
Hint 1
sum(a_list_of_booleans) / len(...) gives a success rate directly, since True counts as 1 and False as 0 in a sum — no need for an explicit count-then-divide loop.
Hint 2
tasks_only_with_helped is specifically about tasks where only the with-experience agent succeeded — zip the two lists together and count positions where the with-experience value is True and the without-experience value is False for that exact same task, not just "count successes in each list separately."
Keywords
No results yet
Run your code to check it against the examples, or submit to run every test.