An LLM-as-Judge Harness for Agent Transcripts
The problem, from first principles
A real LLM-as-judge sends the whole transcript to a model with a scoring rubric and asks it to grade the run. To make this exercise deterministic and checkable without touching a model — the same philosophy as the accuracy(r) simulator in Reproduce "Lost in the Middle" and Compare Orderings — you're given a deterministic stand-in: a rubric of text patterns, each worth some number of points (positive or negative), and the job is computing the score a judge following that exact rubric mechanically would produce.
From theory to code
You're given transcript ((role, text) turns making up one agent run, in order) and rubric (a substring pattern mapped to a point value, which may be negative). Implement score_transcript(transcript, rubric). For each turn's text, add every rubric pattern's point value to a running total if that pattern appears anywhere in that turn's text — each pattern counts at most once per turn, even if it appears multiple times within that turn's text.
Return the total score across the whole transcript.
Constraints
- 0 to 200 turns; 0 to 50 rubric patterns; matching is case-sensitive, plain substring.
Hints
Hint 1
For each turn, loop over every rubric pattern once and check pattern in text — a plain in check naturally only tells you "does it appear," not "how many times," which is exactly the "at most once per turn" rule this problem wants.
Hint 2
Negative point values need no special handling — total += points works identically whether points is positive or negative, since addition already does the right thing either way.
Keywords
No results yet
Run your code to check it against the examples, or submit to run every test.