Learning to Summarize: The Win Rate
Beginner
The problem, from first principles
The paper compares summaries by win rate: how often a human or model prefers one system's summary over another's. Counting ties as half a win makes the metric symmetric when two systems are tied.
From theory to code
Implement win_rate(a, b), returning the fraction of prompts where a beats b, with ties counting half.
Constraints
- Both score arrays cover the same prompts.
Hints
Hint 1
Average the strict wins and add half the average of the ties.
Keywords
research-papersreinforcement-learningalignmentrlhfsummarization
Python
Run your code to see results
No results yet
Run your code to check it against the examples, or submit to run every test.