Learning to Summarize: The Sequence KL
Intermediate
The problem, from first principles
Learning to summarize from human feedback (Stiennon et al., 2020) fine-tunes a summarizer with reinforcement learning, while keeping it close to a supervised reference. The penalty is a per-sequence KL estimate: how much more likely each sampled token is under the policy than under the reference.
From theory to code
Implement per_token_kl(logp_policy, logp_ref), summing the per-token log-ratio over the sequence.
Constraints
- The estimate is taken on sampled tokens.
Hints
Hint 1
Subtract the reference log-probabilities from the policy log-probabilities and sum.
Keywords
research-papersreinforcement-learningalignmentrlhfsummarization
Python
Run your code to see results
No results yet
Run your code to check it against the examples, or submit to run every test.