DPO: The Implicit Reward
Intermediate
The problem, from first principles
DPO's key identity is that the reward a policy implicitly assigns to a response is beta times its log-ratio against the reference. Looking at this implicit reward explains why DPO needs no separate reward model.
From theory to code
Implement implicit_reward(logp, ref, beta), returning beta * (logp - ref).
Constraints
- Works element-wise on arrays.
Hints
Hint 1
Subtract the reference log-probabilities and multiply by beta.
Keywords
research-papersreinforcement-learningalignmentdpo
Python
Run your code to see results
No results yet
Run your code to check it against the examples, or submit to run every test.