DPO: The Preference Loss
Advanced
The problem, from first principles
The DPO loss is a logistic loss on the implicit reward margin. It is the same as the reward-model loss from the reward-modelling question, but applied directly to the policy's log-probabilities.
From theory to code
Implement dpo_loss(logit), returning -log(sigmoid(logit)) in a numerically stable form.
Constraints
- Use
logaddexpto avoid overflow for large magnitudes.
Hints
Hint 1
The loss equals log(1 + exp(-logit)), which np.logaddexp(0, -logit) computes safely.
Keywords
research-papersreinforcement-learningalignmentdpo
Python
Run your code to see results
No results yet
Run your code to check it against the examples, or submit to run every test.