Human Preferences: The Reward Model Loss
Intermediate
The problem, from first principles
Training the reward model is a binary classification problem: given two segments and a human label, maximize the probability of the label. The loss is the negative log-likelihood of that label under the preference model.
From theory to code
Implement reward_model_nll(y, r_a, r_b), the negative log-likelihood of the human label.
Constraints
- Clamp probabilities away from 0 and 1 for numerical safety.
Hints
Hint 1
Compute p = sigmoid(r_a - r_b), then the binary cross-entropy against y.
Keywords
research-papersreinforcement-learningalignmentreward-learning
Python
Run your code to see results
No results yet
Run your code to check it against the examples, or submit to run every test.