InstructGPT: The Pairwise Reward Loss
Intermediate
The problem, from first principles
InstructGPT (Ouyang et al., 2022) trains a reward model from human rankings. Labelers compare two responses, and the reward model is trained so the preferred response scores higher. The loss only depends on the difference of the two scores.
From theory to code
Implement pairwise_reward_loss(r_chosen, r_rejected), returning -log(sigmoid(r_chosen - r_rejected)).
Constraints
- Use a numerically stable form such as
logaddexp.
Hints
Hint 1
Let d = r_chosen - r_rejected. Then -log(sigmoid(d)) = log(1 + exp(-d)), which np.logaddexp(0, -d) computes safely.
Keywords
research-paperstransformersllmalignmentrlhf
Python
Run your code to see results
No results yet
Run your code to check it against the examples, or submit to run every test.