InstructGPT: The KL-Penalized Reward
Intermediate
The problem, from first principles
Optimizing a reward model alone makes the policy drift into strange outputs that exploit the reward. InstructGPT keeps the policy close to the supervised model by subtracting a penalty for how far its log-probabilities move from the reference.
From theory to code
Implement kl_penalized_reward(reward, logp_policy, logp_ref, beta), which subtracts beta times the log-probability difference from the reward.
Constraints
- Positive
logp_policy - logp_refmeans the policy has moved toward the response.
Hints
Hint 1
The penalty is beta * (logp_policy - logp_ref); subtract it from the reward.
Keywords
research-paperstransformersllmalignmentrlhf
Python
Run your code to see results
No results yet
Run your code to check it against the examples, or submit to run every test.