Human Preferences: The Preference Probability

The problem, from first principles

Christiano et al. (2017) learned a reward function from human comparisons instead of hand-written rewards. Humans watch two short clips and say which is better. The model turns reward differences into a probability of preference, and that probability is fitted to the human choices.

From theory to code

Implement preference_probability(r_a, r_b), returning sigmoid(r_a - r_b).

Constraints

• Preference depends only on the reward difference.

Hints