Reward Modeling

The problem, from first principles

People are bad at assigning a score to a response but good at saying which of two responses is better. A reward model is trained from exactly that kind of data: pairs of a chosen and a rejected response to the same prompt. The Bradley-Terry model says the probability that the chosen response wins is a logistic function of the difference between their scalar rewards. Training maximizes the likelihood of the observed choices. Only the difference matters, so the reward scale and offset are arbitrary, which is why reward models are often normalized before use.

From theory to code

Implement reward_model_loss(r_chosen, r_rejected) and preference_accuracy(r_chosen, r_rejected).

Constraints

• r_chosen and r_rejected are float arrays of length n, the scalar reward of each response in each pair.
• reward_model_loss returns the mean of -log sigmoid(r_chosen - r_rejected). It must be stable for differences of magnitude 1000: use np.logaddexp(0, -d).
• preference_accuracy returns the fraction of pairs with r_chosen > r_rejected. Ties count as incorrect.
• Both return Python floats.

Hints