Direct Preference Optimization (DPO)

The problem, from first principles

Classic RLHF trains a reward model and then runs reinforcement learning against it while a KL penalty keeps the policy near a reference model. Direct Preference Optimization observes that the optimum of that KL-regularized objective has a closed form: the reward of a response is beta times the log-ratio between the policy and the reference. Substituting that reward into the Bradley-Terry likelihood removes the reward model and the reinforcement learning loop. What remains is a logistic loss on the difference of two log-ratios, trained directly on preference pairs with ordinary gradient descent.

From theory to code

Implement dpo_implicit_reward and dpo_loss.

Constraints

• Inputs are summed sequence log-probabilities (one float per response), as NumPy arrays of length n: pi_chosen, pi_rejected from the policy and ref_chosen, ref_rejected from the frozen reference.
• dpo_implicit_reward(pi, ref, beta) is beta * (pi - ref).
• dpo_loss(pi_chosen, pi_rejected, ref_chosen, ref_rejected, beta) is the mean of -log sigmoid(r_chosen - r_rejected) where each reward is the implicit reward. Use np.logaddexp(0, -d).
• beta > 0. Return Python floats.

Hints