DPO: The Implicit Reward Margin

The problem, from first principles

DPO (Rafailov et al., 2023) skips the reward model and the RL loop. It shows that the optimal policy under a KL-constrained reward has an implicit reward, beta times the log-ratio against the reference. Preferences can then be learned directly from that margin.

From theory to code

Implement dpo_logit(logp_w, logp_l, ref_w, ref_l, beta), returning beta times the difference of the two log-ratios.

Constraints

• Both log-ratios are taken against the same reference model.

Hints