Preference Loss Variants: IPO & SimPO

The problem, from first principles

DPO has two known weak points. Its logistic loss keeps rewarding larger margins forever, so on clean data it can push the policy far from the reference and overfit. And it needs a reference model and the log-ratio, which costs memory and ties the training signal to a quantity that does not match what the model is judged on at generation time. Two variants repair one thing each. IPO replaces the logistic loss with a squared error towards a fixed target margin, so the margin is regularized instead of maximized. SimPO drops the reference model, uses the average token log-probability so that response length cannot be exploited, and adds a margin gamma that the winner must beat the loser by.

From theory to code

Implement ipo_loss and simpo_loss.

Constraints

• ipo_loss(pi_chosen, pi_rejected, ref_chosen, ref_rejected, tau) is the mean of (h - 1 / (2 * tau)) ** 2 where h = (pi_chosen - ref_chosen) - (pi_rejected - ref_rejected). Inputs are summed sequence log-probabilities.
• simpo_loss(pi_chosen, pi_rejected, len_chosen, len_rejected, beta, gamma) is the mean of -log sigmoid(beta * (pi_chosen / len_chosen - pi_rejected / len_rejected) - gamma). Use np.logaddexp(0, -z).
• SimPO uses no reference model. len_* are the response token counts, positive integers.
• tau > 0, beta > 0, gamma >= 0. Return Python floats.

Hints