Why can DPO omit a separate reward model?

The bottleneck

Reward-model RLHF includes reward fitting, online sampling and policy optimization. DPO uses the relationship in KL-regularized reward maximization to turn preference learning into a classification-like policy objective, reducing separate training components.

The mechanism

For a winner y_w and loser y_l under prompt x, compare log-probability ratios between current and reference policies. Increase the winner’s relative advantage without first fitting a separate scalar reward network. The derivation assumes a preference model; preference data and a reference policy are still needed.

L_DPO = −log σ(β{log[π(y_w|x)/π_ref(y_w|x)] − log[π(y_l|x)/π_ref(y_l|x)]})

Examples and evidence

If the winner’s ratio to reference is 2 and the loser’s is 1, the margin is log 2; reversed it is −log 2. The objective separates pairs. It does not guarantee an increase in absolute answer probability or directly judge factual truth.

New constraints

New policy behavior outside offline preference coverage can still fail. Noisy labels, length bias, β and the reference choice matter. Meta reports SFT and DPO for Llama 3.1; this does not establish every company’s training recipe.