What do SFT, reward models and RLHF each solve?

The bottleneck

Pretraining learns text continuation rather than directly optimizing instruction fulfillment. InstructGPT adds demonstrations and comparative feedback during post-training. Demonstration quality, annotator preferences and task coverage shape behavior.

The mechanism

SFT learns from demonstrated answers. A reward model learns scores from answer rankings for the same prompt. RL then samples answers and optimizes reward, typically constraining deviation from a reference policy. This is one concrete pipeline; RLHF does not uniquely imply PPO.

J(π) = E[r(x,y)] − β · D_KL(π ∥ π_ref)

Examples and evidence

This simplified objective rewards better scores but penalizes deviation. Too little regularization can exploit reward errors; too much can prevent useful change. KL estimators, sampling and optimization details belong to the specific implementation.

New constraints

A learned reward is not proof of factual correctness. Preferences may favor length or tone, and optimization can amplify scoring loopholes. Validate using independent tasks, human review and out-of-distribution tests.