After pretraining, a model knows a lot but hasn't learned what humans prefer. Preference tuning fixes that — and two methods dominate: PPO and DPO.
PPO: the reinforcement-learning route
Proximal Policy Optimization was the classic recipe behind early instruction-tuned models. It works in two stages: first train a reward model on human comparisons (which of two answers is better?), then use reinforcement learning to push the model toward higher-reward outputs. It's powerful but fiddly — you're running an RL loop with a separate reward model, which is complex and can be unstable.
DPO: skip the RL loop
Direct Preference Optimization asks a sharper question: if the goal is to prefer good answers over bad ones, why not optimize that directly? DPO reformulates the objective so you train on preference pairs in one step — no separate reward model, no RL loop. It's simpler, cheaper, and often just as good.
DPO's popularity is itself a clue: the ranking signal, not the RL machinery, was doing most of the work all along.
When to use which
DPO is the default for most teams now — easier to run and stable. PPO (and newer RL variants) still wins when you need online exploration or a reward signal that's more nuanced than static pairs, especially for reasoning tasks where you reward correct outcomes. The trend is toward simpler preference methods where they suffice, and RL where genuine exploration pays off.
