# Direct Preference Optimization (DPO) > Align model outputs to human preferences using chosen/rejected pairs — no reward model, no PPO, just a classification loss that directly shapes the policy. - **Category**: Language Models - **Subcategory**: Fine-Tuning - **Canonical URL**: https://designpattern.fyi/patterns/dpo/ --- ## Description **Intent**: Align model behavior with human preferences more simply than RLHF — no reward model to train, no RL instability, just supervised training on preference pairs. **Context**: RLHF requires training a separate reward model and running PPO reinforcement learning — expensive, unstable, and sensitive to hyperparameters. DPO derives a mathematically equivalent alignment objective that can be optimized directly from preference pairs using a standard supervised loss. **Solution**: Collect preference pairs: for each prompt, a chosen response (human-preferred) and a rejected response (human-dispreferred). Train the model with the DPO loss: increase log probability of chosen responses and decrease rejected ones, relative to a reference model (the SFT checkpoint). The reference model provides an implicit KL regularizer that keeps the policy close to the SFT baseline without explicit RL. ## Use Cases Aligning instruction-tuned models with human preferences after SFT. Reducing harmful, verbose, or low-quality outputs. Any alignment task where preference pair data exists and RLHF complexity is unwarranted. ## Trade-offs ### Advantages - No reward model to train and maintain — dramatically simplifies the alignment pipeline - Stable training dynamics — standard supervised learning, no PPO instability - Competitive alignment quality with RLHF at a fraction of the infrastructure cost ### Considerations & Drawbacks - Quality depends heavily on preference data quality — noisy or inconsistent labels degrade alignment - Reference model must be kept accessible during training for the implicit KL computation - May underperform full RLHF on complex multi-dimensional alignment objectives