# Reinforcement Learning from Human Feedback (RLHF)

> Fine-tune with a human-preference-trained reward model as the signal — PPO pushes the policy toward high-reward outputs while a KL penalty prevents reward hacking.

- **Category**: Language Models
- **Subcategory**: Fine-Tuning
- **Canonical URL**: https://designpattern.fyi/patterns/rlhf/

---

## Description
**Intent**: Align model behavior with nuanced human preferences — helpfulness, harmlessness, truthfulness, tone — that supervised training data can't fully capture.

**Context**: Instruction fine-tuning teaches format and task completion. It doesn't capture what makes a response genuinely good by human standards — appropriate length, nuanced helpfulness, avoiding subtle harms. A learned reward model captures these preferences; RL optimizes against them.

**Solution**: Three-stage pipeline. (1) SFT — fine-tune the base on high-quality demonstrations. (2) Reward Model — train a separate scorer on human preference pairs using the Bradley-Terry model. (3) PPO — use PPO to update the SFT model to maximize reward model scores, with a KL divergence penalty against the SFT checkpoint to prevent the policy from exploiting reward model weaknesses.



## Use Cases
Producing the final aligned model for safety-critical or user-facing deployment where DPO's simpler approach is insufficient. Training models that must simultaneously optimize helpfulness, harmlessness, and honesty. The alignment foundation of GPT-4, Claude, and Gemini-class assistants.





## Trade-offs


### Advantages

- Strong alignment quality — the foundation of every major commercial aligned LLM

- Reward model captures nuanced human preferences that supervised labels can't express directly

- Can simultaneously optimize multiple alignment dimensions (helpfulness, safety, honesty)




### Considerations & Drawbacks

- Three-stage pipeline — SFT, reward model, PPO — is expensive and complex to tune

- PPO training is notoriously unstable and sensitive to hyperparameters, especially KL coefficient

- Reward hacking — the model learns to exploit reward model weaknesses rather than genuinely aligning