# Knowledge Distillation

> Train a small student model on a large teacher's soft output probabilities — not just hard labels — so it learns the teacher's uncertainty structure, not just its answers.

- **Category**: Language Models
- **Subcategory**: Model Distillation
- **Canonical URL**: https://designpattern.fyi/patterns/knowledge-distillation/

---

## Description
**Intent**: Compress a large expensive model's knowledge into a smaller deployable one without training from scratch — soft labels carry far richer signal than one-hot targets.

**Context**: Training a small model from scratch on the same task consistently underperforms the teacher because hard labels encode only the correct answer. The teacher's output probability distribution over all classes encodes which wrong answers are "almost right" and why — richer signal the student can learn from.

**Solution**: Run the teacher on the training data at temperature τ > 1 to soften its output distribution (spreading probability mass across near-correct classes). Train the student using a weighted combination: α × KL(student || teacher soft labels) + (1-α) × cross-entropy(student || hard labels). The soft labels teach the student the teacher's learned similarity structure. At inference the student uses τ = 1.



## Use Cases
Deploying frontier-quality reasoning at edge or mobile scale. Reducing inference cost while preserving quality. Creating domain-specialized small models from general large ones where training from scratch is impractical.





## Trade-offs


### Advantages

- Student trained on soft labels outperforms same-architecture models trained on hard labels alone

- Soft probability distributions encode the teacher's learned class similarity structure

- Works across different student and teacher architectures — no structural coupling required




### Considerations & Drawbacks

- Requires teacher inference to generate soft labels over the full training set — adds upfront cost

- Student cannot exceed teacher quality — distillation compresses, it does not amplify

- Temperature τ is empirical and dataset-dependent — needs tuning per task