# Progressive Distillation

> Distill through intermediate model sizes rather than jumping from the largest teacher to the smallest target — each stage is a shorter compression jump with a better-matched teacher.

- **Category**: Language Models
- **Subcategory**: Model Distillation
- **Canonical URL**: https://designpattern.fyi/patterns/progressive-distillation/

---

## Description
**Intent**: Direct distillation from a very large teacher to a very small student loses too much quality in one step. Progressive distillation closes the capacity gap in stages, with each intermediate model serving as teacher for the next.

**Context**: Distilling a 70B teacher directly to a 1B student is a 70x compression in one pass — too large a jump for the student to faithfully approximate the teacher's distribution. Quality degrades sharply. Mid-size intermediates provide smoother knowledge gradients.

**Solution**: Define a distillation chain: Teacher (70B) → Intermediate (13B) → Intermediate (7B) → Target (1B). Each step is a manageable compression ratio where the student and teacher are close enough in capacity for effective knowledge transfer. Each stage uses standard knowledge distillation with soft labels. Intermediate checkpoints are themselves deployable production models.



## Use Cases
Extreme compression targets where direct large-to-small distillation degrades quality unacceptably. Model family development (70B → 13B → 7B → 3B → 1B) where each tier needs production quality. Research exploring theoretical compression limits.





## Trade-offs


### Advantages

- Better final quality than direct distillation at the same target compression ratio

- Intermediate checkpoints are independently deployable — a 13B and 7B model as byproducts

- Smoother knowledge gradient — each student has a capacity-matched teacher, not a 70x-larger one




### Considerations & Drawbacks

- Multiple training runs multiply total compute cost proportionally

- Pipeline management complexity grows with chain length

- Errors in an intermediate model can compound and degrade all downstream stages