# Continuous Model Evaluation

> Ongoing pipeline to detect model performance degradation over time

- **Category**: Machine Learning
- **Subcategory**: Serving and Operational Resilience
- **Canonical URL**: https://designpattern.fyi/machine_learning/continuous-model-evaluation/

---

## Description
**Intent**: Detect when a model's real-world performance degrades after deployment due to input distribution shifts or relationship changes between inputs and outcomes.

**Context**: A model's real-world performance can quietly degrade after deployment—the input distribution can shift, or the relationship between inputs and true outcome can change. Without ongoing evaluation, this decay goes unnoticed until it causes damage.

**Solution**: Set up an ongoing pipeline that logs live predictions, reconciles them against actual outcomes once available, and computes evaluation metrics on a rolling basis. This catches degradation automatically, distinguishes genuine decay from normal noise, and provides evidence triggering retraining decisions.



## Use Cases
- Production ML systems in dynamic environments
- Models subject to data drift or concept drift
- High-stakes predictions where degradation is costly
- Any production system requiring ongoing monitoring






## Trade-offs


### Advantages

- Automatic detection of performance degradation

- Distinguishes real decay from normal noise

- Provides evidence for retraining decisions

- Can slice evaluation by relevant segments




### Considerations & Drawbacks

- Depends on reliable ground-truth collection

- Ground truth can be slow, incomplete, or expensive

- Requires infrastructure for logging and reconciliation

- Needs alerting and escalation processes







---
**Reference**: [Original Source](https://github.com/GoogleCloudPlatform/ml-design-patterns)