# Bias-Variance Tradeoff

> Model complexity as a dial between underfitting (high bias) and overfitting (high variance). The CAP equivalent of ML.

- **Category**: Trade-offs
- **Subcategory**: Machine Learning
- **Canonical URL**: https://designpattern.fyi/trade_offs/bias-variance-tradeoff/

---

## Description
**Intent**: Balance model complexity to achieve optimal generalization performance. Too simple = underfits (high bias). Too complex = overfits (high variance). Sweet spot in middle.

**Context**: You are training a machine learning model. A linear model might be too simple to capture patterns (high bias). A deep neural network might memorize training data (high variance). The bias-variance tradeoff shows that total error = bias² + variance + irreducible error. Minimizing one often increases the other.

**Solution**: Start simple, increase complexity gradually. Use cross-validation to detect overfitting. Apply regularization (L1/L2, dropout) to reduce variance. Use ensemble methods to balance both. Monitor training vs. validation performance curves. The goal is not zero training error, but minimal validation error.



## Use Cases
Training a fraud detection model. Too simple model like logistic regression misses complex patterns (high bias). Too complex model like deep net flags legitimate transactions as fraud (high variance). Solution uses gradient boosting with regularization and cross-validation.



## Implementation Example

```python
// Bias-Variance in practice: Regularization as the complexity dial

from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import cross_val_score
from sklearn.preprocessing import StandardScaler

# High Bias (Underfitting): Too simple
simple_model = LogisticRegression(C=0.01)  # Strong regularization
simple_score = cross_val_score(simple_model, X_train, y_train, cv=5)
print(f"High Bias: Train {simple_score.mean():.3f} (low)")

# High Variance (Overfitting): Too complex  
complex_model = RandomForestClassifier(n_estimators=1000, max_depth=None, min_samples_leaf=1)
complex_score_train = complex_model.score(X_train, y_train)  # Near 1.0
complex_score_val = cross_val_score(complex_model, X_train, y_train, cv=5)
print(f"High Variance: Train {complex_score_train:.3f}, Val {complex_score_val.mean():.3f} (gap)")

# Sweet Spot: Balanced complexity
balanced_model = RandomForestClassifier(
    n_estimators=200, 
    max_depth=10,           # Limit depth
    min_samples_leaf=5,     # Regularize
    max_features='sqrt'     # Decorrelate trees
)
balanced_score = cross_val_score(balanced_model, X_train, y_train, cv=5)
print(f"Balanced: {balanced_score.mean():.3f} (optimal)")

```



## Trade-offs


### Advantages

- Provides systematic framework for model selection

- Explains why more complex is not always better

- Guides regularization and feature engineering decisions

- Helps diagnose training issues via learning curves




### Considerations & Drawbacks

- Assumes bias and variance can be cleanly separated — not always true in practice

- Modern deep learning often challenges traditional tradeoff (double descent)

- Does not account for computational cost tradeoffs

- Hard to measure bias and variance directly in production







---
**Reference**: [Original Source](https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff)