# ML Pipeline

> Automates the workflow of data ingestion, preprocessing, model training, evaluation, and deployment.

- **Category**: Machine Learning
- **Subcategory**: MLOps
- **Canonical URL**: https://designpattern.fyi/patterns/ml-pipeline/

---

## Description
An ML Pipeline structures the end-to-end flow of data and model operations as a series of modular, sequential stages. This guarantees reproducibility, simplifies debugging, and enables continuous training/retraining (CT) loops.



## Use Cases
Implementing automated weekly retraining loops for e-commerce recommendation systems or fraud detection models.



## Implementation Example

```python
# A modular Python ML Pipeline implementation
class PipelineStep:
    def transform(self, data):
        raise NotImplementedError()

class DataIngestion(PipelineStep):
    def transform(self, data):
        print("Ingesting raw data...")
        return data + " -> [Ingested]"

class FeatureEngineering(PipelineStep):
    def transform(self, data):
        print("Extracting features...")
        return data + " -> [Features]"

class ModelTraining(PipelineStep):
    def transform(self, data):
        print("Training model...")
        return data + " -> [Model Trained]"

class MLPipeline:
    def __init__(self):
        self.steps = []

    def add_step(self, step):
        self.steps.append(step)
        return self

    def execute(self, raw_input):
        current_data = raw_input
        for step in self.steps:
            current_data = step.transform(current_data)
        return current_data

# Usage
pipeline = (MLPipeline()
            .add_step(DataIngestion())
            .add_step(FeatureEngineering())
            .add_step(ModelTraining()))
print(pipeline.execute("Raw Telemetry"))

```



## Trade-offs


### Advantages

- Ensures consistent data transformation between training and online inference.

- Highly modular and reusable steps.

- Simplifies tracking of data lineage, parameters, and versioning.




### Considerations & Drawbacks

- Can introduce significant engineering overhead for small, experimental models.

- Debugging intermediate steps in a running pipeline can be complex.