# Workflow Pipeline

> Orchestrates ML process as discrete, executable components with clear dependencies

- **Category**: Machine Learning
- **Subcategory**: Serving and Operational Resilience
- **Canonical URL**: https://designpattern.fyi/patterns/workflow-pipeline/

---

## Description
**Intent**: Make the full ML process reproducible, schedulable, and partially re-runnable by breaking it into discrete, independently executable components with clearly defined inputs and outputs.

**Context**: A full ML process involves many distinct steps often written by different people in different tools, frequently run manually and ad hoc. This makes the process hard to reproduce, schedule, partially re-run, and collaborate on safely.

**Solution**: Break ML process into discrete, independently executable components with clearly defined inputs and outputs, connected into explicit pipeline definition managed by orchestration system. This makes entire process reproducible, lets individual steps be cached or retried, and allows automatic triggering on schedule or events.



## Use Cases
- Production ML systems beyond exploratory analysis
- Systems needing periodic retraining
- Multi-contributor ML projects
- Any ML process requiring automation and reproducibility






## Trade-offs


### Advantages

- Reproducible end-to-end ML processes

- Individual steps can be cached or independently retried

- Automated execution on schedules or events

- Enables safe collaboration across teams




### Considerations & Drawbacks

- Upfront engineering investment in orchestration infrastructure

- Overkill for one-off exploratory analysis

- Requires tooling and operational expertise

- Adds complexity to ML workflow







---
**Reference**: [Original Source](https://github.com/GoogleCloudPlatform/ml-design-patterns)