# Checkpoints

> Periodic saving of model state during training for fault tolerance and early stopping

- **Category**: Machine Learning
- **Subcategory**: Training Process and Optimization
- **Canonical URL**: https://designpattern.fyi/machine_learning/checkpoints/

---

## Description
**Intent**: Handle training interruption and identify the best-performing point in training, which isn't necessarily the final step, by periodically saving complete model state.

**Context**: Training a nontrivial model can take a long time and is vulnerable to interruption—crashes, preempted instances, hardware failure. The best-performing point in training might be several epochs earlier, before overfitting set in.

**Solution**: Periodically save the complete state of a model during training—weights at minimum, ideally optimizer state and training metadata too. This buys fault tolerance (resume after crash), natural implementation of early stopping (pick best checkpoint by validation performance), and flexibility to pause, resume, or fine-tune from any earlier point.



## Use Cases
- Long-running training jobs on expensive hardware
- Training where early stopping is needed
- Distributed training prone to failures
- Any non-trivial training run where time/cost matters






## Trade-offs


### Advantages

- Fault tolerance and recovery from interruptions

- Natural early stopping implementation

- Flexibility to resume or fine-tune from any point

- Ability to select best model by validation performance




### Considerations & Drawbacks

- Storage overhead for multiple checkpoints

- Requires discipline about which checkpoint is the model

- Adds complexity to training infrastructure

- Needs careful management of checkpoint retention







---
**Reference**: [Original Source](https://github.com/GoogleCloudPlatform/ml-design-patterns)