Distribution Strategy
Scales training across multiple processors or machines through parallelism
Intent & Description
🎯 Intent
Handle modern datasets and models that are too large to train in reasonable time, or too large to even fit in memory, on a single machine or accelerator.
📋 Context
Large-scale ML workloads can be constrained by single-device memory limits or training time. Training on a single device may be prohibitive in cost or simply impossible for models that don’t fit.
💡 Solution
Scale training across multiple processors or machines through data parallelism (replicate model on every worker, split each batch, synchronize gradients) or model parallelism (split model across devices when it doesn’t fit on one). Gradient synchronization can be synchronous (deterministic) or asynchronous (higher throughput, noisier).
Real-world Use Case
- Training large models that don’t fit on single device memory
- Reducing training time for large datasets
- Distributed training across multiple GPUs/TPUs
- Any workload where single-device training is prohibitive
Source
📌 TL;DR
Scale training across multiple devices using data parallelism or model parallelism to handle large models or reduce training time
Advantages
- Enables training of models too large for single device
- Reduces training time through parallelism
- Scales to use available cluster resources
- Flexible strategies for different bottlenecks
Disadvantages
- Adds infrastructure and orchestration complexity
- Communication overhead can eat into expected speedup
- Requires careful hyperparameter tuning at scale
- Debugging distributed systems is more complex