Batch Serving
Precomputes predictions asynchronously using distributed data processing
Intent & Description
🎯 Intent
Handle prediction needs that don’t require real-time freshness by precomputing predictions asynchronously and serving them via simple lookup, avoiding the overhead of online serving.
📋 Context
Not every prediction need is real-time. Scoring an entire customer database overnight doesn’t need low-latency per-request serving, and using online serving for bulk workloads wastes overhead without benefit.
💡 Solution
Use distributed data processing infrastructure to run trained model inference across large volume of inputs asynchronously, writing predictions to a fast-lookup store. Requests are then served by simple lookup against precomputed results rather than live computation.
Real-world Use Case
- Large-scale recommendation systems with periodic updates
- Customer database scoring overnight
- Reporting and analytics workloads
- Any latency-tolerant prediction needs
Source
📌 TL;DR
Precompute predictions asynchronously using distributed processing and serve via lookup for latency-tolerant, large-scale workloads
Advantages
- Efficient for large-scale, latency-tolerant workloads
- Leverages existing distributed data processing infrastructure
- Simple serving via fast lookup
- Cost-effective for bulk prediction needs
Disadvantages
- Predictions can go stale between batch runs
- Wrong choice when real-time freshness is required
- Requires scheduling and orchestration infrastructure
- Needs fallback for new entities not in batch