Keyed Predictions
Passes identifying keys through serving pipeline to match predictions with inputs
Intent & Description
🎯 Intent
Keep track of which prediction corresponds to which input when running inference on many inputs at once, especially in batch jobs or highly concurrent asynchronous systems.
📋 Context
When running inference on many inputs at once in batch jobs or highly concurrent systems, it’s easy to lose track of which prediction corresponds to which input, especially if nothing guarantees strict input/output ordering.
💡 Solution
Client passes an identifying key alongside each input, and serving function passes that key through unchanged, attaching it to corresponding output. Predictions can be reliably matched back to inputs regardless of processing order, batching, or parallelism.
Real-world Use Case
- Batch inference pipelines
- Highly concurrent asynchronous serving systems
- Distributed inference jobs
- Any scenario where input/output ordering isn’t guaranteed
Source
📌 TL;DR
Pass identifying keys through the serving pipeline to reliably match predictions with their inputs regardless of processing order, batching, or parallelism
Advantages
- Reliable key-to-output matching regardless of processing
- Works across batching, parallelism, and out-of-order processing
- Simple design habit with minimal cost
- Enables proper joining with downstream systems
Disadvantages
- Model interface must explicitly support pass-through field
- Small design overhead rather than genuine downside
- Requires key definition and management