Repeatable Sampling
Deterministic, reproducible train/validation/test splits that prevent data leakage
Intent & Description
🎯 Intent
Ensure reproducible splits across runs and prevent information leakage when data has natural groupings that shouldn’t be split across train and test sets.
📋 Context
A fresh random shuffle for every split isn’t reproducible across runs. Naive row-level splits can leak information when data has natural groupings (multiple rows for same customer), putting correlated records in both train and test and inflating apparent performance.
💡 Solution
Use a deterministic mechanism—typically hashing a stable identifier like customer ID into a fixed numeric range—to assign every record connected to the same real-world entity to the same split, consistently every time. This makes splits reproducible and prevents leakage by keeping related records together.
Real-world Use Case
- Customer data with multiple transactions per customer
- User activity data with multiple sessions per user
- Medical data with multiple visits per patient
- Any data with natural groupings that shouldn’t be split
Source
📌 TL;DR
Use deterministic hashing of stable identifiers to create reproducible train/validation/test splits that keep related records together and prevent information leakage
Advantages
- Exactly reproducible splits across runs and environments
- Prevents information leakage from grouped data
- Deterministic and reliable
- Enables proper evaluation without inflated metrics
Disadvantages
- Depends on having stable, well-distributed identifier
- Picking wrong grouping key defeats the purpose
- May need adjustment if hash doesn’t distribute evenly
- Requires understanding of data’s natural groupings