Data splits, leakage and data quality

Why it matters

Many impressive offline results fail in production because the evaluation did not resemble deployment. A split that lets the model see the same user, the same document or the future during training measures memorization, not generalization. Interviewers probe this with "offline AUC is excellent but production is poor: what happened?" and with design questions where choosing the right split is most of the answer.

Core ideas

Three roles. Training data fits parameters. Validation data, or cross-validation, chooses hyperparameters, features, thresholds and stopping points. The test set estimates final performance, once. Every decision made by looking at a set leaks information about it, so a test set that has guided many choices quietly becomes a second validation set with an optimistic score. Keep a held-out set you consult rarely, and refresh it once it has been used up.

Split by the unit of deployment. Ask what the model will face: new rows, new users, new documents or next month's data.

• Random, stratified splits fit only when examples are independent and the future resembles the past; stratify on the label when classes are rare.
• Group splits keep every example from one entity (patient, user, session, document, company) on one side. Otherwise the model can score well by recognizing the entity: a user's habits, a document's boilerplate, one company's writing style.
• Time-based splits train on the past and evaluate on a later window, with several forward-chaining folds if needed. Leave a gap when labels mature slowly, such as churn defined over the following 30 days.

Leakage types.

• Preprocessing leakage: fitting a scaler, imputer, vocabulary, PCA, feature selector or target encoder on the full dataset before splitting, so its statistics include test information. Fit every transformation inside a pipeline, on training folds only. Feature selection and target encoding leak far more than scaling.
• Target leakage: features that are consequences or proxies of the label, such as "refund issued" when predicting fraud, or a status field updated after the outcome. For each feature ask whether it exists at prediction time, with the value it would have then.
• Temporal leakage: aggregates over windows that extend past the prediction time, or joins against current tables instead of point-in-time snapshots.
• Duplicate and near-duplicate leakage: the same image, reposted text, augmented copies or templated documents in both splits. For LLMs, benchmark items present in pretraining data (contamination) are the same problem at web scale.

Cross-validation. K-fold CV reduces the variance of the estimate on small data, but every fold must respect grouping and time order. If you tune hyperparameters with CV and report that same CV score, it is optimistic; nested CV (an inner loop for selection, an outer loop for estimation) measures the performance of the whole selection procedure.

Data quality is part of evaluation. Before modeling, check label agreement on a sample, missingness patterns (missingness can itself encode the label), duplicates, class balance per split, value ranges and schema. A model trained on silently broken data can still produce a smooth loss curve.

Common misconceptions

• "Scaling before splitting is harmless because it is unsupervised." It still uses test statistics; the effect is usually small for scaling but large for feature selection or target encoding. Fit every transformation on training data only.
• "Shuffled k-fold is always the safe default." For time series or grouped data it overstates performance; use group-aware or forward-chaining splits.
• "Stratification prevents leakage." It only balances label proportions; it does nothing about shared entities or time.
• "A very predictive feature is a very useful feature." Suspiciously strong features are often label proxies unavailable at prediction time; check their timing and provenance.
• "The test set is safe as long as I never train on it." Repeatedly choosing models by test score is fitting to it.
• "Deduplication means removing exact matches." Near-duplicates (re-encoded images, lightly edited text, templated pages) leak just as well; find them with hashing or embedding similarity.

How to approach it

Begin every design answer with "what will the model see at prediction time, and about which entities?" and derive the split from that. Build a deliberately leaky pipeline, measure the inflation, then fix it. When reviewing someone's work, run through a leakage taxonomy as a checklist. Senior-level discussions add how labels are generated, how features are joined point-in-time, and how to detect leakage after launch (a large offline-online gap, or one feature dominating importance).