Train, validation, and test splits
Introduction
Separate parameter fitting, model selection, and final evaluation so each reported result answers a clear question.
- Learning goal
Design a split that matches the real prediction unit and prevents preprocessing leakage.
- Before you start
Tensor indexing, supervised targets, and the purpose of a held-out evaluation.
Lesson plan
- Assign each split one job
- Split by the deployment unit
- Fit preprocessing on training data only
The problem
A model can memorize its training examples. A score on those same examples estimates fit, not performance on future data. Therefore, keep held-out data. Held-out data is a set of examples that parameter updates exclude.
Three splits have different jobs. Training data changes the weights. Validation data selects settings, such as model width, learning rate, and stop epoch. You use test data after those choices are final. If you change the model again after you view the test results, the test set becomes part of selection.
Work through a small example
Suppose twelve messages are ordered by time. IDs 0–7 train the model, 8–9 select the recipe, and 10–11 test the finished recipe. The proportions are less important than the prediction question.
If the model will predict later messages, this chronological split is more honest than random rows. If several rows belong to one person, split by person. Different row numbers do not guarantee independent evidence.
The general rule
First define the deployment unit and the time. Then assign whole units to the splits. Only after that should you fit data-dependent preprocessing. A vocabulary, a standardization mean, a missing-value replacement, or a feature selector must learn from training data only.
Formally, a test score estimates expected performance under the sampled test distribution. It is not a guarantee under distribution shift. It also has sampling uncertainty. A score from 20 test examples is much less precise than one from 20,000 comparable examples.
Implement and inspect
This next snippet shows a random split for independent examples. It does not show the time-ordered split above.
import torch
torch.manual_seed(7)
x = torch.arange(12)
order = torch.randperm(len(x))
train, validation, test = order[:8], order[8:10], order[10:]
print(train.tolist())
print(validation.tolist())
print(test.tolist())
print(len(set(train.tolist()) & set(test.tolist())))
The expected output is the train indices [3,4,0,5,7,9,11,6], the validation indices [8,1], the test indices [10,2], then the overlap count 0.
Each index tensor has shape (split_size,) and dtype torch.int64. The final zero confirms no exact index appears in both train and test. For grouped data, compare group IDs too. Index disjointness alone is not sufficient.
Engineering checks
- Save the split indices and random seed so a rerun uses the same evidence.
- Search for exact and near duplicates across splits.
- Assert every preprocessing object's
fitmethod sees training rows only. - Check class and subgroup counts. A split can be disjoint but unrepresentative.
Go deeper
A random split assumes examples are exchangeable. Their order and identity do not change the target question. Time series, repeated users, medical visits, and document chunks usually violate that assumption. A grouped split or a temporal split better matches the intended generalization.
Cross-validation rotates several train/validation partitions and reduces dependence on one validation draw. It still does not replace a final untouched test set when many model choices are made.
Practice and recap
Question: One patient has five visits. Why is a split with four training visits and one test visit risky?
Worked answer
The model can learn patient-specific signals from the four training visits. The test visit is a new row, but not a new patient. If deployment targets unseen patients, assign all five visits to one split. Then fit preprocessing on training patients only.
Split around the real prediction unit. Fit transforms on training data. Select with validation data. Reserve test data for the final estimate.