Train, validation, and test splits

Introduction

Separate parameter fitting, model selection, and final evaluation so each reported result answers a clear question.

Learning goal: Design a split that matches the real prediction unit and prevents preprocessing leakage.
Before you start: Tensor indexing, supervised targets, and the purpose of a held-out evaluation.

Lesson plan

Assign each split one job
Split by the deployment unit
Fit preprocessing on training data only

The problem

A model can memorize its training examples. Measuring it on those same examples therefore estimates fit, not performance on future data. We need held-out data: examples excluded from parameter updates.

Three splits have different jobs. Training data changes weights. Validation data chooses settings such as model width, learning rate, and stopping epoch. Test data is used after those choices are finished. Repeatedly changing a model after viewing test results makes the test set part of selection.

Work through a small example

Suppose twelve messages are ordered by time. IDs 0–7 train the model, 8–9 select the recipe, and 10–11 test the finished recipe. The proportions are less important than preserving the prediction question.

If the model will predict later messages, this chronological split is more honest than random rows. If several rows belong to one person, split by person. Different row numbers do not guarantee independent evidence.

The general rule

First define the deployment unit and time. Then assign whole units to splits. Only after that should you fit data-dependent preprocessing. A vocabulary, standardization mean, missing-value replacement, or feature selector must learn from training data only.

Formally, a test score estimates expected performance under the sampled test distribution. It is not a guarantee under distribution shift. It also has sampling uncertainty: a score from 20 test examples is much less precise than one from 20,000 comparable examples.

Implement and inspect

This next snippet demonstrates random splitting for independent examples, not the time-ordered split above.

import torch

torch.manual_seed(7)
x = torch.arange(12)
order = torch.randperm(len(x))
train, validation, test = order[:8], order[8:10], order[10:]

print(train.tolist())
print(validation.tolist())
print(test.tolist())
print(len(set(train.tolist()) & set(test.tolist())))

Expected output is train indices [3,4,0,5,7,9,11,6], validation indices [8,1], test indices [10,2], then overlap count 0.

Each index tensor has shape (split_size,) and dtype torch.int64. The final zero confirms no exact index appears in both train and test. For grouped data, compare group IDs too; index disjointness alone is insufficient.

Engineering checks

Save the split indices and random seed so a rerun uses the same evidence.
Search for exact and near duplicates across splits.
Assert every preprocessing object's fit method sees training rows only.
Check class and subgroup counts; a split can be disjoint but unrepresentative.

Go deeper

Random splitting assumes examples are exchangeable: their order and identity do not change the target question. Time series, repeated users, medical visits, and document chunks usually violate that assumption. Grouped or temporal splitting better matches the intended generalization.

Cross-validation rotates several train/validation partitions and reduces dependence on one validation draw. It still does not replace a final untouched test set when many model choices are made.

Practice and recap

Question: One patient has five visits. Why is putting four visits in training and one in test risky?

Worked answer

The model can learn patient-specific signals from the four training visits. The test visit is a new row, but not a new patient. If deployment targets unseen patients, assign all five visits to one split. Then fit preprocessing on training patients only.

Split around the real prediction unit, fit transforms on training data, select with validation data, and reserve test data for the final estimate.