02.05 · UNIT 02 · Supervised learning and reliable experiments · Lesson

Datasets, batching, splits, and trustworthy metrics

Separate data gives parameter learning, model choices, and final testing different jobs.

PLAIN-LANGUAGE INTRODUCTION

What is this?

Separate data gives parameter learning, model choices, and final testing different jobs.

One simple example

Split 100 rows into 70 training, 15 validation, and 15 test rows.

Examples, targets, and a split rule chosen before training.

Separate batches and honest measurements on untouched rows.

A final test must not guide training or model choices.

A low training error does not show performance on new data.

WORK THROUGH THE IDEA

Training rows change model parameters. Validation rows help choose settings. Test rows give the final check.
For 100 rows, this example assigns 70, 15, and 15 rows.
Suppose training values have mean 10 and standard deviation 2. Transform validation value 14 to (14−10)/2 = 2.
Fit preprocessing only on training rows. Then reuse those fitted numbers everywhere else.
Common mistake: fitting a scaler on all rows leaks information from validation and test data.