02.09 · UNIT 02 · Supervised learning and reliable experiments · Lesson

Optimization, initialization, and generalization

Optimization changes parameters. Initialization starts them. Validation checks generalization.

PLAIN-LANGUAGE INTRODUCTION

What is this?

Optimization changes parameters. Initialization starts them. Validation checks generalization.

One simple example

Training losses are [1.0,0.6,0.3]. Validation losses are [1.1,0.7,0.8]. Epoch 2 is the best checkpoint here.

Initialized parameters, training batches, and separate validation batches.

Updated parameters plus training and validation measurements.

The validation curve can reveal overfitting before training loss does.

The lowest training loss is not always the best saved model.

WORK THROUGH THE IDEA

Initialization sets the first parameter values. Good scales help signals and gradients stay usable.
The optimizer lowers training loss from 1.0 to 0.6, then 0.3 in this illustration.
Validation loss moves from 1.1 to 0.7, then rises to 0.8.
Epoch 2 has the lowest validation loss. Save it if validation loss is the chosen measure.
Common mistake: using test results to choose an epoch turns the test set into validation data.