02.14 · UNIT 04 · Supervised learning and reliable experiments · Lesson
Optimization, initialization, and generalization
Optimization changes parameters. Initialization starts them. Validation checks generalization.
PLAIN-LANGUAGE INTRODUCTION
The idea
Optimization changes parameters. Initialization starts them. Validation checks generalization.
A worked example
Training losses are
[1.0,0.6,0.3]. Validation losses are [1.1,0.7,0.8]. Epoch 2 is the best checkpoint here.The input
Initialized parameters, training batches, and separate validation batches.
The result
Updated parameters plus training and validation measurements.
Why you use it
The validation curve can reveal overfitting before training loss does.
A limit to remember
The lowest training loss is not always the best saved model.
Check your understanding
- Initialization sets the first parameter values. Good scales help signals and gradients stay usable.
- The optimizer lowers training loss from
1.0to0.6, then0.3in this illustration. - Validation loss moves from
1.1to0.7, then rises to0.8. - Epoch
2has the lowest validation loss. Save it if validation loss is the chosen measure. - Common mistake: using test results to choose an epoch turns the test set into validation data.