Tensor by Tensor

02.14 · UNIT 04 · Supervised learning and reliable experiments · Lesson

Optimization, initialization, and generalization

Optimization changes parameters. Initialization starts them. Validation checks generalization.

PLAIN-LANGUAGE INTRODUCTION

The idea

Optimization changes parameters. Initialization starts them. Validation checks generalization.

A worked example

Training losses are [1.0,0.6,0.3]. Validation losses are [1.1,0.7,0.8]. Epoch 2 is the best checkpoint here.

The input

Initialized parameters, training batches, and separate validation batches.

The result

Updated parameters plus training and validation measurements.

Why you use it

The validation curve can reveal overfitting before training loss does.

A limit to remember

The lowest training loss is not always the best saved model.

Check your understanding

  1. Initialization sets the first parameter values. Good scales help signals and gradients stay usable.
  2. The optimizer lowers training loss from 1.0 to 0.6, then 0.3 in this illustration.
  3. Validation loss moves from 1.1 to 0.7, then rises to 0.8.
  4. Epoch 2 has the lowest validation loss. Save it if validation loss is the chosen measure.
  5. Common mistake: using test results to choose an epoch turns the test set into validation data.
Open the detailed notes ↗