Regularization and generalization

Introduction

Prefer solutions that transfer beyond training examples and diagnose overfitting with held-out curves.

Learning goal

Explain L2 penalties, weight decay, dropout, and early stopping without confusing their roles.

Before you start

Training and validation loss, model parameters, and gradient descent.

Lesson plan

  1. Identify overfitting evidence
  2. Add one controlled regularizer
  3. Select strength with validation data

The problem

A model can drive training loss down when it fits details that do not repeat in new data. The result is overfitting: strong training performance and weaker validation performance.

Regularization changes the learning problem to prefer solutions expected to transfer better. It can increase training loss and lower validation loss. It cannot repair leakage, wrong labels, or missing predictive information.

Work through a small example

Suppose two models fit the training data in a similar way. One uses weights [4,3]. The other uses [1,1]. Their squared L2 penalties are 25 and 2. A term lambda*sum(w^2) prefers the second model when the data losses are close.

Schematic training and validation loss curves during overfittingA schematic blue training-loss curve falls steadily. A schematic green validation-loss curve falls then rises. A scalar circle marks the lowest validation loss as the selected checkpoint. The curves are illustrative, not measured data. trainingvalidationbest checkpointlossepochsschematic

The preference is useful only if a smaller norm matches a real inductive bias. An inductive bias is an assumption. The assumption helps choose among functions that agree on observed data.

The general rule

L2 regularization optimizes data_loss + lambda*sum(w^2). Its extra gradient is proportional to w. It pulls large weights toward zero. For plain SGD, this rule matches weight decay. For adaptive optimizers, decoupled AdamW weight decay is not the same as an added L2 loss term.

Dropout removes random activations during training and rescales the survivors. Early stopping selects the checkpoint with the best validation evidence. Data augmentation encodes invariances. It creates examples that keep their labels.

Implement and inspect

import torch

w = torch.tensor([4.0, 3.0], requires_grad=True)
data_loss = torch.tensor(0.5)
strength = 0.1
objective = data_loss + strength * (w ** 2).sum()
objective.backward()

print(round(objective.item(), 3))
print([round(value, 3) for value in w.grad.tolist()])

The expected objective is 3.0: data loss 0.5 plus penalty 2.5. The penalty gradient is [0.8,0.6]. Weight and gradient tensors both have shape (2,).

Engineering checks

Too much regularization causes underfitting: both training and validation performance stay poor. More regularization is not always safer.

Go deeper

Regularization can be explicit, such as a norm penalty, or implicit, such as the bias from gradient descent, batch noise, architecture, and stopping time. Parameter count alone does not determine modern neural-network generalization.

Validation selection can itself overfit when many experiments reuse one validation set. Nested evaluation, a final untouched test, and a report of variation across seeds reduce the chance of a lucky configuration.

Practice and recap

Question: Training loss falls while validation loss rises after epoch 12. What is a controlled first response?

Worked answer

Keep the epoch-12 checkpoint that validation loss selects. Compare it with later checkpoints on the same fixed validation split. Do not consult test results. Then try one change, such as modest weight decay. Keep the seed, data split, and model fixed.

Regularization expresses a preference among fitting solutions. Diagnose the failure first. Tune with validation evidence. Keep the test set separate.