Regularization and generalization
Introduction
Prefer solutions that transfer beyond training examples and diagnose overfitting with held-out curves.
- Learning goal
Explain L2 penalties, weight decay, dropout, and early stopping without confusing their roles.
- Before you start
Training and validation loss, model parameters, and gradient descent.
Lesson plan
- Identify overfitting evidence
- Add one controlled regularizer
- Select strength with validation data
The problem
A model can drive training loss down when it fits details that do not repeat in new data. The result is overfitting: strong training performance and weaker validation performance.
Regularization changes the learning problem to prefer solutions expected to transfer better. It can increase training loss and lower validation loss. It cannot repair leakage, wrong labels, or missing predictive information.
Work through a small example
Suppose two models fit the training data in a similar way. One uses weights [4,3]. The other uses [1,1]. Their squared L2 penalties are 25 and 2. A term lambda*sum(w^2) prefers the second model when the data losses are close.
The preference is useful only if a smaller norm matches a real inductive bias. An inductive bias is an assumption. The assumption helps choose among functions that agree on observed data.
The general rule
L2 regularization optimizes data_loss + lambda*sum(w^2). Its extra gradient is proportional to w. It pulls large weights toward zero. For plain SGD, this rule matches weight decay. For adaptive optimizers, decoupled AdamW weight decay is not the same as an added L2 loss term.
Dropout removes random activations during training and rescales the survivors. Early stopping selects the checkpoint with the best validation evidence. Data augmentation encodes invariances. It creates examples that keep their labels.
Implement and inspect
import torch
w = torch.tensor([4.0, 3.0], requires_grad=True)
data_loss = torch.tensor(0.5)
strength = 0.1
objective = data_loss + strength * (w ** 2).sum()
objective.backward()
print(round(objective.item(), 3))
print([round(value, 3) for value in w.grad.tolist()])
The expected objective is 3.0: data loss 0.5 plus penalty 2.5. The penalty gradient is [0.8,0.6]. Weight and gradient tensors both have shape (2,).
Engineering checks
- Confirm overfitting from separate training and validation curves before you add regularization.
- Tune regularization strength on validation data, never test data.
- Call
model.eval()for validation so dropout is disabled. - Compare the result with more data or a model with less capacity.
Too much regularization causes underfitting: both training and validation performance stay poor. More regularization is not always safer.
Go deeper
Regularization can be explicit, such as a norm penalty, or implicit, such as the bias from gradient descent, batch noise, architecture, and stopping time. Parameter count alone does not determine modern neural-network generalization.
Validation selection can itself overfit when many experiments reuse one validation set. Nested evaluation, a final untouched test, and a report of variation across seeds reduce the chance of a lucky configuration.
Practice and recap
Question: Training loss falls while validation loss rises after epoch 12. What is a controlled first response?
Worked answer
Keep the epoch-12 checkpoint that validation loss selects. Compare it with later checkpoints on the same fixed validation split. Do not consult test results. Then try one change, such as modest weight decay. Keep the seed, data split, and model fixed.
Regularization expresses a preference among fitting solutions. Diagnose the failure first. Tune with validation evidence. Keep the test set separate.