Regularization and generalization

Introduction

Prefer solutions that transfer beyond training examples and diagnose overfitting with held-out curves.

Learning goal

Explain L2 penalties, weight decay, dropout, and early stopping without confusing their roles.

Before you start

Training and validation loss, model parameters, and gradient descent.

Lesson plan

  1. Identify overfitting evidence
  2. Add one controlled regularizer
  3. Select strength with validation data

The problem

A model can drive training loss down by fitting details that do not repeat in new data. The result is overfitting: strong training performance and weaker validation performance.

Regularization changes the learning problem to prefer solutions expected to transfer better. It can increase training loss while lowering validation loss. It cannot repair leakage, wrong labels, or missing predictive information.

Work through a small example

Suppose two models fit training data similarly. One uses weights [4,3]; the other uses [1,1]. Their squared L2 penalties are 25 and 2. Adding lambda*sum(w^2) prefers the second when data losses are close.

Schematic training and validation loss curves during overfittingA schematic blue training-loss curve keeps falling. A schematic green validation-loss curve falls then rises. A scalar circle marks the lowest validation loss as the selected checkpoint; the curves are illustrative, not measured data. trainingvalidationbest checkpointlossepochsschematic

The preference is useful only if smaller norm matches a real inductive bias. An inductive bias is an assumption that helps choose among functions agreeing on observed data.

The general rule

L2 regularization optimizes data_loss + lambda*sum(w^2). Its extra gradient is proportional to w, pulling large weights toward zero. For plain SGD this closely matches weight decay; for adaptive optimizers, decoupled AdamW weight decay is not identical to adding L2 to the loss.

Dropout randomly removes activations during training and rescales survivors. Early stopping selects the checkpoint with best validation evidence. Data augmentation encodes invariances by creating label-preserving examples.

Implement and inspect

import torch

w = torch.tensor([4.0, 3.0], requires_grad=True)
data_loss = torch.tensor(0.5)
strength = 0.1
objective = data_loss + strength * (w ** 2).sum()
objective.backward()

print(round(objective.item(), 3))
print([round(value, 3) for value in w.grad.tolist()])

Expected objective is 3.0: data loss 0.5 plus penalty 2.5. The penalty gradient is [0.8,0.6]. Weight and gradient tensors both have shape (2,).

Engineering checks

Too much regularization causes underfitting: both training and validation performance remain poor. More regularization is not automatically safer.

Go deeper

Regularization can be explicit, such as a norm penalty, or implicit, such as the bias introduced by gradient descent, batch noise, architecture, and stopping time. Parameter count alone does not determine modern neural-network generalization.

Validation selection can itself overfit when many experiments reuse one validation set. Nested evaluation, a final untouched test, and reporting variation across seeds reduce the chance of presenting a lucky configuration.

Practice and recap

Question: Training loss falls while validation loss rises after epoch 12. What is a controlled first response?

Worked answer

Keep the epoch-12 checkpoint selected by validation loss and compare it with later checkpoints on the same fixed validation split. Do not consult test results. Then try one change, such as modest weight decay, while holding seed, data split, and model fixed.

Regularization expresses a preference among fitting solutions. Diagnose the failure first, tune with validation evidence, and preserve the test set.