Tensor by Tensor

02.07 · UNIT 02 · PyTorch in One Hour: models and training · Lesson

The training loop

A training loop repeats five steps: forward, loss, zero_grad, backward, and step.

PLAIN-LANGUAGE INTRODUCTION

The idea

A training loop repeats five steps: forward, loss, zero_grad, backward, and step.

A worked example

On the toy data, the loss falls from 0.75 to 0.00 in three epochs.

The input

A model, a data loader, a loss function, and an optimizer.

The result

Updated parameters and a falling loss.

Why you use it

The loop connects predictions, error, gradients, and parameter updates.

A limit to remember

A falling training loss does not prove good generalization.

Check your understanding

  1. A training step changes parameters once. Compute predictions and loss. Clear old gradients before the backward pass. Update parameters after you compute new gradients.
  2. model.train() and model.eval() set the mode for layers such as dropout and batch normalization.
  3. Pass logits to cross_entropy. The function applies softmax internally.
  4. After training, argmax turns logits into labels. An accuracy function measures them.
  5. Common mistake: without zero_grad(), new gradients add to the old gradients.
Open the detailed notes ↗