Tensor by Tensor

02.04 · UNIT 01 · PyTorch in One Hour: foundations · Lesson

Automatic differentiation

Autograd records tensor operations and computes their gradients backward with the chain rule.

PLAIN-LANGUAGE INTRODUCTION

The idea

Autograd records tensor operations and computes their gradients backward with the chain rule.

A worked example

For the worked graph, grad(loss, w1) returns (tensor([-0.0898]),) and grad(loss, b) returns (tensor([-0.0817]),).

The input

Tensors marked with requires_grad=True.

The result

A gradient stored in .grad for each tracked leaf tensor.

Why you use it

A model can have millions of parameters, so hand differentiation is not practical.

A limit to remember

Autograd computes gradients. It does not update parameters.

Check your understanding

  1. requires_grad=True tells PyTorch to track every operation that uses a tensor and build a graph.
  2. Reverse-mode automatic differentiation applies the chain rule from the loss back to the inputs.
  3. grad() returns gradients for the requested tensors as a tuple. .backward() accumulates gradients in tracked leaf tensors connected to the loss.
  4. Gradients accumulate. Call optimizer.zero_grad() or set .grad = None before a normal step.
  5. Common mistake: calling backward() twice without a clear sums both gradients.
Open the detailed notes ↗