02.04 · UNIT 01 · PyTorch in One Hour: foundations · Lesson
Automatic differentiation
Autograd records tensor operations and computes their gradients backward with the chain rule.
PLAIN-LANGUAGE INTRODUCTION
The idea
Autograd records tensor operations and computes their gradients backward with the chain rule.
A worked example
For the worked graph,
grad(loss, w1) returns (tensor([-0.0898]),) and grad(loss, b) returns (tensor([-0.0817]),).The input
Tensors marked with requires_grad=True.
The result
A gradient stored in .grad for each tracked leaf tensor.
Why you use it
A model can have millions of parameters, so hand differentiation is not practical.
A limit to remember
Autograd computes gradients. It does not update parameters.
Check your understanding
requires_grad=Truetells PyTorch to track every operation that uses a tensor and build a graph.- Reverse-mode automatic differentiation applies the chain rule from the loss back to the inputs.
grad()returns gradients for the requested tensors as a tuple..backward()accumulates gradients in tracked leaf tensors connected to the loss.- Gradients accumulate. Call
optimizer.zero_grad()or set.grad = Nonebefore a normal step. - Common mistake: calling
backward()twice without a clear sums both gradients.