02.03 · UNIT 01 · Regression and tensor mechanics · Lesson

Derivatives, autograd, and gradient checking

Autograd records tensor calculations. It can then calculate their local gradients backward.

PLAIN-LANGUAGE INTRODUCTION

What is this?

Autograd records tensor calculations. It can then calculate their local gradients backward.

One simple example

For L=w² at w=3, autograd gives gradient 6. A 0.1 descent step gives w=2.4.

A loss made from tensors that track gradients.

A gradient stored for each tracked parameter.

Large models have too many connected derivatives to calculate by hand.

A gradient is not an update. The optimizer chooses how to use it.

WORK THROUGH THE IDEA

Set w=3 and calculate L=w²=9. Autograd records the square operation.
Calling backward() applies the derivative dL/dw=2w. It stores 6 in w.grad.
With learning rate 0.1, descent computes 3−0.1×6 = 2.4.
The new loss is 2.4² = 5.76. This one step lowered the loss from 9.
Common mistake: gradients accumulate. Clear old gradients before an ordinary new batch.