Lesson 2

Derivatives, autograd, and gradient checking

Why gradients matter

A model learns by changing its parameters, the numbers it controls. A loss function measures how wrong the current prediction is. A gradient answers a local question: if one parameter increases a tiny amount, how will the loss change? A positive gradient says increasing the parameter increases the loss, so gradient descent moves the parameter downward. A negative gradient points the other way.

PyTorch autograd records tensor operations while the forward calculation runs. These records form a computation graph: values are connected by the operations that created them. Calling backward() follows that graph in reverse and applies the calculus chain rule. You do not need to write every derivative yourself.

Important terms

Worked example by hand and in PyTorch

Let w = 2, prediction y = 3w, target 9, and squared loss L = (y - 9)². First, y = 6 and L = 9. The derivative of the square is 2(y - 9), which is -6. The derivative of y with respect to w is 3. The chain rule multiplies them: dL/dw = -6 × 3 = -18.

Predict before backward(): the prediction 6 is below the target 9. Should training raise or lower w? A negative gradient is consistent with raising w, because gradient descent subtracts that negative number.

import torch

w = torch.tensor(2.0, requires_grad=True)
target = torch.tensor(9.0)
y = w * 3
loss = (y - target) ** 2

loss.backward()
print("prediction:", y.item())
print("loss:", loss.item())
print("gradient:", w.grad.item())

# Expected:
# prediction: 6.0
# loss: 9.0
# gradient: -18.0

The result in w.grad is information, not an update. Autograd does not change w. An optimizer uses the gradient later. For one manual gradient-descent step with learning rate 0.01, the new value would be 2 - 0.01 × (-18) = 2.18. We make updates inside torch.no_grad() so the update itself is not added to the next computation graph.

with torch.no_grad():
    w -= 0.01 * w.grad
print(round(w.item(), 2))  # Expected: 2.18

Gradients accumulate

PyTorch adds each new gradient to the existing .grad value. This is useful when combining several small batches, but it is a common source of bugs. If you call backward() twice without clearing the gradient, w.grad contains the sum. Training loops therefore call optimizer.zero_grad() before the next backward pass, or set a manual gradient to None.

For example, two backward passes that each produce gradient 4 leave .grad = 8. A common wrong answer is 4 because the learner expects replacement. Check accumulation by printing .grad after each backward pass.

w.grad = None
y = w * 3
loss = (y - target) ** 2
loss.backward()
print(round(w.grad.item(), 4))  # Expected: about -14.76

The gradient changed because the updated w gives a different prediction. PyTorch creates a fresh computation graph for the new forward pass. By default, a graph used by backward() is then freed, so do a new forward pass before the next ordinary backward pass.

Try it

For x = 4 and f = x³ + 2x, predict the gradient by hand. Then ask autograd to check it.

import torch
x = torch.tensor(4.0, requires_grad=True)
f = x ** 3 + 2 * x
f.backward()
print(f.item())       # Expected: 72.0
print(x.grad.item())  # Expected: 50.0

Recap

Autograd records a forward computation and uses the chain rule in reverse. requires_grad=True marks leaf values whose gradients we want. backward() calculates those gradients, and .grad stores them. The optimizer, not autograd, performs the parameter update. Remember to clear accumulated gradients between normal training steps.

Next, nn.Module will collect many tracked parameters under one model. The gradient rule stays the same; only the organization changes.

References: PyTorch autograd mechanics and automatic differentiation tutorial.