Gradient descent and learning rates
Introduction
Use local gradient information to lower loss while controlling step size and instability.
- Learning goal
Calculate gradient-descent updates and diagnose learning-rate failures.
- Before you start
Derivatives, scalar loss, autograd, and parameter updates.
Lesson plan
- Calculate one gradient step
- Compare loss across updates
- Check gradient and learning-rate failures
The problem
Training seeks parameter values with low loss. The gradient gives the local direction of fastest loss increase. Gradient descent moves in the opposite direction.
The learning rate controls step size. A rate that is too small wastes updates. A rate that is too large can jump across a minimum, oscillate, or make the loss non-finite.
Work through a small example
Use loss L(w)=(w-3)^2. Its minimum is at w=3. Its gradient is dL/dw=2(w-3). Start at w=0: the gradient is -6.
With learning rate 0.1, one update gives w=0-0.1*(-6)=0.6. The loss falls from 9 to 5.76.
The general rule
For parameter vector theta, plain gradient descent uses theta_next=theta-learning_rate*gradient. With mini-batches, the gradient is stochastic because it estimates the full-data gradient from sampled examples.
For the one-dimensional quadratic a(w-w*)^2 with a>0, the error follows error_next=(1-2a*learning_rate)*error. Convergence requires 0<learning_rate<1/a. Neural networks have many directions with different curvature, so one rate can be cautious in flat directions and unstable in sharp directions.
Implement and inspect
import torch
w = torch.tensor(0.0, requires_grad=True)
learning_rate = 0.1
for _ in range(4):
loss = (w - 3.0) ** 2
loss.backward()
print(round(w.detach().item(), 3), round(loss.detach().item(), 3))
with torch.no_grad():
w -= learning_rate * w.grad
w.grad = None
The expected lines are 0.0 9.0, 0.6 5.76, 1.08 3.686, and 1.464 2.359. The parameter and the loss are scalar tensors with shape ().
Engineering checks
- Clear gradients before the next backward pass. PyTorch accumulates them.
- Log loss before and after updates with unambiguous step numbers.
- Assert parameters, activations, loss, and gradients are finite.
- Overfit a tiny batch before you tune a full experiment.
If the loss rises immediately, lower the learning rate first. Then verify the update sign. If the loss stays exactly flat, check whether the parameters are registered and the gradients are nonzero.
Go deeper
Momentum averages recent directions. It can reduce zig-zag motion in narrow valleys. Adam divides a moving gradient estimate by a moving scale estimate. Each parameter then gets a different effective step size. These methods change the optimization dynamics, not the representational capacity of the model.
A decreasing training loss does not prove generalization. Optimizer choice, learning-rate schedules, batch noise, and stopping time are model-selection choices that need validation evidence and controlled comparisons.
Practice and recap
Question: At w=4, what update does learning rate 0.2 make for this loss?
Worked answer
The gradient is 2*(4-3)=2. The update is 4-0.2*2=3.6. The loss falls from 1 to 0.36, so the step moves toward the minimum.
Gradient descent uses local slope information. Learning rate, curvature, batch noise, and gradient hygiene determine whether those steps are useful.