Gradient descent and learning rates

The problem

Training means finding parameter values with low loss. The gradient gives the local direction of fastest loss increase. Gradient descent moves in the opposite direction.

The learning rate controls step size. A rate that is too small wastes updates. A rate that is too large can jump across a minimum, oscillate, or make the loss non-finite.

Work through a small example

Use loss L(w)=(w-3)^2. Its minimum is at w=3, and gradient is dL/dw=2(w-3). Starting at w=0, the gradient is -6.

With learning rate 0.1, one update gives w=0-0.1*(-6)=0.6. Loss falls from 9 to 5.76.

The general rule

For parameter vector theta, plain gradient descent uses theta_next=theta-learning_rate*gradient. With mini-batches, the gradient is stochastic because it estimates the full-data gradient from sampled examples.

For the one-dimensional quadratic a(w-w*)^2 with a>0, the error follows error_next=(1-2a*learning_rate)*error. Convergence requires 0<learning_rate<1/a. Neural networks have many directions with different curvature, so one rate can be cautious in flat directions and unstable in sharp directions.

Implement and inspect

import torch

w = torch.tensor(0.0, requires_grad=True)
learning_rate = 0.1

for _ in range(4):
    loss = (w - 3.0) ** 2
    loss.backward()
    print(round(w.detach().item(), 3), round(loss.detach().item(), 3))
    with torch.no_grad():
        w -= learning_rate * w.grad
    w.grad = None

Expected lines are 0.0 9.0, 0.6 5.76, 1.08 3.686, and 1.464 2.359. The parameter and loss are scalar tensors with shape ().

Engineering checks

• Clear gradients before the next backward pass; PyTorch accumulates them.
• Log loss before and after updates with unambiguous step numbers.
• Assert parameters, activations, loss, and gradients are finite.
• Overfit a tiny batch before tuning a full experiment.

If loss rises immediately, first lower the learning rate and verify the update sign. If it stays exactly flat, inspect whether parameters are registered and gradients are nonzero.

Go deeper

Momentum averages recent directions and can reduce zig-zagging in narrow valleys. Adam divides a moving gradient estimate by a moving scale estimate, giving parameters different effective step sizes. These methods change optimization dynamics, not the model's representational capacity.

A decreasing training loss does not prove generalization. Optimizer choice, learning-rate schedules, batch noise, and stopping time are model-selection choices that need validation evidence and controlled comparisons.

Practice and recap

Question: At w=4, what update does learning rate 0.2 make for this loss?

Gradient descent uses local slope information. Learning rate, curvature, batch noise, and gradient hygiene determine whether those steps are useful.