Gradient descent and learning rates

Introduction

Use local gradient information to lower loss while controlling step size and instability.

Learning goal

Calculate gradient-descent updates and diagnose learning-rate failures.

Before you start

Derivatives, scalar loss, autograd, and parameter updates.

Lesson plan

  1. Calculate one gradient step
  2. Compare loss across updates
  3. Check gradient and learning-rate failures

The problem

Training seeks parameter values with low loss. The gradient gives the local direction of fastest loss increase. Gradient descent moves in the opposite direction.

The learning rate controls step size. A rate that is too small wastes updates. A rate that is too large can jump across a minimum, oscillate, or make the loss non-finite.

Work through a small example

Use loss L(w)=(w-3)^2. Its minimum is at w=3. Its gradient is dL/dw=2(w-3). Start at w=0: the gradient is -6.

With learning rate 0.1, one update gives w=0-0.1*(-6)=0.6. The loss falls from 9 to 5.76.

Three numeric gradient-descent states that approach a minimumA step strip shows scalar parameter and loss pairs: zero and nine, zero point six and five point seven six, then one point zero eight and three point six eight six four. A separate green circle labels the minimum at parameter three. w=0L=9w=.6L=5.76w=1.08L=3.6864w=3minimumstep 0step 1step 2target

The general rule

For parameter vector theta, plain gradient descent uses theta_next=theta-learning_rate*gradient. With mini-batches, the gradient is stochastic because it estimates the full-data gradient from sampled examples.

For the one-dimensional quadratic a(w-w*)^2 with a>0, the error follows error_next=(1-2a*learning_rate)*error. Convergence requires 0<learning_rate<1/a. Neural networks have many directions with different curvature, so one rate can be cautious in flat directions and unstable in sharp directions.

Implement and inspect

import torch

w = torch.tensor(0.0, requires_grad=True)
learning_rate = 0.1

for _ in range(4):
    loss = (w - 3.0) ** 2
    loss.backward()
    print(round(w.detach().item(), 3), round(loss.detach().item(), 3))
    with torch.no_grad():
        w -= learning_rate * w.grad
    w.grad = None

The expected lines are 0.0 9.0, 0.6 5.76, 1.08 3.686, and 1.464 2.359. The parameter and the loss are scalar tensors with shape ().

Engineering checks

If the loss rises immediately, lower the learning rate first. Then verify the update sign. If the loss stays exactly flat, check whether the parameters are registered and the gradients are nonzero.

Go deeper

Momentum averages recent directions. It can reduce zig-zag motion in narrow valleys. Adam divides a moving gradient estimate by a moving scale estimate. Each parameter then gets a different effective step size. These methods change the optimization dynamics, not the representational capacity of the model.

A decreasing training loss does not prove generalization. Optimizer choice, learning-rate schedules, batch noise, and stopping time are model-selection choices that need validation evidence and controlled comparisons.

Practice and recap

Question: At w=4, what update does learning rate 0.2 make for this loss?

Worked answer

The gradient is 2*(4-3)=2. The update is 4-0.2*2=3.6. The loss falls from 1 to 0.36, so the step moves toward the minimum.

Gradient descent uses local slope information. Learning rate, curvature, batch noise, and gradient hygiene determine whether those steps are useful.