Optimizers

Learn how changing several inputs changes one output. Begin with partial derivatives, then explore gradient descent, backpropagation, and optimizers. The later sections include proofs and deeper mathematical detail.

A first reading path

Start with sections 2–4, then section 7. Return to the directional-derivative proof after you understand a gradient-descent update. An optimizer is a rule for updating the model’s adjustable numbers.

For L(a, b) = a² + 2b², the gradient at (1, 2) is [2, 8]. The first entry measures change along a while b stays fixed. The second measures change along b while a stays fixed. With step size 0.1, subtract [0.2, 0.8] to reach (0.8, 1.2). The loss falls from 9 to 3.52.

Check: why subtract the gradient?

The gradient points toward the steepest local increase under the usual Euclidean length measure. Subtracting it moves toward a local decrease when the step is small enough. A large step can still overshoot and increase the loss.

7. Gradient Descent

Introduction

For a bowl-shaped loss starting at parameter 3, a learning rate of 0.1 reduces loss from 4 to about 1.05 after three updates. Larger steps can move faster, oscillate, or diverge.

Learning goal: Perform several gradient-descent updates and choose stable learning rates using curvature and visual trajectory evidence.
Before you start: Gradient vectors, derivatives of quadratic functions, repeated updates, coordinate points, and basic algebra.

Lesson plan

Calculate several one-dimensional updates and track parameter and loss values.
Explore two-dimensional parameter trajectories where unequal curvature causes visible zig-zagging behavior.
Derive a stable learning-rate bound and compare optimizer paths on Rosenbrock loss.

Calculate one step. If w=3, gradient is 4, and learning rate is 0.1, descent gives w=3−0.1×4=2.6. A positive gradient leads to a smaller parameter because descent moves opposite the increase direction.

The simplest optimiser: iterate

$$\boxed{\theta_{k+1} = \theta_k - \eta\,\nabla L(\theta_k)}$$

where $\eta>0$ is the learning rate. A first-order Taylor expansion gives $L(\theta - \eta\nabla L) = L(\theta) - \eta\|\nabla L\|^2 + o(\eta)$, so for small $\eta$ the loss drops by approximately $\eta\|\nabla L\|^2$.

7.1 One-dimensional demo

Experiment · $L(w)=(w-1)^2$, start at $w_0=3$

η 0.10 Steps 20

Adjust η and press Run.

Regimes: $\eta<0.5$ converges monotonically · $\eta=0.5$ converges in one step · $0.5<\eta<1.0$ oscillates with decay · $\eta\ge 1.0$ diverges. The threshold $\eta=1$ matches the theory $2/\lambda_{\max}=2/2=1$.

7.2 Two-dimensional ill-conditioned bowl

The loss $L(w,b)=(w-1)^2 + 4(b+2)^2$ has Hessian with eigenvalues $2$ and $8$, giving $\kappa=4$. The optimal step is $\eta < 2/\lambda_{\max}=0.25$. Larger $\eta$ causes the canonical zig-zag.

Experiment · trajectory on the bowl $L(w,b)=(w-1)^2+4(b+2)^2$

η 0.10 Steps 40

7.2.1 Learning-rate bound, derived

For $L(\theta)=\tfrac{1}{2}\theta^\top A\theta - b^\top\theta + c$ with $A\succ 0$, gradient descent becomes $\theta_{k+1}-\theta^\star = (I-\eta A)(\theta_k-\theta^\star)$, which converges iff $|1-\eta\lambda_i|<1$ for all eigenvalues $\lambda_i$. This forces

$$\boxed{0 < \eta < \frac{2}{\lambda_{\max}(A)}}.$$

Convergence rate is $(\kappa-1)/(\kappa+1)$ per step, so $\kappa\gg 1$ means slow convergence — the motivation for adaptive and second-order methods.

7.3 Optimiser race on Rosenbrock

The Rosenbrock function $f(x,y)=(1-x)^2+100(y-x^2)^2$ has a narrow curved valley with minimum at $(1,1)$. It is the classic test for optimisers. Run them head-to-head on the same landscape.

Experiment · compare SGD, SGD+momentum, and Adam

Steps 2000

SGD (η=0.0005) SGD + momentum (η=0.0005, μ=0.9) Adam (η=0.02)

8. Numerical Gradient Checking

Introduction

A backward pass can silently return a wrong gradient even when the loss decreases. At parameters 3 and negative 1, centered finite differences reproduce analytic gradients 4 and 8 to high precision.

Learning goal: Check an analytic gradient numerically with centered differences and interpret both absolute and relative disagreement.
Before you start: Derivatives, function evaluation, subtraction, small decimal values, and Python arrays.

Lesson plan

Perturb one parameter upward and downward while holding the others fixed.
Form a centered difference and compare it with the analytic derivative.
Use relative error and suitable step sizes to diagnose a broken implementation.

Gradient checking asks whether two independent calculations agree: backpropagation and a small finite difference. Use it as a correctness test on a tiny deterministic case. Do not use the finite difference as the training algorithm.

Always verify analytic gradients numerically before training. The centred-difference formula has $O(\varepsilon^2)$ truncation error:

$$\frac{\partial L}{\partial \theta_i} \approx \frac{L(\theta+\varepsilon e_i)-L(\theta-\varepsilon e_i)}{2\varepsilon}.$$

With $\varepsilon=10^{-5}$ and double precision, relative error should be $\sim 10^{-9}$. Anything above $10^{-5}$ is a bug.

import numpy as np
def L(w, b): return (w-1)**2 + 4*(b+2)**2
def analytic(w, b): return np.array([2*(w-1), 8*(b+2)])
def numerical(w, b, eps=1e-5):
    dw = (L(w+eps, b) - L(w-eps, b)) / (2*eps)
    db = (L(w, b+eps) - L(w, b-eps)) / (2*eps)
    return np.array([dw, db])
a, n = analytic(3, -1), numerical(3, -1)
print(np.abs(a - n) / (np.abs(a) + 1e-12))   # ~1e-11

9. The Hessian and Conditioning

Introduction

Two directions can have different curvature even at the same point. A quadratic loss with Hessian entries 2 and 8 bends four times more strongly in one direction, affecting stability and convergence.

Learning goal: Read a Hessian matrix, classify critical points, measure conditioning, and explain one Newton update.
Before you start: Gradients, second derivatives, matrices, eigenvalues as directional scaling, and quadratic functions.

Lesson plan

Differentiate the gradient to form the matrix of second partial derivatives.
Use eigenvalue signs to classify minima, maxima, and saddle points.
Connect unequal curvature to conditioning, learning-rate limits, and Newton's method.

Picture two curvatures. A bowl can be steep left-to-right and shallow front-to-back. One learning rate then moves too aggressively on one axis and too slowly on the other. The Hessian records this local curvature interaction.

The second-order structure of $L$ is captured by

$$H_{ij}(\theta) = \partial_i\partial_j L(\theta),\qquad L(\theta+\Delta) = L(\theta)+\langle\nabla L,\Delta\rangle+\tfrac{1}{2}\Delta^\top H\,\Delta + o(\|\Delta\|^2).$$

Classifying critical points ($\nabla L=0$)

Spectrum of $H$	Local behaviour
All $\lambda_i>0$	Strict local minimum
All $\lambda_i<0$	Strict local maximum
Mixed signs	Saddle point
Some $\lambda_i=0$, rest same sign	Degenerate

Newton's method

$$\theta_{k+1} = \theta_k - H(\theta_k)^{-1}\,\nabla L(\theta_k).$$

Rescales each direction by its curvature; quadratic convergence near a minimum. Cost $O(n^3)$ motivates quasi-Newton (L-BFGS) and second-order preconditioners (Shampoo, Muon).

10. Backpropagation

Introduction

A simple calculation path changes an input by factor 2 and then changes loss by factor negative 3. Reverse-mode differentiation multiplies those local effects to obtain total derivative negative 6.

Learning goal: Trace reverse-mode gradients through a computation graph and connect the process to neural-network backpropagation equations.
Before you start: Chain rule, gradients, matrix-vector products, and basic neural-network layer notation.

Lesson plan

Compare forward mode and reverse mode by the derivatives each one propagates.
Read the four backpropagation equations and identify every local factor.
Walk backward through the interactive graph while accumulating shared gradient contributions.

Trace one path: if x changes u by a factor of 2 and u changes the loss by a factor of -3, then x changes the loss by 2×(-3)=-6. Backpropagation reuses such local factors from output to input.

For a composition $L=\ell\circ f_K\circ\cdots\circ f_1(\theta)$, the chain rule gives $\nabla_\theta L = J_1^\top\cdots J_K^\top \nabla_{f_K}\ell$. Naively computing all $n$ components costs $O(n)$ forward passes. Backprop evaluates the entire gradient in one reverse pass.

10.1 Forward vs. reverse mode

Mode	Direction	Cost	Good when
Forward	left→right	$n$ forward passes	$n\ll m$
Reverse	right→left	1 forward + 1 reverse	$m\ll n$ (always in ML)

10.2 The four backprop equations

For $z^{(\ell)}=W^{(\ell)}a^{(\ell-1)}+b^{(\ell)}$, $a^{(\ell)}=\sigma(z^{(\ell)})$, with $\delta^{(\ell)}=\nabla_{z^{(\ell)}}L$:

$$\begin{aligned} \delta^{(L)} &= \nabla_{a^{(L)}}L \odot \sigma'(z^{(L)}) \\ \delta^{(\ell)} &= (W^{(\ell+1)})^\top \delta^{(\ell+1)} \odot \sigma'(z^{(\ell)}) \\ \nabla_{W^{(\ell)}}L &= \delta^{(\ell)}(a^{(\ell-1)})^\top \\ \nabla_{b^{(\ell)}}L &= \delta^{(\ell)}. \end{aligned}$$

10.3 Computation-graph walkthrough

Trace forward and backward passes through a tiny three-operation graph. Edit $x$ and watch both values and gradients propagate. The "local grad" column is the derivative of each node with respect to its input; backprop multiplies them right-to-left.

Interactive · trace the chain rule through a small graph

Node	Op	Value	Local grad	Upstream $\bar v$	Gradient at node

Input $x$ = 0.50 Graph: $x \xrightarrow{\times 2} a \xrightarrow{\sin} b \xrightarrow{(\cdot)^2} L$

11. Pathologies in Deep Learning

Introduction

Multiplying many local derivatives can erase or amplify a learning signal. With sigmoid derivatives bounded by one quarter, an eight-layer network chain can shrink a unit gradient below two hundred-thousandths.

Learning goal: Recognize vanishing gradients, exploding gradients, saddle points, and sharp-minimum behavior and connect each to practical remedies.
Before you start: Backpropagation through layers, repeated multiplication, activation functions, gradient vectors, and the idea of curvature.

Lesson plan

Calculate gradient decay through depth and inspect it with the interactive control.
Contrast vanishing and exploding gradient signals and review practical stabilization methods.
Distinguish saddle points from flat and sharp minima using local geometry.

Predict the symptom before choosing a remedy. Vanishing gradients produce tiny early-layer updates. Exploding gradients produce very large values or nan. Noisy gradients vary across batches. These failures can look similar if you inspect only the final loss.

11.1 Vanishing gradients

When each Jacobian $J_\ell$ has spectral radius $\rho(J_\ell)<1$, the chain-rule product $J_K^\top\cdots J_1^\top$ decays exponentially with depth. Early layers receive near-zero gradients. Fixes: ReLU activations, residual connections, careful initialisation (He, Xavier).

11.1.1 Interactive · gradient decay with depth

A deep chain of sigmoid units $a_{\ell+1}=\sigma(w\,a_\ell)$, with $L=a_N$. Drag the depth $N$ and weight $w$ to watch $|\partial L/\partial a_0|$ collapse. For $\sigma'(z)=\sigma(z)(1-\sigma(z))\le 1/4$, the gradient magnitude is bounded by $(|w|/4)^N$.

Interactive · $a_{\ell+1}=\sigma(w\,a_\ell)$, $L=a_N$

Depth $N$ 10 Weight $w$ 1.00 $a_0$ 0.00

11.2 Exploding gradients

The dual problem ($\rho(J_\ell)>1$). Common in RNNs. Fix: gradient clipping $g \leftarrow g\cdot\min(1,\, c/\|g\|)$, typically $c=1$.

11.3 Saddle points

In high dimensions, most critical points are saddles, not minima. The loss is flat, gradients are tiny, SGD crawls. Momentum and stochasticity help escape.

11.4 Flat vs. sharp minima

Flat minima generalise better (Hochreiter & Schmidhuber, 1997). SGD's implicit bias toward them is one of the deep-learning regularisation effects.

12. Modern Optimisers

Introduction

Two optimizers can receive the same current gradient yet take different steps because they remember different history. SGD uses the present value, while momentum, RMSProp, and AdamW track running first or second moments.

Learning goal: Compare major gradient optimizers by their stored state, update behavior, assumptions, and appropriate diagnostic questions.
Before you start: Gradient descent, learning rates, moving averages, squared values, and parameter updates.

Lesson plan

Start from plain SGD and identify what information one update uses.
Add first-moment and second-moment histories to understand momentum and adaptive scaling.
Compare AdamW and newer methods without treating any optimizer as universally best.

Optimizers change how gradients become parameter steps. They do not replace the gradient calculation. Compare methods under the same initialization, batches, and evaluation rule so a changed result has one plausible cause.

Optimiser	Update	Notes
SGD	$\theta\leftarrow\theta-\eta g$	Baseline.
SGD + momentum	$v\leftarrow\mu v-\eta g;\;\theta\leftarrow\theta+v$	Damps oscillation.
Nesterov	Evaluate $g$ at $\theta+\mu v$.	Look-ahead momentum.
AdaGrad	$\theta\leftarrow\theta - \frac{\eta}{\sqrt{G+\varepsilon}}g,\; G\mathrel{+}=g^2$	Per-parameter rate, decays.
RMSProp	EMA of $g^2$.	Non-stationary AdaGrad.
Adam	Momentum + RMSProp + bias correction.	Default for most DL.
AdamW	Decoupled weight decay.	LLM pre-training standard.
LAMB / LARS	Layer-wise scaling.	Large-batch distributed.
Shampoo	Precondition by $H^{-1/(2d)}$.	Approx 2nd-order at scale.
Muon	Polar decomposition of update.	Recent SOTA for LLMs.

Warning · Decoupled weight decay matters

In Adam, naive $L_2$ regularisation couples into the adaptive learning rate and is not equivalent to AdamW. Use AdamW for real weight decay.

$f$	function	A rule that maps inputs to an output.
$f(x)$	value of $f$ at $x$	The output you get when you feed $x$ in. Not "$f$ times $x$".
$x, y$	input variables	Usually scalars or vectors that the function consumes.
$\theta$	parameters	The tunable numbers of a model (weights, biases). $\theta \in \mathbb{R}^n$.
$w, b$	weight, bias	Common names for a single parameter pair in a linear layer.
$L$	loss	The scalar quantity you want to minimise (e.g. mean squared error).

$f'(x)$	derivative	Rate of change of a 1-D function at $x$.
$\dfrac{\partial f}{\partial x_i}$	partial derivative	Rate of change of $f$ when only $x_i$ moves; everything else frozen. Also written $f_{x_i}$ or $\partial_i f$.
$\nabla f$	gradient (del / nabla)	Stack of all partial derivatives into one column vector. Points in the direction of steepest ascent.
$\nabla_\theta L$	gradient w.r.t. $\theta$	"Take the gradient, treating $\theta$ as the variable and other symbols as constants."
$Df$	total derivative	The linear map behind the gradient. Row-vector form of the same information.
$D_u f$	directional derivative	Rate of change of $f$ as you move along the unit vector $u$. Equals $\langle \nabla f, u\rangle$.
$J$	Jacobian	Matrix of partial derivatives for a vector-valued function $f:\mathbb{R}^n\to\mathbb{R}^m$. Row $i$, column $j$: $\partial f_i/\partial x_j$.
$H$	Hessian	Matrix of second derivatives: $H_{ij}=\partial_i\partial_j L$. Captures curvature.
$o(h)$	little-o	"Anything that shrinks faster than $h$." Formalises "negligible at small $h$".

$\mathbb{R}^n$	$n$-dimensional reals	The set of all length-$n$ vectors of real numbers.
$\mathbf{v} \in \mathbb{R}^n$	"$\mathbf{v}$ lives in…"	"$\mathbf{v}$ is an $n$-dimensional real vector."
$\langle a, b \rangle$	inner product	Also $a^\top b$ or $a \cdot b$. Sum of coordinate-wise products. Measures alignment.
$\\|v\\|$	norm (length)	$\sqrt{\langle v, v\rangle}$. The Euclidean length of $v$.
$A^\top$	transpose	Flip a matrix across its diagonal: $(A^\top)_{ij}=A_{ji}$.
$A^{-1}$	inverse	Matrix such that $AA^{-1}=I$. Only exists for square, non-singular matrices.
$\lambda_i$	eigenvalue	Scalar where $Hv=\lambda v$ for some non-zero $v$. Measures how $H$ stretches along $v$.
$\odot$	Hadamard product	Element-wise multiplication: $(a\odot b)_i = a_i b_i$.

$\eta$	eta — learning rate	Step size in gradient descent. Too big → diverge; too small → crawl.
$\lambda$	lambda — eigenvalue / regulariser	As eigenvalue of $H$: curvature. As $\lambda\\|\theta\\|^2$ in loss: weight-decay strength.
$\varepsilon$	epsilon — small number	Used in numerical checks ($\sim 10^{-5}$) and stabilisers ($\sim 10^{-8}$).
$\mu$	mu — momentum coefficient	In SGD+momentum: how much of yesterday's velocity to keep. Typical: $0.9$.
$\sigma$	sigma — activation function	Usually the sigmoid $\sigma(z)=1/(1+e^{-z})$. Also used for standard deviation.
$\rho$	rho — spectral radius	Largest $\|\lambda\|$ of a matrix. $\rho(J)<1$ means gradients vanish along a chain.
$\kappa$	kappa — condition number	$\lambda_{\max}/\lambda_{\min}$ of $H$. Large $\kappa$ = ill-conditioned = slow descent.
$\ell$	ell — per-example loss / layer index	$\ell(f_\theta(x), y)$ is the loss on one data point; superscript $(\ell)$ indexes layers.
$\delta^{(\ell)}$	delta — backprop error	$\delta^{(\ell)} = \nabla_{z^{(\ell)}} L$, the gradient at layer $\ell$'s pre-activation.

$\in$	"is in"	$x \in \mathbb{R}$ means "$x$ is a real number."
$\forall$	"for all"	$\forall i$: for every $i$.
$\approx$	"approximately"	$L(\theta+h)\approx L(\theta)+\langle \nabla L, h\rangle$.
$\propto$	"proportional to"	"Scales with…", ignoring constants.
$i, j, k$	indices	Integer indices, usually summed over. $a_i$: the $i$-th component of $a$.

$f_\theta$	parameterised model	A function whose behaviour depends on tunable $\theta$.
$z^{(\ell)}$	pre-activation	Linear output of layer $\ell$ before the non-linearity.
$a^{(\ell)}$	activation	Post-nonlinearity output: $a^{(\ell)}=\sigma(z^{(\ell)})$.
$g$ or $g_k$	gradient shorthand	$g_k = \nabla L(\theta_k)$, the gradient at iteration $k$.
$v$	velocity	In momentum methods: running average of past gradients.
$m, v$ (Adam)	1st / 2nd moment	$m$: EMA of $g$. $v$: EMA of $g^2$. Bias-corrected before use.
$\mathbb{E}[\cdot]$	expectation	Average over randomness (data, mini-batches, actions).