import numpy as np
def L(w, b): return (w-1)**2 + 4*(b+2)**2
def analytic(w, b): return np.array([2*(w-1), 8*(b+2)])
def numerical(w, b, eps=1e-5):
    dw = (L(w+eps, b) - L(w-eps, b)) / (2*eps)
    db = (L(w, b+eps) - L(w, b-eps)) / (2*eps)
    return np.array([dw, db])
a, n = analytic(3, -1), numerical(3, -1)
print(np.abs(a - n) / (np.abs(a) + 1e-12))   # ~1e-11

9. The Hessian and Conditioning

Introduction

Two directions can have different curvature even at the same point. A quadratic loss with Hessian entries 2 and 8 bends four times more strongly in one direction, affecting stability and convergence.

Learning goal: Read a Hessian matrix, classify critical points, measure conditioning, and explain one Newton update.
Before you start: Gradients, second derivatives, matrices, eigenvalues as directional scaling, and quadratic functions.

Lesson plan

Differentiate the gradient to form the matrix of second partial derivatives.
Use eigenvalue signs to classify minima, maxima, and saddle points.
Connect unequal curvature to conditioning, learning-rate limits, and Newton's method.

Picture two curvatures. A bowl can be steep left-to-right and shallow front-to-back. One learning rate then moves too aggressively on one axis and too slowly on the other. The Hessian records this local curvature interaction.

The second-order structure of $L$ is captured by

$$H_{ij}(\theta) = \partial_i\partial_j L(\theta),\qquad L(\theta+\Delta) = L(\theta)+\langle\nabla L,\Delta\rangle+\tfrac{1}{2}\Delta^\top H\,\Delta + o(\|\Delta\|^2).$$

Classifying critical points ($\nabla L=0$)

Spectrum of $H$	Local behaviour
All $\lambda_i>0$	Strict local minimum
All $\lambda_i<0$	Strict local maximum
Mixed signs	Saddle point
Some $\lambda_i=0$, rest same sign	Degenerate

Newton's method

$$\theta_{k+1} = \theta_k - H(\theta_k)^{-1}\,\nabla L(\theta_k).$$

Rescales each direction by its curvature; quadratic convergence near a minimum. Cost $O(n^3)$ motivates quasi-Newton (L-BFGS) and second-order preconditioners (Shampoo, Muon).

10. Backpropagation

Introduction

A simple calculation path changes an input by factor 2 and then changes loss by factor negative 3. Reverse-mode differentiation multiplies those local effects to obtain total derivative negative 6.

Learning goal: Trace reverse-mode gradients through a computation graph and connect the process to neural-network backpropagation equations.
Before you start: Chain rule, gradients, matrix-vector products, and basic neural-network layer notation.

Lesson plan

Compare forward mode and reverse mode by the derivatives each one propagates.
Read the four backpropagation equations and identify every local factor.
Walk backward through the interactive graph while accumulating shared gradient contributions.

Trace one path: if x changes u by a factor of 2 and u changes the loss by a factor of -3, then x changes the loss by 2×(-3)=-6. Backpropagation reuses such local factors from output to input.

For a composition $L=\ell\circ f_K\circ\cdots\circ f_1(\theta)$, the chain rule gives $\nabla_\theta L = J_1^\top\cdots J_K^\top \nabla_{f_K}\ell$. Naively computing all $n$ components costs $O(n)$ forward passes. Backprop evaluates the entire gradient in one reverse pass.

10.1 Forward vs. reverse mode

Mode	Direction	Cost	Good when
Forward	left→right	$n$ forward passes	$n\ll m$
Reverse	right→left	1 forward + 1 reverse	$m\ll n$ (always in ML)

10.2 The four backprop equations

For $z^{(\ell)}=W^{(\ell)}a^{(\ell-1)}+b^{(\ell)}$, $a^{(\ell)}=\sigma(z^{(\ell)})$, with $\delta^{(\ell)}=\nabla_{z^{(\ell)}}L$:

$$\begin{aligned} \delta^{(L)} &= \nabla_{a^{(L)}}L \odot \sigma'(z^{(L)}) \\ \delta^{(\ell)} &= (W^{(\ell+1)})^\top \delta^{(\ell+1)} \odot \sigma'(z^{(\ell)}) \\ \nabla_{W^{(\ell)}}L &= \delta^{(\ell)}(a^{(\ell-1)})^\top \\ \nabla_{b^{(\ell)}}L &= \delta^{(\ell)}. \end{aligned}$$

10.3 Computation-graph walkthrough

Trace forward and backward passes through a tiny three-operation graph. Edit $x$ and watch both values and gradients propagate. The "local grad" column is the derivative of each node with respect to its input; backprop multiplies them right-to-left.

Interactive · trace the chain rule through a small graph

Node	Op	Value	Local grad	Upstream $\bar v$	Gradient at node

Input $x$ = 0.50 Graph: $x \xrightarrow{\times 2} a \xrightarrow{\sin} b \xrightarrow{(\cdot)^2} L$

11. Pathologies in Deep Learning

Introduction

Multiplying many local derivatives can erase or amplify a learning signal. With sigmoid derivatives bounded by one quarter, an eight-layer network chain can shrink a unit gradient below two hundred-thousandths.

Learning goal: Recognize vanishing gradients, exploding gradients, saddle points, and sharp-minimum behavior and connect each to practical remedies.
Before you start: Backpropagation through layers, repeated multiplication, activation functions, gradient vectors, and the idea of curvature.

Lesson plan

Calculate gradient decay through depth and inspect it with the interactive control.
Contrast vanishing and exploding gradient signals and review practical stabilization methods.
Distinguish saddle points from flat and sharp minima using local geometry.

Predict the symptom before choosing a remedy. Vanishing gradients produce tiny early-layer updates. Exploding gradients produce very large values or nan. Noisy gradients vary across batches. These failures can look similar if you inspect only the final loss.

11.1 Vanishing gradients

When each Jacobian $J_\ell$ has spectral radius $\rho(J_\ell)<1$, the chain-rule product $J_K^\top\cdots J_1^\top$ decays exponentially with depth. Early layers receive near-zero gradients. Fixes: ReLU activations, residual connections, careful initialisation (He, Xavier).

11.1.1 Interactive · gradient decay with depth

A deep chain of sigmoid units $a_{\ell+1}=\sigma(w\,a_\ell)$, with $L=a_N$. Drag the depth $N$ and weight $w$ to watch $|\partial L/\partial a_0|$ collapse. For $\sigma'(z)=\sigma(z)(1-\sigma(z))\le 1/4$, the gradient magnitude is bounded by $(|w|/4)^N$.

Interactive · $a_{\ell+1}=\sigma(w\,a_\ell)$, $L=a_N$

Depth $N$ 10 Weight $w$ 1.00 $a_0$ 0.00

11.2 Exploding gradients

The dual problem ($\rho(J_\ell)>1$). Common in RNNs. Fix: gradient clipping $g \leftarrow g\cdot\min(1,\, c/\|g\|)$, typically $c=1$.

11.3 Saddle points

In high dimensions, most critical points are saddles, not minima. The loss is flat, gradients are tiny, SGD crawls. Momentum and stochasticity help escape.

11.4 Flat vs. sharp minima

Flat minima generalise better (Hochreiter & Schmidhuber, 1997). SGD's implicit bias toward them is one of the deep-learning regularisation effects.

12. Modern Optimisers

Introduction

Two optimizers can receive the same current gradient yet take different steps because they remember different history. SGD uses the present value, while momentum, RMSProp, and AdamW track running first or second moments.

Learning goal: Compare major gradient optimizers by their stored state, update behavior, assumptions, and appropriate diagnostic questions.
Before you start: Gradient descent, learning rates, moving averages, squared values, and parameter updates.

Lesson plan

Start from plain SGD and identify what information one update uses.
Add first-moment and second-moment histories to understand momentum and adaptive scaling.
Compare AdamW and newer methods without treating any optimizer as universally best.

Optimizers change how gradients become parameter steps. They do not replace the gradient calculation. Compare methods under the same initialization, batches, and evaluation rule so a changed result has one plausible cause.

Optimiser	Update	Notes
SGD	$\theta\leftarrow\theta-\eta g$	Baseline.
SGD + momentum	$v\leftarrow\mu v-\eta g;\;\theta\leftarrow\theta+v$	Damps oscillation.
Nesterov	Evaluate $g$ at $\theta+\mu v$.	Look-ahead momentum.
AdaGrad	$\theta\leftarrow\theta - \frac{\eta}{\sqrt{G+\varepsilon}}g,\; G\mathrel{+}=g^2$	Per-parameter rate, decays.
RMSProp	EMA of $g^2$.	Non-stationary AdaGrad.
Adam	Momentum + RMSProp + bias correction.	Default for most DL.
AdamW	Decoupled weight decay.	LLM pre-training standard.
LAMB / LARS	Layer-wise scaling.	Large-batch distributed.
Shampoo	Precondition by $H^{-1/(2d)}$.	Approx 2nd-order at scale.
Muon	Polar decomposition of update.	Recent SOTA for LLMs.

Warning · Decoupled weight decay matters

In Adam, naive $L_2$ regularisation couples into the adaptive learning rate and is not equivalent to AdamW. Use AdamW for real weight decay.

14. References

Introduction

This reading list supports different next steps rather than one linear lesson. It groups sources for mathematical foundations, neural-network backpropagation, convex optimization, and the design and behavior of modern optimizers.

Learning goal: Choose an appropriate source for a specific gap and understand what each reference category contributes.
Before you start: Familiarity with the main gradient topics in this guide and a question to investigate.

Lesson plan

Use foundation texts to strengthen derivatives, gradients, and deep-learning notation.
Choose neural-network or convex-optimization sources for proofs and broader context.
Read optimizer surveys and papers when comparing update rules and practical assumptions.

Goodfellow, Bengio, Courville. Deep Learning, §4.3–4.4.
Nielsen. Neural Networks and Deep Learning, Ch. 2.
Boyd & Vandenberghe. Convex Optimization.
Ruder. "An overview of gradient descent optimization algorithms." arXiv:1609.04747.
Kingma & Ba. "Adam." arXiv:1412.6980.
Loshchilov & Hutter. "Decoupled Weight Decay Regularization." arXiv:1711.05101.
Jordan. "Dynamical, Symplectic and Stochastic Perspectives on Gradient-Based Optimization." ICM 2018.

$f$	function	A rule that maps inputs to an output.
$f(x)$	value of $f$ at $x$	The output you get when you feed $x$ in. Not "$f$ times $x$".
$x, y$	input variables	Usually scalars or vectors that the function consumes.
$\theta$	parameters	The tunable numbers of a model (weights, biases). $\theta \in \mathbb{R}^n$.
$w, b$	weight, bias	Common names for a single parameter pair in a linear layer.
$L$	loss	The scalar quantity you want to minimise (e.g. mean squared error).

$f'(x)$	derivative	Rate of change of a 1-D function at $x$.
$\dfrac{\partial f}{\partial x_i}$	partial derivative	Rate of change of $f$ when only $x_i$ moves; everything else frozen. Also written $f_{x_i}$ or $\partial_i f$.
$\nabla f$	gradient (del / nabla)	Stack of all partial derivatives into one column vector. Points in the direction of steepest ascent.
$\nabla_\theta L$	gradient w.r.t. $\theta$	"Take the gradient, treating $\theta$ as the variable and other symbols as constants."
$Df$	total derivative	The linear map behind the gradient. Row-vector form of the same information.
$D_u f$	directional derivative	Rate of change of $f$ as you move along the unit vector $u$. Equals $\langle \nabla f, u\rangle$.
$J$	Jacobian	Matrix of partial derivatives for a vector-valued function $f:\mathbb{R}^n\to\mathbb{R}^m$. Row $i$, column $j$: $\partial f_i/\partial x_j$.
$H$	Hessian	Matrix of second derivatives: $H_{ij}=\partial_i\partial_j L$. Captures curvature.
$o(h)$	little-o	"Anything that shrinks faster than $h$." Formalises "negligible at small $h$".

$\mathbb{R}^n$	$n$-dimensional reals	The set of all length-$n$ vectors of real numbers.
$\mathbf{v} \in \mathbb{R}^n$	"$\mathbf{v}$ lives in…"	"$\mathbf{v}$ is an $n$-dimensional real vector."
$\langle a, b \rangle$	inner product	Also $a^\top b$ or $a \cdot b$. Sum of coordinate-wise products. Measures alignment.
$\\|v\\|$	norm (length)	$\sqrt{\langle v, v\rangle}$. The Euclidean length of $v$.
$A^\top$	transpose	Flip a matrix across its diagonal: $(A^\top)_{ij}=A_{ji}$.
$A^{-1}$	inverse	Matrix such that $AA^{-1}=I$. Only exists for square, non-singular matrices.
$\lambda_i$	eigenvalue	Scalar where $Hv=\lambda v$ for some non-zero $v$. Measures how $H$ stretches along $v$.
$\odot$	Hadamard product	Element-wise multiplication: $(a\odot b)_i = a_i b_i$.

$\eta$	eta — learning rate	Step size in gradient descent. Too big → diverge; too small → crawl.
$\lambda$	lambda — eigenvalue / regulariser	As eigenvalue of $H$: curvature. As $\lambda\\|\theta\\|^2$ in loss: weight-decay strength.
$\varepsilon$	epsilon — small number	Used in numerical checks ($\sim 10^{-5}$) and stabilisers ($\sim 10^{-8}$).
$\mu$	mu — momentum coefficient	In SGD+momentum: how much of yesterday's velocity to keep. Typical: $0.9$.
$\sigma$	sigma — activation function	Usually the sigmoid $\sigma(z)=1/(1+e^{-z})$. Also used for standard deviation.
$\rho$	rho — spectral radius	Largest $\|\lambda\|$ of a matrix. $\rho(J)<1$ means gradients vanish along a chain.
$\kappa$	kappa — condition number	$\lambda_{\max}/\lambda_{\min}$ of $H$. Large $\kappa$ = ill-conditioned = slow descent.
$\ell$	ell — per-example loss / layer index	$\ell(f_\theta(x), y)$ is the loss on one data point; superscript $(\ell)$ indexes layers.
$\delta^{(\ell)}$	delta — backprop error	$\delta^{(\ell)} = \nabla_{z^{(\ell)}} L$, the gradient at layer $\ell$'s pre-activation.

$\in$	"is in"	$x \in \mathbb{R}$ means "$x$ is a real number."
$\forall$	"for all"	$\forall i$: for every $i$.
$\approx$	"approximately"	$L(\theta+h)\approx L(\theta)+\langle \nabla L, h\rangle$.
$\propto$	"proportional to"	"Scales with…", ignoring constants.
$i, j, k$	indices	Integer indices, usually summed over. $a_i$: the $i$-th component of $a$.

$f_\theta$	parameterised model	A function whose behaviour depends on tunable $\theta$.
$z^{(\ell)}$	pre-activation	Linear output of layer $\ell$ before the non-linearity.
$a^{(\ell)}$	activation	Post-nonlinearity output: $a^{(\ell)}=\sigma(z^{(\ell)})$.
$g$ or $g_k$	gradient shorthand	$g_k = \nabla L(\theta_k)$, the gradient at iteration $k$.
$v$	velocity	In momentum methods: running average of past gradients.
$m, v$ (Adam)	1st / 2nd moment	$m$: EMA of $g$. $v$: EMA of $g^2$. Bias-corrected before use.
$\mathbb{E}[\cdot]$	expectation	Average over randomness (data, mini-batches, actions).

Further reading

14. References

Introduction

Lesson plan