Tensor by Tensor

MODULE 01 · LESSON 04

Gradients

A gradient collects one local slope for every adjustable input.

PLAIN-LANGUAGE INTRODUCTION

The idea

A gradient collects one local slope for every adjustable input.

A worked example

For L=(w−1)²+(b+2)² at (3,−1), the gradient is [4,2].

The input

A loss function and current values. Here, w=3 and b=−1.

The result

One slope per value. Here, ∂L/∂w=4 and ∂L/∂b=2.

Why you use it

Moving opposite the gradient can lower the loss.

A limit to remember

A gradient is not the next parameter value. A learning rate sets the move size.

Check your understanding

  1. The symbol ∂ means we change one input while holding the other fixed.
  2. At (w,b)=(3,−1), the loss is 2² + 1² = 5.
  3. The two slopes are 2(w−1)=4 and 2(b+2)=2. The gradient is [4,2].
  4. With learning rate 0.1, descent gives (2.6,−1.2). The new loss is 3.2.
  5. Common mistake: adding the gradient increases this loss. Descent subtracts it.
Open the detailed notes ↗