MODULE 01 · LESSON 04
Gradients
A gradient collects one local slope for every adjustable input.
PLAIN-LANGUAGE INTRODUCTION
What is this?
A gradient collects one local slope for every adjustable input.
One simple example
For L=(w−1)²+(b+2)² at (3,−1), the gradient is [4,2].
What goes in?
A loss function and current values. Here, w=3 and b=−1.
What comes out?
One slope per value. Here, ∂L/∂w=4 and ∂L/∂b=2.
Why does it matter?
Moving opposite the gradient can lower the loss.
What is it not?
A gradient is not the next parameter value. A learning rate sets the move size.
WORK THROUGH THE IDEA
See the idea in more detail
- The symbol
∂means we change one input while holding the other fixed. - At
(w,b)=(3,−1), the loss is2² + 1² = 5. - The two slopes are
2(w−1)=4and2(b+2)=2. The gradient is[4,2]. - With learning rate
0.1, descent gives(2.6,−1.2). The new loss is3.2. - Common mistake: adding the gradient increases this loss. Descent subtracts it.