02.15 · UNIT 04 · Sequence models and memory · Lesson
Recurrent neural networks and backpropagation through time
An RNN updates one hidden state while reading tokens in order.
PLAIN-LANGUAGE INTRODUCTION
What is this?
An RNN updates one hidden state while reading tokens in order.
One simple example
Use hₜ=0.5hₜ₋₁+xₜ with inputs [1,0,1]. Hidden states become [1,0.5,1.25].
What goes in?
The next token representation and the previous hidden state.
What comes out?
A new hidden state and optional prediction.
Why does it matter?
The hidden state carries earlier information into later steps.
What is it not?
The hidden state is a compressed summary. It is not the full earlier sequence.
WORK THROUGH THE IDEA
See the idea in more detail
- Recurrent neural network (RNN) means the same update rule runs at every sequence position.
- Start with
h₀=0. Forx₁=1, computeh₁=0.5×0+1=1. - For
x₂=0, computeh₂=0.5×1+0=0.5. - For
x₃=1, computeh₃=0.5×0.5+1=1.25. The values are illustrative. - Common mistake: resetting the hidden state between tokens removes the sequence memory.