Tensor by Tensor

02.20 · UNIT 06 · Sequence models and memory · Lesson

Recurrent neural networks and backpropagation through time

An RNN updates one hidden state while reading tokens in order.

PLAIN-LANGUAGE INTRODUCTION

The idea

An RNN updates one hidden state while reading tokens in order.

A worked example

Use hₜ=0.5hₜ₋₁+xₜ with inputs [1,0,1]. Hidden states become [1,0.5,1.25].

The input

The next token representation and the previous hidden state.

The result

A new hidden state and optional prediction.

Why you use it

The hidden state carries earlier information into later steps.

A limit to remember

The hidden state is a compressed summary. It is not the full earlier sequence.

Check your understanding

  1. Recurrent neural network (RNN) means the same update rule runs at every sequence position.
  2. Start with h₀=0. For x₁=1, compute h₁=0.5×0+1=1.
  3. For x₂=0, compute h₂=0.5×1+0=0.5.
  4. For x₃=1, compute h₃=0.5×0.5+1=1.25. The values are illustrative.
  5. Common mistake: resetting the hidden state between tokens removes the sequence memory.
Open the detailed notes ↗