Tensor by Tensor

02.21 · UNIT 06 · Sequence models and memory · Lesson

LSTM and GRU gates: controlling memory

LSTM and GRU gates control how much old and new information enters memory.

PLAIN-LANGUAGE INTRODUCTION

The idea

LSTM and GRU gates control how much old and new information enters memory.

A worked example

Old memory 2 with forget gate 0.75 keeps 1.5. New candidate 0.5 with input gate 0.4 adds 0.2. New memory is 1.7.

The input

The current token, previous state, and learned gate values.

The result

An updated memory and hidden state.

Why you use it

Gates give gradients and information a controlled path through time.

A limit to remember

A gate does not make a hard yes-or-no choice unless its value reaches 0 or 1.

Check your understanding

  1. A gate is a number between 0 and 1. It scales information.
  2. The forget gate keeps 0.75×2 = 1.5 from the old memory.
  3. The input gate adds 0.4×0.5 = 0.2 from the new candidate.
  4. The new long short-term memory (LSTM) cell value is 1.5+0.2 = 1.7.
  5. Common mistake: gate values are learned for each step. They are not fixed constants in a trained model.
Open the detailed notes ↗