Tensor by Tensor

02.29 · UNIT 09 · Transformer architecture and BERT-family encoders · Lab

Encoder-decoder transformers and teacher forcing

An encoder reads the source. A decoder predicts target tokens from source memory and earlier targets.

PLAIN-LANGUAGE INTRODUCTION

The idea

An encoder reads the source. A decoder predicts target tokens from source memory and earlier targets.

A worked example

For source [2,5] and target [5,2], decoder input is [<start>,5]. Its labels are [5,2].

The input

Source IDs, shifted target IDs, and their masks.

The result

One next-token score vector at each target position.

Why you use it

Cross-attention lets each target position use the encoded source.

A limit to remember

Teacher forcing during training does not match free generation exactly.

Check your understanding

  1. The encoder turns source [2,5] into contextual source vectors called memory.
  2. Shift target [5,2] right. Decoder input becomes [<start>,5].
  3. The labels remain [5,2]. Position zero predicts 5. Position one predicts 2.
  4. A causal mask hides later target tokens. Cross-attention can still read allowed source positions.
  5. Common mistake: supplying the unshifted target lets the decoder see the answer at its own position.
Open the detailed notes ↗