02.24 · UNIT 07 · Transformer architecture and BERT-family encoders · Lab

Encoder-decoder transformers and teacher forcing

An encoder reads the source. A decoder predicts target tokens from source memory and earlier targets.

PLAIN-LANGUAGE INTRODUCTION

What is this?

An encoder reads the source. A decoder predicts target tokens from source memory and earlier targets.

One simple example

For source [2,5] and target [5,2], decoder input is [<start>,5]. Its labels are [5,2].

Source IDs, shifted target IDs, and their masks.

One next-token score vector at each target position.

Cross-attention lets each target position use the encoded source.

Teacher forcing during training does not match free generation exactly.

WORK THROUGH THE IDEA

The encoder turns source [2,5] into contextual source vectors called memory.
Shift target [5,2] right. Decoder input becomes [<start>,5].
The labels remain [5,2]. Position zero predicts 5. Position one predicts 2.
A causal mask hides later target tokens. Cross-attention can still read allowed source positions.
Common mistake: supplying the unshifted target lets the decoder see the answer at its own position.