02.29 · UNIT 09 · Transformer architecture and BERT-family encoders · Lab
Encoder-decoder transformers and teacher forcing
An encoder reads the source. A decoder predicts target tokens from source memory and earlier targets.
PLAIN-LANGUAGE INTRODUCTION
The idea
An encoder reads the source. A decoder predicts target tokens from source memory and earlier targets.
A worked example
For source
[2,5] and target [5,2], decoder input is [<start>,5]. Its labels are [5,2].The input
Source IDs, shifted target IDs, and their masks.
The result
One next-token score vector at each target position.
Why you use it
Cross-attention lets each target position use the encoded source.
A limit to remember
Teacher forcing during training does not match free generation exactly.
Check your understanding
- The encoder turns source
[2,5]into contextual source vectors called memory. - Shift target
[5,2]right. Decoder input becomes[<start>,5]. - The labels remain
[5,2]. Position zero predicts5. Position one predicts2. - A causal mask hides later target tokens. Cross-attention can still read allowed source positions.
- Common mistake: supplying the unshifted target lets the decoder see the answer at its own position.