02.28 · UNIT 08 · Train, evaluate, and serve your transformer · Lab
Lab: train complete transformer models
A complete transformer lab joins data, masks, loss, updates, and held-out generation.
PLAIN-LANGUAGE INTRODUCTION
What is this?
A complete transformer lab joins data, masks, loss, updates, and held-out generation.
One simple example
For reversal, source [2,5,7] has target [7,5,2]. Evaluation must generate all three target tokens.
What goes in?
Training batches, shifted targets, masks, and a fixed model configuration.
What comes out?
Updated parameters and free-running held-out predictions.
Why does it matter?
The lab proves that separate components form one learning system.
What is it not?
Low teacher-forced loss does not prove correct free generation.
WORK THROUGH THE IDEA
See the idea in more detail
- This task maps source
[2,5,7]to target[7,5,2]. Keep some sequences held out. - Training uses shifted targets. The loss compares each predicted next token with its correct label.
- An optimizer updates parameters after backpropagation. Record the configuration and random seed.
- During evaluation, generate
[7,5,2]without supplying correct earlier target tokens. - Common mistake: a separately sampled test tensor can still repeat a generated training sequence. Check overlap.