Tensor by Tensor

02.33 · UNIT 10 · Train, evaluate, and serve your transformer · Lab

Lab: train complete transformer models

A complete transformer lab joins data, masks, loss, updates, and held-out generation.

PLAIN-LANGUAGE INTRODUCTION

The idea

A complete transformer lab joins data, masks, loss, updates, and held-out generation.

A worked example

For reversal, source [2,5,7] has target [7,5,2]. Evaluation must generate all three target tokens.

The input

Training batches, shifted targets, masks, and a fixed model configuration.

The result

Updated parameters and free-running held-out predictions.

Why you use it

The lab proves that separate components form one learning system.

A limit to remember

Low teacher-forced loss does not prove correct free generation.

Check your understanding

  1. This task maps source [2,5,7] to target [7,5,2]. Keep some sequences held out.
  2. Training uses shifted targets. The loss compares each predicted next token with its correct label.
  3. An optimizer updates parameters after backpropagation. Record the configuration and random seed.
  4. During evaluation, generate [7,5,2] without supplying correct earlier target tokens.
  5. Common mistake: a separately sampled test tensor can still repeat a generated training sequence. Check overlap.
Open the detailed notes ↗