02.27 · UNIT 08 · Train, evaluate, and serve your transformer · Lesson
Decoder-only language modeling and generation
A decoder-only language model predicts each next token from earlier tokens only.
PLAIN-LANGUAGE INTRODUCTION
What is this?
A decoder-only language model predicts each next token from earlier tokens only.
One simple example
For hello, training input is hell. Targets are ello. After prefix he, the next target is l.
What goes in?
A prefix of token IDs and a causal attention mask.
What comes out?
Next-token scores at each position, then generated tokens.
Why does it matter?
One shifted objective trains all prefix positions in parallel.
What is it not?
A small next-token model is not automatically a helpful chat system.
WORK THROUGH THE IDEA
See the idea in more detail
- Shift the sequence by one position. Input
hellaligns with targetello. - At input position one, the visible prefix is
he. Its correct next token isl. - A causal mask blocks later input positions. Training still evaluates all allowed positions together.
- Generation chooses one next token, appends it, then repeats with the longer prefix.
- Common mistake: forgetting the shift trains the model to copy its current token.