Tensor by Tensor

02.32 · UNIT 10 · Train, evaluate, and serve your transformer · Lesson

Decoder-only language modeling and generation

A decoder-only language model predicts each next token from earlier tokens only.

PLAIN-LANGUAGE INTRODUCTION

The idea

A decoder-only language model predicts each next token from earlier tokens only.

A worked example

For hello, training input is hell. Targets are ello. After prefix he, the next target is l.

The input

A prefix of token IDs and a causal attention mask.

The result

Next-token scores at each position, then generated tokens.

Why you use it

One shifted objective trains all prefix positions in parallel.

A limit to remember

A small next-token model is not automatically a helpful chat system.

Check your understanding

  1. Shift the sequence by one position. Input hell aligns with target ello.
  2. At input position one, the visible prefix is he. Its correct next token is l.
  3. A causal mask blocks later input positions. Training still evaluates all allowed positions together.
  4. Generation chooses one next token, appends it, then repeats with the longer prefix.
  5. Common mistake: forgetting the shift trains the model to copy its current token.
Open the detailed notes ↗