Tensor by Tensor

02.16 · UNIT 05 · Text representations and neural language baselines · Lesson

Tokenization, vocabularies, and sequence batches

Tokenization splits text into pieces. A vocabulary maps each piece to an integer ID.

PLAIN-LANGUAGE INTRODUCTION

The idea

Tokenization splits text into pieces. A vocabulary maps each piece to an integer ID.

A worked example

With cats→2, nap→3, and <pad>→0, “cats nap” becomes [2,3,0] after padding.

The input

Text plus one fixed tokenizer and vocabulary.

The result

Token IDs and a mask showing which positions contain real tokens.

Why you use it

Models process numbers. Batching also needs equal sequence lengths.

A limit to remember

A token is not always one word.

Check your understanding

  1. The tokenizer turns “cats nap” into two tokens: [cats, nap].
  2. The vocabulary maps them to IDs [2,3]. IDs are labels, not measured quantities.
  3. Padding adds 0, producing [2,3,0]. The attention mask becomes [1,1,0].
  4. Mask value 1 marks a real token. Mask value 0 marks padding in this example.
  5. Common mistake: changing the vocabulary mapping after training changes what every ID means.
Open the detailed notes ↗