02.11 · UNIT 03 · Text representations and neural language baselines · Lesson

Tokenization, vocabularies, and sequence batches

Tokenization splits text into pieces. A vocabulary maps each piece to an integer ID.

PLAIN-LANGUAGE INTRODUCTION

What is this?

Tokenization splits text into pieces. A vocabulary maps each piece to an integer ID.

One simple example

With cats→2, nap→3, and <pad>→0, “cats nap” becomes [2,3,0] after padding.

Text plus one fixed tokenizer and vocabulary.

Token IDs and a mask showing which positions contain real tokens.

Models process numbers. Batching also needs equal sequence lengths.

A token is not always one word.

WORK THROUGH THE IDEA

The tokenizer turns “cats nap” into two tokens: [cats, nap].
The vocabulary maps them to IDs [2,3]. IDs are labels, not measured quantities.
Padding adds 0, producing [2,3,0]. The attention mask becomes [1,1,0].
Mask value 1 marks a real token. Mask value 0 marks padding in this example.
Common mistake: changing the vocabulary mapping after training changes what every ID means.