02.16 · UNIT 05 · Text representations and neural language baselines · Lesson
Tokenization, vocabularies, and sequence batches
Tokenization splits text into pieces. A vocabulary maps each piece to an integer ID.
PLAIN-LANGUAGE INTRODUCTION
The idea
Tokenization splits text into pieces. A vocabulary maps each piece to an integer ID.
A worked example
With
cats→2, nap→3, and <pad>→0, “cats nap” becomes [2,3,0] after padding.The input
Text plus one fixed tokenizer and vocabulary.
The result
Token IDs and a mask showing which positions contain real tokens.
Why you use it
Models process numbers. Batching also needs equal sequence lengths.
A limit to remember
A token is not always one word.
Check your understanding
- The tokenizer turns “cats nap” into two tokens:
[cats, nap]. - The vocabulary maps them to IDs
[2,3]. IDs are labels, not measured quantities. - Padding adds
0, producing[2,3,0]. The attention mask becomes[1,1,0]. - Mask value
1marks a real token. Mask value0marks padding in this example. - Common mistake: changing the vocabulary mapping after training changes what every ID means.