Tokenization and vocabulary design

Introduction

Define the stable text units and integer IDs that every later language model consumes.

Learning goal

Create padded token-ID batches and explain vocabulary, unknown-token, and coverage choices.

Before you start

Python strings and dictionaries, tensor shapes, and train-validation separation.

Lesson plan

  1. Choose text units
  2. Map tokens to stable IDs
  3. Pad and mask sequence batches

The problem

Models receive numeric tensors, not raw strings. A tokenizer divides text into units called tokens. A vocabulary gives each token a stable integer ID.

The split rule is part of the model. Word tokenization is easy to inspect but struggles with unseen words. Character tokenization avoids most unseen units but creates longer sequences. Subword tokenization balances these goals by learning reusable pieces.

Work through a small example

Use vocabulary <pad>=0, <unk>=1, cats=2, and nap=3. “cats nap” becomes [2,3]. To batch it with a three-token sequence, append padding: [2,3,0].

Text becomes tokens, IDs, and a padding maskThe text cats nap splits into two token circles. They map to ID circles two and three, followed by padding ID zero. A mask marks real tokens one, one, then padding zero. cats nap catsnap 230 mask 1mask 1mask 0

The IDs are category labels, not quantities. Token 3 is not “larger” or closer to token 2 than token 100 is. An embedding table later uses IDs as row addresses.

The general rule

A batch of IDs has shape (B,T), where B is batch size and T is padded sequence length. IDs use integer dtype torch.long. An attention mask of the same shape records which positions contain real tokens.

Fit a learned tokenizer and vocabulary on training text only. Freeze their exact version for validation, test, and deployment. Otherwise token meanings and sequence lengths can change after training.

Implement and inspect

import torch

vocabulary = {"<pad>": 0, "<unk>": 1, "cats": 2, "nap": 3, "quietly": 4}
texts = ["cats nap", "cats nap quietly"]
rows = []
for text in texts:
    rows.append([vocabulary.get(token, 1) for token in text.split()])

width = max(map(len, rows))
ids = torch.tensor([row + [0] * (width - len(row)) for row in rows])
mask = (ids != 0).long()
print(ids.tolist())
print(mask.tolist())
print(tuple(ids.shape), ids.dtype)

Expected IDs are [[2,3,0],[2,3,4]]; the mask is [[1,1,0],[1,1,1]]. Shape is (2,3) and dtype is torch.int64.

Engineering checks

Padding side matters for position-sensitive models. Truncation can remove label evidence, so record which side and maximum length are used.

Go deeper

Subword training is a compression and modeling choice. Frequent strings become single tokens; rarer strings decompose into reusable pieces. The resulting vocabulary affects compute because attention cost grows strongly with sequence length.

Tokenization can represent languages and writing styles unevenly. A fixed character count can become very different token counts, changing latency and available context. Evaluate coverage by language and domain rather than reporting only average length.

Practice and recap

Question: With the shown vocabulary, encode “dogs nap” and its mask at width three.

Worked answer

dogs is unknown, so IDs are [1,3,0]. The mask is [1,1,0]. Unknown is a real token position; padding is not.

Tokenization defines the model's input units. Freeze the mapping, preserve integer shapes, and test the text cases that deployment will contain.