Lesson 5

Tokenization, vocabularies, and sequence batches

Introduction

The text 'hi hi' becomes token IDs 0, 1, 2, 0, 1. Sliding a short window over these IDs creates inputs and targets shifted by one position for next-token learning.

Learning goal

Tokenize a short text, create correctly shifted sequence pairs, and group them into shape-safe mini-batches.

Before you start

Python strings, dictionaries, lists, tensor shapes, slicing, and integer data types.

Lesson plan

  1. Define a tiny vocabulary and encode text as categorical integer IDs.
  2. Build sliding input-target windows and verify the one-position shift by hand.
  3. Batch examples without shuffling positions or leaking overlapping windows across splits.

Why text needs preparation

PyTorch layers receive tensors, not Python strings. We therefore need a tokenizer: a rule that splits text into units called tokens and maps each token to an integer ID. A practical tokenizer may use words or word pieces. For learning, characters make every step visible. In our tiny vocabulary, h = 0, i = 1, and space = 2. The text hi hi becomes [0, 1, 2, 0, 1].

The numbers are category labels. They are not measurements. We can encode text with a token-to-ID dictionary and decode IDs with the reverse dictionary. The training target is created by shifting the text one place left: each input position is paired with the token that actually came next.

Terms and shape names

VocabularyThe complete set of tokens the model knows.
TokenizerThe rule that converts text to tokens and IDs.
SequenceAn ordered row of token IDs.
Context lengthThe number of input positions shown at once.
BatchSeveral sequences placed on the first tensor axis.
TargetThe correct answer used to calculate loss.

We will use shape (B, T), where B means batch size and T means sequence length. Both input IDs and target IDs have integer dtype torch.long. A later model will turn each integer into learned floating-point features.

Worked example: sliding windows

Predict first: for token IDs [0, 1, 2, 3, 4] and a context length of three, the first input should be [0, 1, 2]. Write the one target ID that must follow it. Then shift the window once and write the second pair.

With context length three, the first four IDs [0, 1, 2, 0] produce input [0, 1, 2] and target [1, 2, 0]. The next window moves one place. Every target position is exactly one token ahead of the matching input position.

import torch

text = "hi hi"
characters = ["h", "i", " "]
to_id = {character: index for index, character in enumerate(characters)}
to_character = {index: character for character, index in to_id.items()}

ids = torch.tensor([to_id[c] for c in text], dtype=torch.long)
block_size = 3

inputs = []
targets = []
for start in range(len(ids) - block_size):
    window = ids[start:start + block_size + 1]
    inputs.append(window[:-1])
    targets.append(window[1:])

x = torch.stack(inputs)
y = torch.stack(targets)
decoded = "".join(to_character[i] for i in ids.tolist())

print(ids.tolist())
print(x.tolist())
print(y.tolist())
print(tuple(x.shape), x.dtype)
print(decoded)

# Expected:
# [0, 1, 2, 0, 1]
# [[0, 1, 2], [1, 2, 0]]
# [[1, 2, 0], [2, 0, 1]]
# (2, 3) torch.int64
# hi hi

Notice that one source string produces several training examples. This reuses the text efficiently. For a larger corpus, you would usually split the token IDs into training and validation sections first, then create windows inside each section. Splitting after windows are made can leak nearly identical sequences into both sets.

Mini-batches

For the prediction above, the first target is 3. The next pair is input [1, 2, 3] and target 4. If you accidentally use target 2 for the first row, the model learns to copy the final input token instead of predicting the next one. Print one decoded input-target pair before training to catch this off-by-one error.

A Dataset describes how many examples exist and how to retrieve one. A DataLoader groups examples into batches, optionally shuffles their order, and can load data in worker processes. For two already prepared examples, TensorDataset is enough.

from torch.utils.data import DataLoader, TensorDataset

loader = DataLoader(TensorDataset(x, y), batch_size=2, shuffle=False)
batch_x, batch_y = next(iter(loader))
print(tuple(batch_x.shape), tuple(batch_y.shape))
# Expected: (2, 3) (2, 3)

Shuffling changes which examples share a batch, not the order of tokens inside each sequence. Never shuffle token positions within a sequence: language meaning depends on order.

Common pitfalls

  • Build the vocabulary deterministically. For example, sort unique characters or save the original token map.
  • Keep IDs as torch.long for embedding lookup and class targets.
  • Do not let a training window cross from the training split into validation text.
  • Check that input and target are shifted by exactly one position.
  • Real tokenizers need an unknown-token policy; a dictionary lookup otherwise fails on unseen text.

Try it

Use the text hihi and context length two. Write all input-target sequence pairs by hand, then create them with code.

Reveal the worked answer

The windows are [h, i, h] and [i, h, i]. Inputs are [h, i] and [i, h]; targets are [i, h] and [h, i].

ids = torch.tensor([0, 1, 0, 1])
x = torch.stack([ids[0:2], ids[1:3]])
y = torch.stack([ids[1:3], ids[2:4]])
print(x.tolist())  # Expected: [[0, 1], [1, 0]]
print(y.tolist())  # Expected: [[1, 0], [0, 1]]

Recap

A tokenizer turns text into categorical integer IDs. A next-token target is the same sequence shifted by one place. Sliding windows create many examples, and batching gives tensors of shape (B, T). Preserve token order, split data before making overlapping windows, and keep IDs as integer tensors.

Reference: PyTorch data loading documentation.