Embedding lookup and learned vectors
Introduction
Use categorical token IDs to select trainable vectors without inventing numeric order between IDs.
- Learning goal
Track embedding shapes and explain how lookup gradients update selected rows.
- Before you start
Token IDs, trainable parameters, backpropagation, and vector similarity.
Lesson plan
- Index rows by token ID
- Track appended feature dimensions
- Inspect sparse row gradients
The problem
Token IDs are arbitrary category labels. If you feed ID 20 as a measured number, it falsely implies a value twice ID 10. An embedding table instead assigns a trainable feature vector to every vocabulary item.
Lookup itself does not create meaning. The training objective changes selected rows. Their geometry then helps to predict contexts, labels, or another target.
Work through a small example
Let a vocabulary have four tokens and an embedding width of two. The parameter table has shape (V=4,D=2). The lookup of IDs [2,0,2] selects rows 2, 0, and 2. The output has shape (T=3,D=2).
The general rule
For IDs shaped (B,T), nn.Embedding(V,D) returns (B,T,D). This result is mathematically the same as a multiplication of one-hot vectors by a (V,D) matrix. However, direct indexing does not build huge sparse one-hot tensors.
During backpropagation, only rows used in the batch receive lookup gradients. Repeated IDs accumulate gradient contributions. With ordinary dense optimizer state, memory may still scale with all V*D parameters.
Implement and inspect
import torch
from torch import nn
table = nn.Embedding(4, 2)
with torch.no_grad():
table.weight.copy_(torch.tensor([[1., 0.], [0., 1.], [2., 3.], [-1., 1.]]))
ids = torch.tensor([2, 0, 2])
vectors = table(ids)
loss = vectors.sum()
loss.backward()
print(vectors.detach().tolist())
print(table.weight.grad.tolist())
The expected vectors are [[2,3],[1,0],[2,3]]. The gradient rows are [[1,1],[0,0],[2,2],[0,0]]. Row 2 appears twice, row 0 appears once, and unused rows receive zero.
Engineering checks
- Use
torch.longIDs and assert0 <= id < V. - Set
padding_idxwhen padding should remain fixed. - Confirm tied input and output embeddings have compatible shapes.
- Evaluate vectors on a downstream task, not only a two-dimensional plot.
Go deeper
Embedding geometry has meaning only relative to the objective. A rotation of every vector, with a later compensation of the weights, can preserve model behavior. Therefore, individual coordinates usually lack a fixed human meaning.
Rare tokens receive fewer direct updates. Frequency-aware sampling, subword sharing, and regularization can help, but each method changes the learned relation. Static embeddings assign one vector for each token. Contextual models create a position-specific vector from the surrounding tokens.
Practice and recap
Question: IDs have shape (32,10), vocabulary size 5,000, and embedding width 64. Name the parameter and output shapes.
Worked answer
The table parameter has shape (5000,64). Lookup appends the feature axis, so the output has shape (32,10,64). The IDs remain categorical addresses. The returned values are floating-point learned features.
An embedding is an indexed parameter table. The objective and data—not the ID values—determine what relations its vectors encode.