Embedding lookup and learned vectors

Introduction

Use categorical token IDs to select trainable vectors without inventing numeric order between IDs.

Learning goal

Track embedding shapes and explain how lookup gradients update selected rows.

Before you start

Token IDs, trainable parameters, backpropagation, and vector similarity.

Lesson plan

  1. Index rows by token ID
  2. Track appended feature dimensions
  3. Inspect sparse row gradients

The problem

Token IDs are arbitrary category labels. If you feed ID 20 as a measured number, it falsely implies a value twice ID 10. An embedding table instead assigns a trainable feature vector to every vocabulary item.

Lookup itself does not create meaning. The training objective changes selected rows. Their geometry then helps to predict contexts, labels, or another target.

Work through a small example

Let a vocabulary have four tokens and an embedding width of two. The parameter table has shape (V=4,D=2). The lookup of IDs [2,0,2] selects rows 2, 0, and 2. The output has shape (T=3,D=2).

Integer IDs select rows from an embedding tableThree ID circles, two, zero, and two, point to rows in a four-by-two table. The selected rows form a three-by-two output. Row two appears twice. 202 row 0row 1row 2row 31, 00, 12, 3−1, 1 [2, 3][1, 0][2, 3]

The general rule

For IDs shaped (B,T), nn.Embedding(V,D) returns (B,T,D). This result is mathematically the same as a multiplication of one-hot vectors by a (V,D) matrix. However, direct indexing does not build huge sparse one-hot tensors.

During backpropagation, only rows used in the batch receive lookup gradients. Repeated IDs accumulate gradient contributions. With ordinary dense optimizer state, memory may still scale with all V*D parameters.

Implement and inspect

import torch
from torch import nn

table = nn.Embedding(4, 2)
with torch.no_grad():
    table.weight.copy_(torch.tensor([[1., 0.], [0., 1.], [2., 3.], [-1., 1.]]))

ids = torch.tensor([2, 0, 2])
vectors = table(ids)
loss = vectors.sum()
loss.backward()

print(vectors.detach().tolist())
print(table.weight.grad.tolist())

The expected vectors are [[2,3],[1,0],[2,3]]. The gradient rows are [[1,1],[0,0],[2,2],[0,0]]. Row 2 appears twice, row 0 appears once, and unused rows receive zero.

Engineering checks

Go deeper

Embedding geometry has meaning only relative to the objective. A rotation of every vector, with a later compensation of the weights, can preserve model behavior. Therefore, individual coordinates usually lack a fixed human meaning.

Rare tokens receive fewer direct updates. Frequency-aware sampling, subword sharing, and regularization can help, but each method changes the learned relation. Static embeddings assign one vector for each token. Contextual models create a position-specific vector from the surrounding tokens.

Practice and recap

Question: IDs have shape (32,10), vocabulary size 5,000, and embedding width 64. Name the parameter and output shapes.

Worked answer

The table parameter has shape (5000,64). Lookup appends the feature axis, so the output has shape (32,10,64). The IDs remain categorical addresses. The returned values are floating-point learned features.

An embedding is an indexed parameter table. The objective and data—not the ID values—determine what relations its vectors encode.