02.12 · UNIT 03 · Text representations and neural language baselines · Lab
Lab: train and evaluate embeddings
An embedding gives each token a short learned vector.
PLAIN-LANGUAGE INTRODUCTION
What is this?
An embedding gives each token a short learned vector.
One simple example
Use red→[1,0] and blue→[0,1]. Looking up [red,blue] returns a tensor with shape (2,2).
What goes in?
Token IDs. Here, the sequence contains red then blue.
What comes out?
One vector per token position.
Why does it matter?
Learned vectors let nearby numeric directions represent useful token patterns.
What is it not?
Embedding coordinates do not have fixed human meanings by default.
WORK THROUGH THE IDEA
See the idea in more detail
- An embedding table has one row per vocabulary item. Each row is trainable.
- This illustration assigns
red→[1,0]andblue→[0,1]. - Looking up two IDs returns two rows. The result has shape
(sequence=2, width=2). - Training changes the selected rows when their gradients are nonzero. Evaluate them on a real task.
- Common mistake: a visually close pair of vectors does not prove equal meaning in every task.