Tensor by Tensor

02.17 · UNIT 05 · Text representations and neural language baselines · Lab

Lab: train and evaluate embeddings

An embedding gives each token a short learned vector.

PLAIN-LANGUAGE INTRODUCTION

The idea

An embedding gives each token a short learned vector.

A worked example

Use red→[1,0] and blue→[0,1]. Looking up [red,blue] returns a tensor with shape (2,2).

The input

Token IDs. Here, the sequence contains red then blue.

The result

One vector per token position.

Why you use it

Learned vectors let nearby numeric directions represent useful token patterns.

A limit to remember

Embedding coordinates do not have fixed human meanings by default.

Check your understanding

  1. An embedding table has one row per vocabulary item. Each row is trainable.
  2. This illustration assigns red→[1,0] and blue→[0,1].
  3. Looking up two IDs returns two rows. The result has shape (sequence=2, width=2).
  4. Training changes the selected rows when their gradients are nonzero. Evaluate them on a real task.
  5. Common mistake: a visually close pair of vectors does not prove equal meaning in every task.
Open the detailed notes ↗