02.25 · UNIT 08 · Attention and position · Lab
Lab: implement multi-head attention
Multi-head attention runs several attention mixes with separate learned projections.
PLAIN-LANGUAGE INTRODUCTION
The idea
Multi-head attention runs several attention mixes with separate learned projections.
A worked example
Head 1 uses weights
[0.7,0.2,0.1]. Head 2 uses [0.1,0.3,0.6]. Each row sums to 1.The input
A token tensor plus one set of query, key, and value projections per head.
The result
One context per head, joined back into the model width.
Why you use it
Different heads can learn different weighted relationships.
A limit to remember
More heads do not automatically improve a model.
Check your understanding
- Split the model width across heads. Each head makes its own query, key, and value tensors.
- For three visible tokens, head 1 has illustrative weights
[0.7,0.2,0.1]. - Head 2 has
[0.1,0.3,0.6]. Each row sums to1after softmax. - Each head makes a weighted value sum. Join the results, then project to the original model width.
- Common mistake: model width must divide cleanly by the number of equal-width heads.