Tensor by Tensor

02.25 · UNIT 08 · Attention and position · Lab

Lab: implement multi-head attention

Multi-head attention runs several attention mixes with separate learned projections.

PLAIN-LANGUAGE INTRODUCTION

The idea

Multi-head attention runs several attention mixes with separate learned projections.

A worked example

Head 1 uses weights [0.7,0.2,0.1]. Head 2 uses [0.1,0.3,0.6]. Each row sums to 1.

The input

A token tensor plus one set of query, key, and value projections per head.

The result

One context per head, joined back into the model width.

Why you use it

Different heads can learn different weighted relationships.

A limit to remember

More heads do not automatically improve a model.

Check your understanding

  1. Split the model width across heads. Each head makes its own query, key, and value tensors.
  2. For three visible tokens, head 1 has illustrative weights [0.7,0.2,0.1].
  3. Head 2 has [0.1,0.3,0.6]. Each row sums to 1 after softmax.
  4. Each head makes a weighted value sum. Join the results, then project to the original model width.
  5. Common mistake: model width must divide cleanly by the number of equal-width heads.
Open the detailed notes ↗