02.26 · UNIT 08 · Attention and position · Lab
Rotary position embeddings and offsets
RoPE rotates query and key pairs by position-dependent angles.
PLAIN-LANGUAGE INTRODUCTION
The idea
RoPE rotates query and key pairs by position-dependent angles.
A worked example
A toy vector
[1,0] stays [1,0] at position 0. A 90° toy rotation at position 1 gives [0,1].The input
Query and key vectors plus their absolute position numbers.
The result
Rotated vectors whose dot products carry relative-position information.
Why you use it
Attention can use token order without adding a separate position vector.
A limit to remember
Real RoPE does not rotate every pair by 90° per token. This angle is illustrative.
Check your understanding
- Rotary position embedding (RoPE) groups vector coordinates into pairs.
- This illustration starts with pair
[1,0]. Position0uses angle0°. - For clear arithmetic, position
1uses90°. The rotated pair becomes[0,1]. - Real models use several fixed frequencies. Cached decoding must continue positions from the cache length.
- Common mistake: restarting positions at zero during cached decoding changes attention scores.