02.21 · UNIT 06 · Attention and position · Lab

Rotary position embeddings and offsets

RoPE rotates query and key pairs by position-dependent angles.

PLAIN-LANGUAGE INTRODUCTION

What is this?

RoPE rotates query and key pairs by position-dependent angles.

One simple example

A toy vector [1,0] stays [1,0] at position 0. A 90° toy rotation at position 1 gives [0,1].

Query and key vectors plus their absolute position numbers.

Rotated vectors whose dot products carry relative-position information.

Attention can use token order without adding a separate position vector.

Real RoPE does not rotate every pair by 90° per token. This angle is illustrative.

WORK THROUGH THE IDEA

Rotary position embedding (RoPE) groups vector coordinates into pairs.
This illustration starts with pair [1,0]. Position 0 uses angle 0°.
For clear arithmetic, position 1 uses 90°. The rotated pair becomes [0,1].
Real models use several fixed frequencies. Cached decoding must continue positions from the cache length.
Common mistake: restarting positions at zero during cached decoding changes attention scores.