Tensor by Tensor

02.26 · UNIT 08 · Attention and position · Lab

Rotary position embeddings and offsets

RoPE rotates query and key pairs by position-dependent angles.

PLAIN-LANGUAGE INTRODUCTION

The idea

RoPE rotates query and key pairs by position-dependent angles.

A worked example

A toy vector [1,0] stays [1,0] at position 0. A 90° toy rotation at position 1 gives [0,1].

The input

Query and key vectors plus their absolute position numbers.

The result

Rotated vectors whose dot products carry relative-position information.

Why you use it

Attention can use token order without adding a separate position vector.

A limit to remember

Real RoPE does not rotate every pair by 90° per token. This angle is illustrative.

Check your understanding

  1. Rotary position embedding (RoPE) groups vector coordinates into pairs.
  2. This illustration starts with pair [1,0]. Position 0 uses angle 0°.
  3. For clear arithmetic, position 1 uses 90°. The rotated pair becomes [0,1].
  4. Real models use several fixed frequencies. Cached decoding must continue positions from the cache length.
  5. Common mistake: restarting positions at zero during cached decoding changes attention scores.
Open the detailed notes ↗