Tensor by Tensor

02.34 · UNIT 10 · Train, evaluate, and serve your transformer · Lab

KV caches, incremental attention, and correctness

A KV cache stores earlier attention keys and values during generation.

PLAIN-LANGUAGE INTRODUCTION

The idea

A KV cache stores earlier attention keys and values during generation.

A worked example

Prefill 3 tokens. Decode 1 new token. Query length is 1, while key and value lengths become 4.

The input

A new token, one cache per layer, and the next position number.

The result

Next-token scores and caches extended by the new token.

Why you use it

Decoding avoids recomputing unchanged keys and values for the prefix.

A limit to remember

A cache does not remove attention over earlier context or its memory cost.

Check your understanding

  1. KV means keys and values. Every attention layer stores its own key tensor and value tensor.
  2. Prefill processes the three-token prompt. Both cached sequence lengths become 3.
  3. Decode processes one new query. Its query length is 1. Append one key and one value.
  4. The new cache length is 4. The new position must continue from the earlier length.
  5. Common mistake: passing the full prompt again with its cache duplicates prompt entries.
Open the detailed notes ↗