02.34 · UNIT 10 · Train, evaluate, and serve your transformer · Lab
KV caches, incremental attention, and correctness
A KV cache stores earlier attention keys and values during generation.
PLAIN-LANGUAGE INTRODUCTION
The idea
A KV cache stores earlier attention keys and values during generation.
A worked example
Prefill
3 tokens. Decode 1 new token. Query length is 1, while key and value lengths become 4.The input
A new token, one cache per layer, and the next position number.
The result
Next-token scores and caches extended by the new token.
Why you use it
Decoding avoids recomputing unchanged keys and values for the prefix.
A limit to remember
A cache does not remove attention over earlier context or its memory cost.
Check your understanding
- KV means keys and values. Every attention layer stores its own key tensor and value tensor.
- Prefill processes the three-token prompt. Both cached sequence lengths become
3. - Decode processes one new query. Its query length is
1. Append one key and one value. - The new cache length is
4. The new position must continue from the earlier length. - Common mistake: passing the full prompt again with its cache duplicates prompt entries.