02.29 · UNIT 08 · Train, evaluate, and serve your transformer · Lab

KV caches, incremental attention, and correctness

A KV cache stores earlier attention keys and values during generation.

PLAIN-LANGUAGE INTRODUCTION

What is this?

A KV cache stores earlier attention keys and values during generation.

One simple example

Prefill 3 tokens. Decode 1 new token. Query length is 1, while key and value lengths become 4.

A new token, one cache per layer, and the next position number.

Next-token scores and caches extended by the new token.

Decoding avoids recomputing unchanged keys and values for the prefix.

A cache does not remove attention over earlier context or its memory cost.

WORK THROUGH THE IDEA

KV means keys and values. Every attention layer stores its own key tensor and value tensor.
Prefill processes the three-token prompt. Both cached sequence lengths become 3.
Decode processes one new query. Its query length is 1. Append one key and one value.
The new cache length is 4. The new position must continue from the earlier length.
Common mistake: passing the full prompt again with its cache duplicates prompt entries.