Tensor by Tensor

02.24 · UNIT 08 · Attention and position · Lesson

Attention scores, masking, and context

Attention gives each visible token a weight, then mixes its value using those weights.

PLAIN-LANGUAGE INTRODUCTION

The idea

Attention gives each visible token a weight, then mixes its value using those weights.

A worked example

Scores [1,0] become rounded weights [0.731,0.269]. Values [10,2] produce context about 7.848.

The input

One query plus keys and values for visible token positions.

The result

Attention weights and one weighted context vector.

Why you use it

Each position can select useful information from its allowed context.

A limit to remember

Attention weights are not explanations of every model decision.

Check your understanding

  1. A query asks what to find. Keys describe available positions. Values hold information to mix.
  2. The query-key comparisons give illustrative scores [1,0].
  3. Softmax turns them into weights [0.731,0.269]. These weights add to 1.
  4. Using the rounded weights gives 0.731×10 + 0.269×2 = 7.848. Exact softmax gives about 7.84847.
  5. Common mistake: applying a causal mask after softmax can leave weight on future tokens.
Open the detailed notes ↗