02.24 · UNIT 08 · Attention and position · Lesson
Attention scores, masking, and context
Attention gives each visible token a weight, then mixes its value using those weights.
PLAIN-LANGUAGE INTRODUCTION
The idea
Attention gives each visible token a weight, then mixes its value using those weights.
A worked example
Scores
[1,0] become rounded weights [0.731,0.269]. Values [10,2] produce context about 7.848.The input
One query plus keys and values for visible token positions.
The result
Attention weights and one weighted context vector.
Why you use it
Each position can select useful information from its allowed context.
A limit to remember
Attention weights are not explanations of every model decision.
Check your understanding
- A query asks what to find. Keys describe available positions. Values hold information to mix.
- The query-key comparisons give illustrative scores
[1,0]. - Softmax turns them into weights
[0.731,0.269]. These weights add to1. - Using the rounded weights gives
0.731×10 + 0.269×2 = 7.848. Exact softmax gives about7.84847. - Common mistake: applying a causal mask after softmax can leave weight on future tokens.