02.19 · UNIT 06 · Attention and position · Lesson
Attention scores, masking, and context
Attention gives each visible token a weight, then mixes its value using those weights.
PLAIN-LANGUAGE INTRODUCTION
What is this?
Attention gives each visible token a weight, then mixes its value using those weights.
One simple example
Scores [1,0] become rounded weights [0.731,0.269]. Values [10,2] produce context about 7.848.
What goes in?
One query plus keys and values for visible token positions.
What comes out?
Attention weights and one weighted context vector.
Why does it matter?
Each position can select useful information from its allowed context.
What is it not?
Attention weights are not explanations of every model decision.
WORK THROUGH THE IDEA
See the idea in more detail
- A query asks what to find. Keys describe available positions. Values hold information to mix.
- The query-key comparisons give illustrative scores
[1,0]. - Softmax turns them into weights
[0.731,0.269]. These weights add to1. - Using the rounded weights gives
0.731×10 + 0.269×2 = 7.848. Exact softmax gives about7.84847. - Common mistake: applying a causal mask after softmax can leave weight on future tokens.