02.19 · UNIT 05 · Text representations and neural language baselines · Lesson
Fixed-window neural language models
A fixed-window model uses several recent token embeddings for one next-token prediction.
PLAIN-LANGUAGE INTRODUCTION
The idea
A fixed-window model uses several recent token embeddings for one next-token prediction.
A worked example
Three embeddings of width
2 become one vector of length 6. Scores [0.2,1.4,0.1] select token 1.The input
A fixed number of recent token IDs. Here, the window length is 3.
The result
One score for each possible next token.
Why you use it
A wider context can separate cases that share only the last token.
A limit to remember
A fixed window cannot remember tokens outside that window.
Check your understanding
- Look up three token embeddings. Each embedding has width
2. - Join them in order. The model receives
3×2 = 6numbers. - An MLP turns those numbers into illustrative scores
[0.2,1.4,0.1]. - The largest score is at index
1. Softmax would turn all three scores into probabilities. - Common mistake: shuffling the three embeddings removes their position meaning.