02.14 · UNIT 03 · Text representations and neural language baselines · Lesson

Fixed-window neural language models

A fixed-window model uses several recent token embeddings for one next-token prediction.

PLAIN-LANGUAGE INTRODUCTION

What is this?

A fixed-window model uses several recent token embeddings for one next-token prediction.

One simple example

Three embeddings of width 2 become one vector of length 6. Scores [0.2,1.4,0.1] select token 1.

A fixed number of recent token IDs. Here, the window length is 3.

One score for each possible next token.

A wider context can separate cases that share only the last token.

A fixed window cannot remember tokens outside that window.

WORK THROUGH THE IDEA

Look up three token embeddings. Each embedding has width 2.
Join them in order. The model receives 3×2 = 6 numbers.
An MLP turns those numbers into illustrative scores [0.2,1.4,0.1].
The largest score is at index 1. Softmax would turn all three scores into probabilities.
Common mistake: shuffling the three embeddings removes their position meaning.