Tensor by Tensor

02.19 · UNIT 05 · Text representations and neural language baselines · Lesson

Fixed-window neural language models

A fixed-window model uses several recent token embeddings for one next-token prediction.

PLAIN-LANGUAGE INTRODUCTION

The idea

A fixed-window model uses several recent token embeddings for one next-token prediction.

A worked example

Three embeddings of width 2 become one vector of length 6. Scores [0.2,1.4,0.1] select token 1.

The input

A fixed number of recent token IDs. Here, the window length is 3.

The result

One score for each possible next token.

Why you use it

A wider context can separate cases that share only the last token.

A limit to remember

A fixed window cannot remember tokens outside that window.

Check your understanding

  1. Look up three token embeddings. Each embedding has width 2.
  2. Join them in order. The model receives 3×2 = 6 numbers.
  3. An MLP turns those numbers into illustrative scores [0.2,1.4,0.1].
  4. The largest score is at index 1. Softmax would turn all three scores into probabilities.
  5. Common mistake: shuffling the three embeddings removes their position meaning.
Open the detailed notes ↗