02.18 · UNIT 05 · Text representations and neural language baselines · Lesson
Bigram language models and probabilistic baselines
A bigram model predicts the next token using only the current token.
PLAIN-LANGUAGE INTRODUCTION
The idea
A bigram model predicts the next token using only the current token.
A worked example
In
abab, a→b appears 2 times. b→a appears 1 time.The input
One current token or its ID.
The result
Counts, probabilities, or scores for the next token.
Why you use it
It is a small language-model baseline with visible arithmetic.
A limit to remember
A bigram cannot use tokens earlier than the current one.
Check your understanding
- Read adjacent pairs in
abab:ab,ba, andab. - The count table records
a→b = 2andb→a = 1. Other counts are zero. - For current token
a, the observed next-token probability forbis2/2 = 1. - A learned bigram replaces counts with trainable next-token scores for each current token.
- Common mistake: probability
1describes this tiny text. It does not prove all future text follows that rule.