02.13 · UNIT 03 · Text representations and neural language baselines · Lesson

Bigram language models and probabilistic baselines

A bigram model predicts the next token using only the current token.

PLAIN-LANGUAGE INTRODUCTION

What is this?

A bigram model predicts the next token using only the current token.

One simple example

In abab, a→b appears 2 times. b→a appears 1 time.

One current token or its ID.

Counts, probabilities, or scores for the next token.

It is a small language-model baseline with visible arithmetic.

A bigram cannot use tokens earlier than the current one.

WORK THROUGH THE IDEA

Read adjacent pairs in abab: ab, ba, and ab.
The count table records a→b = 2 and b→a = 1. Other counts are zero.
For current token a, the observed next-token probability for b is 2/2 = 1.
A learned bigram replaces counts with trainable next-token scores for each current token.
Common mistake: probability 1 describes this tiny text. It does not prove all future text follows that rule.