Tensor by Tensor

02.18 · UNIT 05 · Text representations and neural language baselines · Lesson

Bigram language models and probabilistic baselines

A bigram model predicts the next token using only the current token.

PLAIN-LANGUAGE INTRODUCTION

The idea

A bigram model predicts the next token using only the current token.

A worked example

In abab, a→b appears 2 times. b→a appears 1 time.

The input

One current token or its ID.

The result

Counts, probabilities, or scores for the next token.

Why you use it

It is a small language-model baseline with visible arithmetic.

A limit to remember

A bigram cannot use tokens earlier than the current one.

Check your understanding

  1. Read adjacent pairs in abab: ab, ba, and ab.
  2. The count table records a→b = 2 and b→a = 1. Other counts are zero.
  3. For current token a, the observed next-token probability for b is 2/2 = 1.
  4. A learned bigram replaces counts with trainable next-token scores for each current token.
  5. Common mistake: probability 1 describes this tiny text. It does not prove all future text follows that rule.
Open the detailed notes ↗