Bag of words

The problem

A classifier cannot multiply words by weights. Bag of words turns each document into a fixed-width numeric vector. First choose a tokenizer and a vocabulary. Each vocabulary token owns one vector coordinate.

The representation is deliberately simple: it records presence or counts, but discards order. This makes it an excellent baseline and an incomplete model of language.

Work through a small example

Use vocabulary [bad, good, movie]. The document “good movie good” has counts [0,2,1]. “bad movie” has [1,0,1]. The coordinate meaning comes from vocabulary order, not the integer size.

A binary bag would be [0,1,1]. Binary features ask whether a token occurs; count features preserve repetition. Validation evidence should choose between them.

The general rule

For document d and vocabulary token t_j, count feature x_j=count(t_j in d). A corpus of N documents becomes a matrix shaped (N,V), where V is vocabulary size.

Fit the vocabulary on training text only. An unseen token is out of vocabulary and contributes no known coordinate unless an explicit unknown-token feature exists. Real matrices are sparse because one document uses few of the possible words.

Implement and inspect

from collections import Counter

vocabulary = ["bad", "good", "movie"]
documents = ["good movie good", "bad movie"]

rows = []
for document in documents:
    counts = Counter(document.split())
    rows.append([counts[token] for token in vocabulary])

print(rows)
print((len(rows), len(vocabulary)))

Expected output is [[0,2,1],[1,0,1]], followed by matrix shape (2,3).

The result acts like a tensor of shape (documents=2, vocabulary=3). Each row must use the same vocabulary order. This snippet uses whitespace tokenization only; punctuation and case need an explicit policy.

Engineering checks

• Save the tokenizer settings and vocabulary with the model.
• Assert training and inference vectors have width V.
• Measure the fraction of inference tokens that are out of vocabulary.
• Use sparse storage when V is large.

A common silent bug is rebuilding a sorted vocabulary independently at inference. The vector width stays correct while every learned weight can refer to the wrong word.

Go deeper

Bag of words is a sufficient statistic only under restrictive models where order adds no useful information. It cannot distinguish “dog bites person” from “person bites dog.” Adding bigrams preserves short local order but increases dimensionality and sparsity.

Linear classifiers work well with sparse bags because each weight directly measures how one token shifts a class score. That interpretability does not establish causality: a large weight may capture a dataset artifact.

Practice and recap

Question: With vocabulary [blue, red, sky], encode “red sky red” as count and binary bags.

Bag of words creates inspectable, sparse vectors and a strong baseline. Its fixed vocabulary and lost order are design limits, not implementation accidents.