Bag of words

Introduction

Turn documents into inspectable sparse count vectors using a fixed training vocabulary.

Learning goal

Construct count and binary bags while preserving vocabulary meaning across splits.

Before you start

Python lists, tokenization, and matrix rows and columns.

Lesson plan

  1. Fix the vocabulary order
  2. Count document tokens
  3. Inspect sparsity and lost order

The problem

A classifier cannot multiply words by weights. Bag of words changes each document into a numeric vector of fixed width. Choose a tokenizer and a vocabulary first. Each vocabulary token owns one coordinate in the vector.

The representation is simple by design. It records presence or counts, but it discards order. Bag of words is a good baseline and an incomplete model of language.

Work through a small example

Use vocabulary [bad, good, movie]. The document “good movie good” has counts [0,2,1]. “bad movie” has [1,0,1]. Vocabulary order sets the meaning of each coordinate, not the integer size.

Words counted into vocabulary coordinatesThree word circles from one document point to three vocabulary columns. Bad receives zero, good receives two, and movie receives one. goodmoviegood badgoodmovie021

A binary bag has the values [0,1,1]. Binary features ask whether a token occurs. Count features preserve repetition. Use validation evidence to choose between them.

The general rule

The count feature for document d and vocabulary token t_j is x_j=count(t_j in d). A corpus of N documents becomes a matrix shaped (N,V). Here V is the vocabulary size.

Fit the vocabulary on training text only. An unseen token is out of vocabulary. It contributes no known coordinate, unless an explicit unknown-token feature exists. Real matrices are sparse. One document uses only a few of the possible words.

Implement and inspect

from collections import Counter

vocabulary = ["bad", "good", "movie"]
documents = ["good movie good", "bad movie"]

rows = []
for document in documents:
    counts = Counter(document.split())
    rows.append([counts[token] for token in vocabulary])

print(rows)
print((len(rows), len(vocabulary)))

The expected output is [[0,2,1],[1,0,1]]. The matrix shape follows as (2,3).

The result acts like a tensor of shape (documents=2, vocabulary=3). Each row must use the same vocabulary order. This snippet uses whitespace tokenization only. Punctuation and case need an explicit policy.

Engineering checks

A common bug is silent. The code builds a sorted vocabulary again at inference. The vector width stays correct, but every learned weight can refer to the wrong word.

Go deeper

Bag of words is a sufficient statistic only under restrictive models. In those models, order adds no useful information. Bag of words cannot distinguish “dog bites person” from “person bites dog.” Bigrams preserve short local order, but they increase dimensionality and sparsity.

Linear classifiers work well with sparse bags. Each weight directly measures how one token shifts a class score. That interpretability does not prove causality. A large weight may capture an artifact in the dataset.

Practice and recap

Question: Use vocabulary [blue, red, sky]. Encode “red sky red” as count and binary bags.

Worked answer

The count bag is [0,2,1]. The binary bag is [0,1,1]. Both have shape (3,). Only the meaning of a repeated token differs.

Bag of words creates sparse vectors that a person can inspect. It is a strong baseline. Its fixed vocabulary and its lost order are design limits, not accidents.