NLP foundations

Bag of words, TF-IDF, and classical text baselines

Introduction

For the text 'cat cat sat,' a count bag records two cats while a binary bag records only presence. This small difference starts a complete classical text pipeline from tokenization to retrieval and classification.

Learning goal

Build bag-of-words and TF-IDF text features, compare documents, and evaluate a leakage-safe classical prediction baseline.

Before you start

Python strings, lists, dictionaries, loops, logarithms, vectors, and train-test separation.

Lesson plan

  1. Create a training-only vocabulary and calculate count and binary bag features.
  2. Derive smoothed TF-IDF values and compare sparse vectors with cosine similarity.
  3. Use nearest-centroid classification, run the program, and inspect out-of-vocabulary limits.

Lesson map: five moves

  1. Tokenize training text and build its vocabulary.
  2. Turn each document into count or binary features.
  3. Weight rare terms with one precise TF-IDF formula.
  4. Compare sparse vectors with cosine similarity.
  5. Test retrieval and nearest-centroid classification on held-out text.

Plain-language start

Idea. Suppose students put words into labelled boxes. One box is cat, another is dog, and so on.

Action. For each document, count how many times each word appears. That list of counts is a bag of words.

Limit. Word order is discarded. “dog bites person” and “person bites dog” have the same counts even though their meanings differ.

This simple representation is still useful. It is fast, easy to debug, and often strong when labels are connected to distinctive words. A support-ticket classifier may learn that refund points toward billing while password points toward account access. A baseline like this tells you whether a larger model adds real value.

Step 1: decide the tokenizer and vocabulary

Definition. A tokenizer is a rule that divides text into tokens.

Example. The program changes "Cats, CATS!" into ["cats", "cats"]. It lowercases text, removes punctuation, and keeps repetition.

Design choice. Keeping apostrophes distinguishes don't from don plus t. Removing punctuation loses signals such as question marks. Lowercasing can erase the difference between US and us.

Vocabulary. Each known training token receives one column. A new validation or test token is out of vocabulary, shortened to OOV, and this baseline ignores it.

Leakage rule. Never add validation or test words to the vocabulary. That would choose the feature space after looking at future data.

training texts -> tokenizer -> training vocabulary
validation text -> same tokenizer -> training columns only
test text       -> same tokenizer -> training columns only

Step 2: count bag-of-words values

Predict one cell: if the vocabulary column is cat and a document contains “cat sat cat,” the count value is 2. A binary bag uses 1 because it records presence, not frequency.

Input. Use training vocabulary [ate, cat, dog, sat] and document cat cat sat.

Count result.

[ate=0, cat=2, dog=0, sat=1]

Binary result. A binary bag records presence only:

[ate=0, cat=1, dog=0, sat=1]

Sparse storage. A short document uses only a few columns, so store non-zero pairs such as {cat: 2, sat: 1}. This avoids a long row full of zeros.

Step 3: calculate TF-IDF on three documents

A zero can mean two different things: the word is absent, or the word was never admitted to the vocabulary. Keep the training vocabulary beside any vector you inspect. A vector without its vocabulary has no readable feature meaning.

Problem. Bag of words can give large values to words that appear in almost every document.

Definition. TF-IDF keeps term frequency, or TF, but reduces the weight of terms found in many documents. Here TF is raw count.

idf(t) = log((1 + N) / (1 + df(t))) + 1
tfidf(t, d) = count(t in d) * idf(t)

Important. Libraries use different formulas. Record the exact formula instead of saying only “TF-IDF”.

Hand calculation input.

d1 = "cat sat"
d2 = "cat ate"
d3 = "dog sat"
N = 3
  1. cat and sat each appear in two documents: log(4 / 3) + 1 = 1.287682.
  2. ate and dog each appear in one document: log(4 / 2) + 1 = 1.693147.
  3. Multiply each count in d1 by its term’s IDF.

Result for d1.

[ate=0, cat=1.287682, dog=0, sat=1.287682]

Notice that dog receives the larger possible weight because it is rarer in this training collection. This does not mean it is universally more meaningful. IDF describes this fitted corpus only. A corpus change requires fitting new IDF values.

Step 4: compare direction with cosine similarity

Problem. A dot product rewards matching weights, but it can favour long documents.

Solution. Cosine similarity divides by both vector lengths:

cosine(q, d) = dot(q, d) / (length(q) * length(d))

Example. Query cat dog becomes [0, 1.287682, 1.693147, 0].

Sparse shortcut. Compute the dot product only on shared coordinates. Return zero if either vector has zero length.

From retrieval to nearest-centroid classification

Definition. A centroid is the arithmetic mean of a group of vectors.

  1. Compute TF-IDF vectors for training documents.
  2. Average vectors separately for each label.
  3. Vectorize a new document with the frozen vocabulary and IDF.
  4. Choose the centroid with highest cosine similarity.

Dataset. The example has space, cooking, and music topics: 15 training documents, six validation documents, and six test documents.

Limit. The hand-written documents contain clear topic words. A perfect score checks this implementation; it does not establish real-world quality.

Run the complete standard-library program

Download. build_text_baselines.py imports only Python’s standard library.

Included. Tokenization, count and binary bags, smoothed TF-IDF, sparse cosine retrieval, nearest-centroid classification, fixed splits, and assertions.

python examples/pytorch/build_text_baselines.py
python examples/pytorch/build_text_baselines.py --binary-demo

Verified result. The first command was tested in this project environment. Its main output was:

training documents: 15
training vocabulary size: 80
majority-class test accuracy: 0.333
validation accuracy: 1.000
held-out test accuracy: 1.000
retrieval for 'rocket moon mission':
  0.244 [space] astronaut trains for space station mission
  0.236 [space] rocket launch reaches orbit around earth
  0.236 [space] spacecraft carries science instruments to moon

The test documents are separate strings, but all text was authored around the same obvious topic rules. This is an implementation check, not evidence for deployment. A real evaluation should include naturally occurring text, class imbalance, ambiguous cases, spelling variation, domain shift, and an untouched final test set.

Mistakes to avoid

  • Do not fit vocabulary, IDF, normalization choices, or thresholds on test data.
  • Do not remove words automatically. A word such as not may be essential for sentiment.
  • Do not call token IDs ordered measurements. Column 20 is not “larger” than column 3.
  • Do not compare accuracy alone when classes are imbalanced.
  • Do not claim that a separately written toy sentence proves generalization to real users.

Solved practice

1. Why is vocabulary fitting on all documents leakage?

The test set is meant to represent unknown future data. If its words influence the columns, the model-building process has already inspected that future data. Fit on training text, then ignore or explicitly handle unseen test words.

2. A term occurs five times in one document and nowhere else. What are TF and document frequency?

With raw-count TF, its TF in that document is 5. Its document frequency is 1, because document frequency counts documents containing the term, not total occurrences.

3. Why can two opposite sentences have the same bag?

Bag of words discards order. “dog bites person” and “person bites dog” contain the same three tokens. Adding n-grams can preserve short local order, while contextual encoders can represent broader context.

4. What should happen for a query made only of OOV words?

Its vector has no non-zero coordinates. Retrieval similarities are all zero, and classification should report that it lacks known evidence rather than silently return a confident label. The downloadable classifier raises a clear error for this case.

Recap

Sources: the Stanford Introduction to Information Retrieval term-frequency chapter, its inverse-document-frequency chapter, and its cosine and dot-product explanation.