TF-IDF

The problem

Raw counts can let common corpus words dominate a text vector. TF-IDF combines term frequency (TF), evidence inside one document, with inverse document frequency (IDF), a downweighting based on how widespread a token is across training documents.

TF-IDF does not discover meaning. It changes feature scale so a token present almost everywhere contributes less than a token concentrated in fewer documents.

Work through a small example

Use three documents: “cat sat,” “cat ate,” and “dog sat.” The token cat appears in two documents, so df(cat)=2. The token dog appears in one, so df(dog)=1.

With smoothed idf(t)=log((1+N)/(1+df(t)))+1, idf(cat)=1.288 and idf(dog)=1.693. Dog receives more weight in this corpus because it is rarer.

The general rule

For raw-count TF, tfidf(t,d)=count(t in d)*idf(t). Other systems may use binary or logarithmic TF, different smoothing, sublinear scaling, and vector normalization. State the exact convention because “TF-IDF” does not identify one unique formula.

IDF is fitted state. Compute document frequencies on training documents only, freeze them, then transform validation and test documents. Otherwise future text changes the features used to train the model.

Implement and inspect

import math
from collections import Counter

docs = ["cat sat", "cat ate", "dog sat"]
tokens = [doc.split() for doc in docs]
vocabulary = sorted(set().union(*map(set, tokens)))
idf = {
    term: math.log((1 + len(docs)) /
                   (1 + sum(term in doc for doc in tokens))) + 1
    for term in vocabulary
}
row = [Counter(tokens[0])[term] * idf[term] for term in vocabulary]
print(vocabulary)
print([round(value, 3) for value in row])

Expected output is vocabulary ['ate','cat','dog','sat'] and first-document row [0.0,1.288,0.0,1.288].

The output row has shape (V=4,). Stacking all documents gives (N=3,V=4). The zero for dog means it is known but absent from the first document.

Engineering checks

• Persist vocabulary and IDF values together.
• Test repeated occurrences: TF changes, but document frequency increases only once per document.
• Handle an all-out-of-vocabulary (OOV) document, whose words are absent from the fitted vocabulary and whose vector can have zero length.
• Do not refit IDF separately on each batch.

Go deeper

IDF resembles an information signal: a rare event is more surprising than a common one. But rarity is not relevance. Misspellings, identifiers, or private data can receive high weight while being useless or harmful.

Very long documents can still have larger norms. L2 normalization separates direction from length and is common before cosine similarity. For classification, normalization strength and TF convention are hyperparameters chosen on validation data.

Practice and recap

Question: In four training documents, a term appears in all four. What is its smoothed IDF?

TF-IDF scales local counts by training-corpus rarity. Freeze the fitted statistics and record the precise formula.