Cosine similarity and text retrieval
Introduction
Rank text vectors by direction while controlling for vector length.
- Learning goal
Calculate cosine similarity and build a safe nearest-document ranking.
- Before you start
Vectors, dot products, L2 norms, and TF-IDF features.
Lesson plan
- Normalize nonzero vectors
- Compute query-document scores
- Audit ranking and zero-vector policy
The problem
Text retrieval needs a score between a query vector and each document vector. A dot product rewards shared coordinates. Its magnitude also grows with vector length. Long documents can rank high only because they contain more weighted terms.
Cosine similarity compares direction. It divides the dot product by both vector lengths. The result is a score from -1 to 1 for general real vectors. Nonnegative TF-IDF vectors produce scores from 0 to 1.
Work through a small example
Let query q=[1,1]. Document a=[2,2] points in exactly the same direction, so cosine is 1. Document b=[2,0] shares one feature. Cosine is 2/(sqrt(2)*2)=0.707.
A scale change does not change direction: cos(q,100q)=1. This property is useful when repetition or document length must matter less than the feature pattern.
The general rule
cos(q,d)=(q dot d)/(||q||_2 ||d||_2). The L2 norm is the square root of the summed squared coordinates. If either norm is zero, the expression is undefined. Retrieval code should return no evidence or a documented zero. It must not divide silently.
For a query matrix shaped (Q,D) and normalized document matrix shaped (N,D), matrix multiplication Q @ documents.T returns all scores with shape (Q,N).
Implement and inspect
import numpy as np
query = np.array([1.0, 1.0])
documents = np.array([[2.0, 2.0], [2.0, 0.0]])
query_norm = np.linalg.norm(query)
document_norms = np.linalg.norm(documents, axis=1, keepdims=True)
if query_norm == 0 or np.any(document_norms == 0):
raise ValueError("cosine similarity needs nonzero vectors")
query = query / query_norm
documents = documents / document_norms
scores = documents @ query
print(np.round(scores, 3).tolist())
print(np.argsort(-scores).tolist())
The expected output is the similarities [1.0,0.707] and the rank order [0,1].
documents has shape (2,2). query has shape (2,). The scores have shape (2,). Negate the scores before the ascending argsort to get the highest similarity first.
Engineering checks
- Normalize along the feature axis, not across documents.
- Guard zero vectors from empty text or all-out-of-vocabulary (OOV) text. OOV words are absent from the fitted vocabulary.
- Exclude the query document from the nearest-neighbour evaluation.
- Use a stable tie rule so equal scores are reproducible.
For sparse TF-IDF, compute dot products only on shared nonzero coordinates. Dense conversion can exhaust memory, but the mathematics does not change.
Go deeper
Cosine ignores norm by design. That can remove nuisance document length. However, vector norm may carry confidence or frequency information in learned representations. Compare cosine with the dot product on the downstream metric.
Nearest-neighbour quality also depends on the representation. Cosine cannot recover the word order that bag of words discards. In high dimensions, similarity distributions can concentrate. Approximate indexes trade exact ranking for speed.
Practice and recap
Question: What is cosine between [1,0] and [-3,0]?
Worked answer
The dot product is -3. The norms are 1 and 3. Cosine is -3/(1*3)=-1. The vectors point in opposite directions, even though one has three times the length.
Cosine ranks directions rather than magnitudes. Normalize the vectors in the same way. Define the zero-vector behavior. Evaluate the full representation and retrieval pipeline.