Cosine similarity and text retrieval

Introduction

Rank text vectors by direction while controlling for vector length.

Learning goal

Calculate cosine similarity and build a safe nearest-document ranking.

Before you start

Vectors, dot products, L2 norms, and TF-IDF features.

Lesson plan

  1. Normalize nonzero vectors
  2. Compute query-document scores
  3. Audit ranking and zero-vector policy

The problem

Text retrieval needs a score between a query vector and each document vector. A dot product rewards shared coordinates. Its magnitude also grows with vector length. Long documents can rank high only because they contain more weighted terms.

Cosine similarity compares direction. It divides the dot product by both vector lengths. The result is a score from -1 to 1 for general real vectors. Nonnegative TF-IDF vectors produce scores from 0 to 1.

Work through a small example

Let query q=[1,1]. Document a=[2,2] points in exactly the same direction, so cosine is 1. Document b=[2,0] shares one feature. Cosine is 2/(sqrt(2)*2)=0.707.

Vector angle used for cosine similarityA query vector and document A point in the same direction. Document B lies along the horizontal axis. It makes a forty-five degree angle with the query. qa: same directionb45°

A scale change does not change direction: cos(q,100q)=1. This property is useful when repetition or document length must matter less than the feature pattern.

The general rule

cos(q,d)=(q dot d)/(||q||_2 ||d||_2). The L2 norm is the square root of the summed squared coordinates. If either norm is zero, the expression is undefined. Retrieval code should return no evidence or a documented zero. It must not divide silently.

For a query matrix shaped (Q,D) and normalized document matrix shaped (N,D), matrix multiplication Q @ documents.T returns all scores with shape (Q,N).

Implement and inspect

import numpy as np

query = np.array([1.0, 1.0])
documents = np.array([[2.0, 2.0], [2.0, 0.0]])

query_norm = np.linalg.norm(query)
document_norms = np.linalg.norm(documents, axis=1, keepdims=True)
if query_norm == 0 or np.any(document_norms == 0):
    raise ValueError("cosine similarity needs nonzero vectors")
query = query / query_norm
documents = documents / document_norms
scores = documents @ query
print(np.round(scores, 3).tolist())
print(np.argsort(-scores).tolist())

The expected output is the similarities [1.0,0.707] and the rank order [0,1].

documents has shape (2,2). query has shape (2,). The scores have shape (2,). Negate the scores before the ascending argsort to get the highest similarity first.

Engineering checks

For sparse TF-IDF, compute dot products only on shared nonzero coordinates. Dense conversion can exhaust memory, but the mathematics does not change.

Go deeper

Cosine ignores norm by design. That can remove nuisance document length. However, vector norm may carry confidence or frequency information in learned representations. Compare cosine with the dot product on the downstream metric.

Nearest-neighbour quality also depends on the representation. Cosine cannot recover the word order that bag of words discards. In high dimensions, similarity distributions can concentrate. Approximate indexes trade exact ranking for speed.

Practice and recap

Question: What is cosine between [1,0] and [-3,0]?

Worked answer

The dot product is -3. The norms are 1 and 3. Cosine is -3/(1*3)=-1. The vectors point in opposite directions, even though one has three times the length.

Cosine ranks directions rather than magnitudes. Normalize the vectors in the same way. Define the zero-vector behavior. Evaluate the full representation and retrieval pipeline.