Cosine Similarity and Vector Search: Math for Semantic Retrieval

Semantic retrieval works in three steps: embed your queries and documents into vectors, compare the vectors, and rank by similarity. The comparison step is usually cosine similarity, and the reason it is the default is that it measures direction rather than magnitude. Two sentences about the same topic point in similar directions in embedding space even when one is a short question and the other is a long answer. That property is exactly what makes vector search useful — and it is simple enough to run in a browser.

Where Embeddings Come From

Sentence embedding models map a piece of text to a fixed-size vector. The all-MiniLM-L6-v2 model used by ReLU.chat, for example, produces 384-dimensional vectors: each sentence becomes a point in a 384-dimensional space where nearby points tend to be semantically related. "What is a Nash equilibrium?" and "Explain the Nash equilibrium concept" land close together; "What is a Nash equilibrium?" and "What is the weather?" land far apart.

These vectors are produced in the browser — ReLU.chat runs the quantized ONNX model through ONNX Runtime and transformers.js — and they are what retrieval, intent classification, and session summarization all operate on. Once you have vectors, the scoring function is the next decision, and cosine similarity is the standard choice.

The Math

Cosine similarity between two vectors a and b is the cosine of the angle between them:

cos(theta) = (a . b) / (||a|| * ||b||)

where the dot product is a . b = sum(a_i * b_i) and the norm is ||a|| = sqrt(sum(a_i^2)). The result ranges from -1 (opposite directions) to 1 (identical direction); for embeddings produced by most sentence models the values are non-negative, so scores fall between 0 and 1.

A quick numeric example makes it concrete. Let a = (1, 2) and b = (2, 1). The dot product is 1*2 + 2*1 = 4, and both norms are sqrt(5), so the cosine is 4/5 = 0.8. The vectors sit at a 36.9-degree angle: similar but not identical. If b were (2, 4), exactly double a, the cosine would be 1.0 even though the vectors have different lengths — that magnitude-invariance is the whole point. In 384 dimensions the intuition is identical, just harder to draw.

Dot Product, Euclidean Distance, and Normalization

The formula hides a practical trick. If every vector is L2-normalized once, so that ||a|| = ||b|| = 1, then cosine similarity reduces to the plain dot product:

function cosine(a, b) {
  let dot = 0, na = 0, nb = 0;
  for (let i = 0; i < a.length; i++) {
    dot += a[i] * b[i];
    na += a[i] * a[i];
    nb += b[i] * b[i];
  }
  return dot / (Math.sqrt(na) * Math.sqrt(nb));
}

Normalize at index time, not at query time: it costs one pass over each stored vector once, and it removes two square roots from every comparison. Euclidean distance is related by ||a - b||^2 = ||a||^2 + ||b||^2 - 2(a . b), so for unit vectors, ranking by cosine is the same as ranking by negative Euclidean distance. On-device, this means you can use whichever formulation your runtime implements most efficiently.

For ReLU.chat's retrieval layer the vectors are 384-dimensional MiniLM embeddings stored as typed arrays. A cosine comparison is a few hundred multiply-adds — negligible next to the cost of computing the embedding itself — which is why brute-force scoring is the practical default in the browser.

Searching Without a Server

Exhaustive search over n candidates is O(n * d): one dot product per candidate. For a few thousand documents at 384 dimensions that is a few million multiply-adds, which a modern phone CPU handles comfortably. Two engineering details make it faster still:

Bounded top-k selection. Keep a fixed-size heap of the best scores instead of sorting the full score array. ReLU.chat uses bounded top-k selection with no full sort, plus LRU caches, to keep retrieval responsive on low-end devices.
Query embedding memoization. Computing the query embedding is the expensive step. ReLU.chat memoizes query embeddings so repeated or rephrased questions skip the model entirely and reuse the cached vector.

Approximate nearest neighbor indexes earn their complexity only at larger scale. HNSW builds a navigable graph and answers queries in logarithmic time but costs extra memory and build time; IVF clusters the corpus and only searches nearby clusters but needs a training pass over the data. For a browser knowledge base measured in thousands of entries, the overhead of maintaining an index is usually worse than the few milliseconds a straight scan takes. The right default is a typed-array scan with a bounded heap.

Ensemble: Cosine Plus BM25

Cosine similarity on its own misses exact terms: "Nash equilibrium" paraphrased as "the concept from game theory" scores well, but a query that literally contains an entry name benefits from lexical matching. ReLU.chat fuses dense cosine scores with field-weighted BM25 at a 70/30 split, so semantic meaning dominates while exact-name matches get a reliable boost. The BM25 side was tuned for the knowledge base — k1=1.5, b=0.75, entry names repeated 3x and aliases 2x during indexing, plus bigram phrase matching — and the two scorers are combined into one rank.

The same cosine machinery also drives intent classification. ReLU.chat compares the query embedding against 19 prototype embeddings per intent category, blends the best and average scores at 70/30, and calibrates the result with a temperature-1.5 softmax. No server, no API call — the entire classifier is vector comparisons in the tab.

Key Takeaway

Cosine similarity measures the angle between embeddings, and with normalized vectors it collapses into a single tight dot-product loop. That makes it the right scoring function for browser-based retrieval: fast, magnitude-invariant, and capable of powering both hybrid ranking and intent classification entirely on-device.

Cosine Similarity and Vector Search: The Math Behind Semantic Retrieval