Cosine similarity is the friendliest operation in machine learning: a dot product divided by two norms. It is also the operation a retrieval system performs thousands of times per query. In a browser, where every millisecond competes with rendering and user input, the naive implementation is rarely the right one. This article breaks down the actual cost of cosine similarity and the four optimizations that matter: precomputed norms, typed arrays, bounded top-k selection, and query memoization.

The Baseline Cost Model

For two vectors a and b, cosine similarity is:

cos(a, b) = (a . b) / (||a|| * ||b||)

The dot product is sum(a_i * b_i) over all dimensions, and each norm is sqrt(sum(a_i^2)). With 384-dimensional embeddings — the dimensionality of all-MiniLM-L6-v2, which ReLU.chat runs in-browser — one comparison is a 384-element dot product plus two norm passes: roughly 2,300 floating-point operations in the naive implementation. A query against 1,000 fragments is about 2.3 million operations; against 10,000 fragments, about 23 million. The naive implementation also computes both norms inside the comparison loop, so every comparison repeats work that could have been done once.

Precompute the Norms

The single cheapest optimization is to stop computing norms at query time. Normalize every stored vector once when the index is built, so that ||b|| = 1 for all fragments. Then cosine similarity reduces to a pure dot product:

// At index time: store v / norm(v), never the raw vector.
function normalize(v) {
  let sum = 0;
  for (let i = 0; i < v.length; i++) sum += v[i] * v[i];
  const norm = Math.sqrt(sum);
  for (let i = 0; i < v.length; i++) v[i] /= norm;
  return v;
}

// At query time: one norm for the query, pure dot products after.
function score(query, normalizedFragment) {
  let dot = 0;
  for (let i = 0; i < query.length; i++) dot += query[i] * normalizedFragment[i];
  return dot;
}

This removes two passes and two square roots from every fragment comparison — for a 10,000-fragment query, that is 20,000 fewer norm passes and 20,000 fewer square roots, plus a one-time pass to normalize the query. It also changes the semantics slightly — scores become dot products in the unit sphere rather than true cosine values — but since normalization is monotonic per vector, ranking is unaffected.

Typed Arrays and Loop Structure

Where the vectors live matters as much as the math. Plain JavaScript arrays of numbers can hold doubles, carry prototype overhead, and force the engine to guess element types. Float32Array is the right container for embedding vectors: embeddings are float32 by default, and 384 float32 values occupy 1,536 bytes — half the 3,072 bytes the same array would use as float64. Smaller memory means better cache behavior, and cache misses are the hidden cost in vector math.

Loop structure matters too. V8 and other engines optimize tight loops best when the loop is simple and the array access is predictable:

function topScore(query, fragments, n) {
  const q = query; // hoist the query reference
  let best = -Infinity;
  for (let f = 0; f < n; f++) {
    const frag = fragments[f];      // one load per fragment
    let dot = 0;
    for (let i = 0; i < q.length; i++) {
      dot += q[i] * frag[i];        // no function calls, no allocations
    }
    if (dot > best) best = dot;
  }
  return best;
}

Avoid function calls and allocations inside the inner loop. A helper that computes a single dot product is fine for readability; a closure that allocates per call is a measurable tax. If you need the top-k fragments rather than the single best, keep the running top-k buffer outside the loop and update it with the same inlined dot product.

Bounded Top-k, Memoization, and Measurement

The most common retrieval mistake is scoring everything, then sorting. A full sort is O(N log N); selecting the top-k is O(N log k) with a min-heap, or simply O(N * k) with a small insertion buffer when k is tiny. For retrieval, k is usually 5 to 20, so the heap or insertion buffer wins by an order of magnitude. ReLU.chat uses bounded top-k selection rather than a full sort: keep a min-heap of the best k scores seen so far, and for each new fragment, compare against the heap root, evicting the smallest when a better score arrives. The heap never exceeds k entries, so memory stays flat and the total work stays linear in the number of fragments.

The same principle applies to sparse scores. BM25 scores can be computed incrementally over inverted-index postings — you only touch fragments that share a term with the query — and the top-k can be maintained without ever materializing a full score array. For hybrid retrieval, fuse the dense and sparse top-k lists rather than scoring every fragment in both systems.

The query side of retrieval repeats work across turns. Embedding the same query twice wastes a full model inference; ReLU.chat memoizes query embeddings so identical queries skip re-embedding entirely. The same pattern applies to the other repeated pieces: normalized fragment vectors (computed once at index time), BM25 idf values (pre-built at index time, not recomputed per query), and dot products for repeated queries. An LRU cache is the right shape for this memoization — queries repeat in bursts, so a small recency-based cache captures most of the savings. Bound the cache size so memory stays predictable, and store only the derived values (query embedding, fused top-k) rather than the raw query strings.

Every optimization above trades complexity for speed, and the right trade depends on your fragment count and query rate. The order that works in practice: profile the scoring loop with performance.now(), normalize at index time and switch to typed arrays, replace the full sort with bounded top-k, memoize repeated query work, then re-measure. In a browser, also test on the weakest device you support — embedding models like the quantized MiniLM ONNX file ReLU.chat loads are memory-heavy, and vector math that is instant on a laptop can stall a phone.

Key Takeaway

Cosine similarity is cheap per comparison and expensive in aggregate: 384 dimensions times thousands of fragments adds up fast in a browser. Precompute norms at index time, store vectors as Float32Array, replace full sorts with bounded top-k selection, and memoize query embeddings with an LRU cache. Each optimization removes a constant factor from the hot path, and together they turn vector search from a jank source into an instant lookup.