A retrieval system without a measurement loop is a system tuned by vibes. You change a BM25 parameter or an ensemble weight, the answers feel better or worse, and nothing records why. Measuring retrieval quality fixes this: it turns "the answers feel off" into "precision@5 dropped from 0.80 to 0.62 after the change," which is actionable.
The standard toolset is four metrics — precision, recall, MRR, and NDCG — each answering a different question. This post defines each one precisely, works a small example, and shows how to use them to tune a knowledge-base retriever like the one behind ReLU.chat.
Build a Labeled Evaluation Set First
Every metric in this post is computed against relevance judgments: for a set of test queries, a human (or a careful script) marks how relevant each retrieved item is to the query. The judgments define the ground truth, so they define what "quality" means.
For graded judgments, use a small ordinal scale such as 0 (irrelevant), 1 (related), 2 (relevant), 3 (exact answer). The structure of ReLU.chat's knowledge base makes this natural: entries are split into categorized fragments — definitions, explanations, examples, formulas, applications — each carrying metadata like truth confidence, source confidence, and difficulty. A query about "expected value formula" should retrieve the formula fragment first, then the definition fragment, then an example.
Keep the set small and clean. Fifty well-judged queries beat five hundred sloppy ones, because the metrics only propagate the quality of the judgments.
Precision and Recall
Precision is the fraction of retrieved items that are relevant. Recall is the fraction of relevant items that were retrieved.
For a query with 8 relevant fragments in the knowledge base, where the retriever returns 10 fragments of which 6 are relevant:
- precision = 6 / 10 = 0.60
- recall = 6 / 8 = 0.75
In a chat UI, the user sees a handful of candidates, so ranked variants matter: precision@k and recall@k look only at the top k results. precision@3 = 1.0 means the first three suggestions are all relevant; recall@10 = 0.75 means three quarters of everything relevant made it into the top ten.
Precision and recall pull in opposite directions. Broadening retrieval (more expansion terms, lower BM25 b) raises recall and typically lowers precision. Any retriever change should be reported as a pair, not a single number.
MRR: The First Relevant Answer
Mean Reciprocal Rank measures how early the first relevant result appears. For a single query, reciprocal rank is 1 divided by the rank of the first relevant result: if the first relevant fragment is at position 2, the query contributes 1/2 = 0.5. MRR averages this over all queries:
MRR = (1 / rank_1 + 1 / rank_2 + ... + 1 / rank_n) / n
MRR suits conversational retrieval, where the user mostly needs one good answer. If the top result is relevant 80% of the time and the second result is relevant the other 20%, MRR is 0.8 + 0.2 / 2 = 0.9. It ignores everything after the first hit, so it is a poor metric for browse-style queries that need several diverse fragments.
NDCG: Graded Relevance with Position Discount
NDCG — Normalized Discounted Cumulative Gain — is the workhorse for graded judgments. It rewards relevant results at the top and discounts results that appear lower. For a ranking of k items with relevance grades g_i:
DCG@k = g_1 + sum_{i=2..k} g_i / log2(i)
The log2(i) discount means a grade-3 fragment at position 4 contributes 3 / log2(4) = 1.5, half of what it would contribute at position 1. NDCG divides DCG by IDCG — the DCG of the ideal ranking, where the highest grades are at the top — so the result is always between 0 and 1.
Worked example. Query: "Nash equilibrium definition." Retrieved grades: [3, 1, 2, 0].
DCG@4 = 3 + 1/log2(2) + 2/log2(3) + 0/log2(4)
= 3 + 1 + 1.26 + 0 = 5.26
IDCG@4 = 3 + 2/log2(2) + 1/log2(3) + 0/log2(4)
= 3 + 2 + 0.63 + 0 = 5.63
NDCG@4 = 5.26 / 5.63 = 0.934
The ranking is near-ideal because the best fragment is first. Swap the first two grades to [1, 3, 2, 0] and NDCG falls to about 0.77 — the metric penalizes burying the exact definition under a merely-related fragment, exactly what a chat UI should care about.
Use the Metrics to Tune the Ensemble
With an eval set and four metrics, tuning becomes a loop: change one parameter, rerun, compare. A healthy workflow for a hybrid retriever is to evaluate each component and the fusion separately. For ReLU.chat, that means dense cosine similarity alone, field-weighted BM25 alone, and the 70/30 dense/sparse rank fusion, tracking precision@3, recall@10, MRR, and NDCG@10 across all three.
Expect component-specific signals. Sparse-only typically wins precision for exact names and aliases; dense-only wins recall for paraphrase; the 70/30 fusion should dominate both on NDCG if the weights are sane. If the fusion loses to a component, the fusion weights — not the components — are the suspect. This is also the loop to run when the knowledge base grows: add fragments, re-judge the affected queries, and confirm the metrics move in the intended direction.
Key Takeaway
Measure retrieval quality with the four standard metrics, each for its purpose: precision and recall for top-k usefulness, MRR for "is there one good answer and how early," and NDCG for graded relevance with position discounting. Build a small, clean labeled eval set from the knowledge base itself, then tune the dense/sparse ensemble against it — one parameter at a time — so every retrieval change lands with a number attached.