Evaluating retrieval: the basics
Every earlier post in this series ended with a parameter you were told to tune and no way to tell whether tuning it helped. This one closes that: a small labeled set of real queries, recall@k as the number you actually watch, and MRR when rank position matters. Shorter than usual, and honest about why.
Evaluating retrieval: the basics
A four-minute read — shorter than the rest of this series on purpose. I’m writing it from the silent work area at Bits & Pretzels in Munich, between sessions, on a laptop balanced on my knees. The topic deserves a longer treatment and will probably get one later. What’s here is the part you need before you change anything else.
The hybrid search post ended by saying that tuning alpha by intuition is guessing, and that doing it properly needs a labeled set of queries and a way to score the results. This is that.
Why you can’t eyeball this
Every post in this series handed you a dial and no gauge. Chunk size and overlap. Which embedding model. top_k. score_threshold. RRF’s k, or alpha if you went with weighted fusion. Each one plausibly improves retrieval and each one plausibly wrecks it, and the two look identical from a handful of demo queries — because the queries you try by hand are the ones you already know work.
I ended the last post admitting I shipped hybrid search onto a corpus of narrative prose and made retrieval measurably worse. The word doing the work there is measurably. I didn’t notice from using it. Nobody noticed from using it. It surfaced when I finally ran a query set against both configurations and the merged one came out behind.
The golden set
A labeled set of queries, each paired with the chunk IDs that should come back. That’s the entire artifact.
Three rules about where the queries come from:
- Take them from real logs, not your imagination. A set you invented measures how well the system answers questions you’d think to ask, which is not the thing you care about.
- Include the ugly ones. Error codes, part numbers, names, half-sentences, typos. The failures from the hybrid search post only show up if those queries are in the set.
- Label chunks, not answers. You’re testing retrieval here, not generation. Did the right passage come back, in what position.
Thirty queries is the floor — below that the numbers move around enough between runs that you’ll chase noise. Fifty to two hundred is the range where this starts gating real decisions. An afternoon’s work, once.
Two metrics, mostly
Recall@k — of the chunks that should have come back, what fraction landed in the top k. This is the one to watch. If the answering chunk isn’t in the top k, nothing downstream can recover it: the RAG post’s whole point.
MRR — the reciprocal of the rank of the first relevant chunk, averaged over queries. Recall@10 treats rank 1 and rank 10 identically; the language model reading those chunks does not. MRR is what tells you a change reordered things for the better even when recall held flat.
def recall_at_k(retrieved: list[str], relevant: set[str], k: int) -> float:
if not relevant:
return 0.0
return len(set(retrieved[:k]) & relevant) / len(relevant)
def reciprocal_rank(retrieved: list[str], relevant: set[str]) -> float:
for rank, chunk_id in enumerate(retrieved, start=1):
if chunk_id in relevant:
return 1 / rank
return 0.0NDCG exists and is the right answer when relevance is graded rather than binary — some chunks partly answer the question. Worth reaching for later. Recall@5 and MRR will carry you a long way first.
Two things that bite beginners
A golden set labeled by chunk ID dies the moment you re-chunk. Change the chunk size and every ID in your labels points at a passage that no longer exists, so the whole set silently scores zero — on precisely the change you built it to evaluate. Label by document plus a distinctive quoted phrase, and resolve to current chunk IDs at evaluation time.
One number going up is not the same as retrieval getting better. Recall@20 improves almost automatically when you raise k, while the prompt gets stuffed with noise and answers get worse. Fix k, change one thing at a time, and watch recall and MRR together.
Next in this series: reranking — the other refinement the RAG post named and declined to explain. It’s the first change I’d actually make now that there’s a way to tell whether it helped.