How to test RAG retrieval
By TryRAG · Updated
When a RAG answer is wrong, there are two possible causes: the right chunk was never retrieved, or it was retrieved and the model still answered badly. The first is far more common, and it can be measured without calling a model at all. This guide shows how, using six questions and the results they actually produce.
Why test retrieval separately
Judging final answers is slow and subjective, and it mixes two problems together. Retrieval has a clear right answer: for a given question, either the chunk that contains the answer is in the retrieved set or it is not. That makes it quick to check, cheap to repeat, and fair to compare between settings.
Step 1: build an evaluation set
An evaluation set is a list of questions, each paired with something that identifies the right chunk. The simplest form is a short phrase that only the right chunk contains, sometimes called the ground truth or expected text. Five to twenty questions are enough to start.
Here is the set used in this guide. It runs against the example handbook in the RAG tool.
| Question | Right chunk must contain | Why it is in the set |
|---|---|---|
| How many days do I have to return a bike? | 30 days | Direct: reuses the document's words |
| How many days do I have to return a helmet? | 14 days | The answer sits on a chunk boundary |
| What is the part number for the chain? | KC-CH-118 | Exact code |
| How long does delivery take? | five to eight | Several chunks mention delivery |
| How often does my bicycle need maintenance? | once a year | Rephrased: shares no key words with the answer |
| Where is the Kite cluster hosted? | Frankfurt | From a different part of the document |
A good set mixes these kinds on purpose. If every question reuses the document's own wording, almost any setting will pass and the test tells you nothing.
Choose the expected text carefully. It should be specific enough that only the right chunk has it, and short enough that a different split does not cut it in two.
Step 2: measure hit rate, MRR and the rest
For each question, run retrieval and look at where the right chunk came. Two numbers summarise the result.
- Hit rate, or hit@k: the share of questions with a right chunk among the k chunks retrieved. Found for 5 of 6 questions is a hit rate of 83 percent.
- Mean reciprocal rank (MRR): for each question, score 1 if the right chunk is first, 1/2 if second, 1/3 if third, and 0 if it is missing, then take the average. MRR rewards putting the right chunk at the top, which matters because models pay most attention to what comes first.
Hit rate tells you whether the answer is reachable at all. MRR tells you how well it is ranked. You need both: two settings can share a hit rate and still differ in how often the right chunk comes first.
Three more measures are common, and the Evaluate tab shows them too, with the working for your own results:
- Recall@k: of the passages a question needs, the share that were retrieved. With one expected passage per question it equals the hit rate. It says more when an answer is spread over several places, such as a question that needs both the return period for bikes and the one for helmets.
- Precision@k: the share of the k retrieved chunks that are right. When each answer sits in one chunk it cannot go above 1/k, so it says most when answers are spread over several chunks.
- nDCG@k: like MRR it rewards right chunks near the top, but it counts every right chunk, not just the first, each worth less the lower it comes. It is scaled so that 1 is the best order possible.
Two more checks make a set more honest. Add a few questions the document cannot answer: a good system retrieves nothing above its minimum score for them, instead of handing the model unrelated text to answer from. And look at context precision, the share of the retrieved text that is actually in the right chunks: the rest is sent to the model, and paid for, without helping.
Finally, remember that six questions is a small sample. In the Evaluate tab every score has a range under it, worked out by resampling your questions. With fewer than about 30 questions those ranges are wide, and a difference between two settings that sits inside them may be chance.
When you change something, keep a record. The Evaluate tab exports a report of the documents (as fingerprints, not text), the settings, every score with its range and each question's result, as Markdown, JSON, CSV or a printable page. Load an earlier report's JSON file and it shows what changed, and whether each change is clear or could be chance.
Worked example: five settings, six questions
These are the real results for the set above, with keyword search and no model. You can reproduce every row.
| Setting | Hit rate | MRR | Recall | nDCG | What happened |
|---|---|---|---|---|---|
| Fixed size, 45 words, no overlap, top 3 | 4 of 6 (67%) | 0.58 | 0.67 | 0.61 | The helmet answer is cut in two at a chunk boundary |
| Fixed size, 45 words, overlap 10, top 3 | 5 of 6 (83%) | 0.75 | 0.83 | 0.77 | Overlap repairs the helmet answer |
| Fixed size, 20 words, overlap 10, top 3 | 5 of 6 (83%) | 0.72 | 0.83 | 0.67 | Same hit rate, but the delivery answer drops to third place |
| By heading, 100 words, overlap 10, top 3 | 5 of 6 (83%) | 0.83 | 0.83 | 0.83 | The delivery answer moves up to first place |
| Fixed size, 45 words, overlap 10, top 1 | 4 of 6 (67%) | 0.67 | 0.67 | 0.67 | Retrieving one chunk loses the delivery answer, which was second |
Open this evaluation set in the RAG tool
Three things are worth noticing.
- One setting change fixed one question. Adding overlap took the hit rate from 67 to 83 percent, and the per-question results show it was the helmet question that changed. Always look at which questions moved, not only at the total.
- MRR separated settings that hit rate could not. Three settings all found 5 of 6, but the delivery answer came third, second or first depending on the chunking. That difference decides what happens when you later retrieve fewer chunks, as the last row shows.
- One question failed under every setting. The maintenance question shares no key words with its answer, which says "service the bike once a year". No chunk size will fix that with keyword search. A failure that survives every setting points to a different kind of fix, here semantic or hybrid search. Switch the search method and run the set again to check.
Step 3: change one thing at a time
The settings with the most effect are usually:
- chunk size and overlap,
- the chunking strategy (fixed size, by sentence, by heading),
- top-k, the number of chunks retrieved,
- the search method: keyword, semantic, or hybrid.
Change one and keep the rest fixed, or you will not know which change made the difference. The auto-tuner on the Evaluate tab does this for you: it changes one setting at a time, keeps the best, and moves on.
Step 4: read the result sensibly
- Prefer the simplest setting among the best. If several settings find every answer, choose the one with fewer or smaller chunks; it will cost less per question.
- Do not over-fit. With six questions, one question is 17 points of hit rate. Add more questions before trusting a small gap.
- Re-test when documents change. A setting tuned on one manual may not suit the next.
After retrieval: checking the answer
Once retrieval is reliable, the remaining checks are about the model's answer. Two questions are worth asking of each one: is every statement in the answer supported by the retrieved chunks, and does the answer actually address the question? These are often called faithfulness and answer relevance. They need a person or a second model to judge, which is why it pays to get retrieval right first.
A ten-minute routine for your own document
- Load the document into the RAG tool.
- Open the Evaluate tab and enter five or more questions with their expected text.
- Press "Evaluate current settings" and note the hit rate and MRR.
- Press "Auto-tune", read why it chose its setup, then "Use these settings".
- Press "Why?" on any question that is still not found.
Comparing two versions
When a document is updated, or you want to check one specific change, run the same questions before and after. RAG Battle does this side by side and marks each question better, worse or the same.
Already have a RAG system?
Everything above can be measured on your own pipeline too. Save each test question with the chunks your system retrieved, and its answer if it writes one, as a CSV or JSON file, and load it into Evaluate your RAG output. It gives the same scores from your system's own results, with the worst questions first, and compares two runs question by question. Ragas column names work as they are.