How to test RAG retrieval

By TryRAG · Updated

When a RAG answer is wrong, there are two possible causes: the right chunk was never retrieved, or it was retrieved and the model still answered badly. The first is far more common, and it can be measured without calling a model at all. This guide shows how, using six questions and the results they actually produce.

Why test retrieval separately

Judging final answers is slow and subjective, and it mixes two problems together. Retrieval has a clear right answer: for a given question, either the chunk that contains the answer is in the retrieved set or it is not. That makes it quick to check, cheap to repeat, and fair to compare between settings.

Step 1: build an evaluation set

An evaluation set is a list of questions, each paired with something that identifies the right chunk. The simplest form is a short phrase that only the right chunk contains, sometimes called the ground truth or expected text. Five to twenty questions are enough to start.

Here is the set used in this guide. It runs against the example handbook in the RAG tool.

QuestionRight chunk must containWhy it is in the set
How many days do I have to return a bike?30 daysDirect: reuses the document's words
How many days do I have to return a helmet?14 daysThe answer sits on a chunk boundary
What is the part number for the chain?KC-CH-118Exact code
How long does delivery take?five to eightSeveral chunks mention delivery
How often does my bicycle need maintenance?once a yearRephrased: shares no key words with the answer
Where is the Kite cluster hosted?FrankfurtFrom a different part of the document

A good set mixes these kinds on purpose. If every question reuses the document's own wording, almost any setting will pass and the test tells you nothing.

Choose the expected text carefully. It should be specific enough that only the right chunk has it, and short enough that a different split does not cut it in two.

Step 2: measure hit rate, MRR and the rest

For each question, run retrieval and look at where the right chunk came. Two numbers summarise the result.

Hit rate tells you whether the answer is reachable at all. MRR tells you how well it is ranked. You need both: two settings can share a hit rate and still differ in how often the right chunk comes first.

Three more measures are common, and the Evaluate tab shows them too, with the working for your own results:

Two more checks make a set more honest. Add a few questions the document cannot answer: a good system retrieves nothing above its minimum score for them, instead of handing the model unrelated text to answer from. And look at context precision, the share of the retrieved text that is actually in the right chunks: the rest is sent to the model, and paid for, without helping.

Finally, remember that six questions is a small sample. In the Evaluate tab every score has a range under it, worked out by resampling your questions. With fewer than about 30 questions those ranges are wide, and a difference between two settings that sits inside them may be chance.

When you change something, keep a record. The Evaluate tab exports a report of the documents (as fingerprints, not text), the settings, every score with its range and each question's result, as Markdown, JSON, CSV or a printable page. Load an earlier report's JSON file and it shows what changed, and whether each change is clear or could be chance.

Worked example: five settings, six questions

These are the real results for the set above, with keyword search and no model. You can reproduce every row.

SettingHit rateMRRRecallnDCGWhat happened
Fixed size, 45 words, no overlap, top 34 of 6 (67%)0.580.670.61The helmet answer is cut in two at a chunk boundary
Fixed size, 45 words, overlap 10, top 35 of 6 (83%)0.750.830.77Overlap repairs the helmet answer
Fixed size, 20 words, overlap 10, top 35 of 6 (83%)0.720.830.67Same hit rate, but the delivery answer drops to third place
By heading, 100 words, overlap 10, top 35 of 6 (83%)0.830.830.83The delivery answer moves up to first place
Fixed size, 45 words, overlap 10, top 14 of 6 (67%)0.670.670.67Retrieving one chunk loses the delivery answer, which was second

Open this evaluation set in the RAG tool

Three things are worth noticing.

Step 3: change one thing at a time

The settings with the most effect are usually:

Change one and keep the rest fixed, or you will not know which change made the difference. The auto-tuner on the Evaluate tab does this for you: it changes one setting at a time, keeps the best, and moves on.

Step 4: read the result sensibly

After retrieval: checking the answer

Once retrieval is reliable, the remaining checks are about the model's answer. Two questions are worth asking of each one: is every statement in the answer supported by the retrieved chunks, and does the answer actually address the question? These are often called faithfulness and answer relevance. They need a person or a second model to judge, which is why it pays to get retrieval right first.

A ten-minute routine for your own document

  1. Load the document into the RAG tool.
  2. Open the Evaluate tab and enter five or more questions with their expected text.
  3. Press "Evaluate current settings" and note the hit rate and MRR.
  4. Press "Auto-tune", read why it chose its setup, then "Use these settings".
  5. Press "Why?" on any question that is still not found.

Comparing two versions

When a document is updated, or you want to check one specific change, run the same questions before and after. RAG Battle does this side by side and marks each question better, worse or the same.

Already have a RAG system?

Everything above can be measured on your own pipeline too. Save each test question with the chunks your system retrieved, and its answer if it writes one, as a CSV or JSON file, and load it into Evaluate your RAG output. It gives the same scores from your system's own results, with the worst questions first, and compares two runs question by question. Ragas column names work as they are.