Evaluate RAG retrieval

Test retrieval on many questions at once: hit rate, MRR, precision, recall and nDCG with ranges, the reason behind every miss, and a report you can download and compare. Nothing is uploaded.

Document

Settings

Several documents at once, your own embedding or re-ranking model, and asking single questions are on the RAG tool.

Evaluation set (test questions)

For each question, enter words that the right chunk must contain. Separate answers that sit in different chunks with ; and phrases that must be in the same chunk with +, for example 30 days + full refund; 14 days. Leave it empty for a question the document cannot answer: it then passes if nothing scores above the minimum score. Tags (optional, separated by commas) give a score per kind of question.

Results

Press a button above to measure how well retrieval works.

Next steps: Auto-tune tries settings for you (on the RAG tool), RAG Battle compares setups question by question, and Compare parsers scores two readings of the same document.

What it is for

How it works

This page is for testing retrieval seriously. The RAG tool on the home page is for trying one question at a time and seeing every step; this one runs a whole set of questions and measures the result.

For each question you give words that the right chunk must contain. A chunk that holds all of them is right, one that holds some is partly right. From that come hit rate, MRR, precision, recall, nDCG and context precision, each with a 95% range from resampling the questions, and the same scores at 1, 3, 5 and 10 chunks retrieved.

Questions the document cannot answer can be tested too: leave the words empty, and the question passes when nothing scores above the minimum score. Every miss gets a reason, and "Why?" on a question checks which single setting change would retrieve it.

Questions

How is this different from the RAG tool?

The RAG tool on the home page answers one question at a time and shows every step, for learning. This page runs many questions and measures retrieval, with all the settings. Your document carries across between the two.

What do I type in "must contain"?

A few words that the right chunk holds, for example 30 days. Separate answers in different chunks with ; and phrases that must be in the same chunk with +, for example 30 days + full refund; 14 days. Leave it empty for a question the document cannot answer.

How many questions do I need?

Five are enough to start. About 30 give scores you can rely on when comparing two settings; with fewer, the ranges under each score are wide and small differences may be chance.

Can I compare two settings?

Yes. Download the report as JSON, change a setting, evaluate again, and load the earlier report with "Compare with an earlier report". Each score shows whether the change is clear or could be chance, and which questions moved.

What do the ranges under each score mean?

Each score is an average over your questions, and a different set of questions would give a somewhat different average. The range shows where 95 of 100 such averages would fall, worked out by resampling your own questions. If two settings have overlapping ranges, the difference between them may be chance. With fewer than about 30 questions the ranges are wide.

Can a question have more than one right chunk, or none?

Yes. Separate answers that sit in different chunks with a semicolon, and recall then measures how many of them were retrieved. Join phrases that must appear in the same chunk with a plus sign; a chunk with only some of them counts as partly right in nDCG. Leave the expected words empty for a question the document does not answer: it passes when nothing scores above the minimum score, which tests whether the system would say it does not know.

Can I save or share the results?

Yes. After "Evaluate current settings", the Report row downloads the results as Markdown, JSON or CSV, or opens a printable page you can save as PDF. The report lists your documents by name, size and fingerprint but leaves their text out, unless you tick the box to include the top chunk for each question. To check a change, load an earlier report's JSON with "Compare with an earlier report": it lists the settings that changed, each score before and after, and which questions got better or worse.

Is my text uploaded?

No. Everything runs in your browser tab. Your documents and questions never leave your device.