Compare Two RAG Setups
A free online RAG A/B testing playground. Run the same questions against two versions of a document, two sets of settings or two prompts, and see what got better or worse. Nothing is uploaded.
- 1Choose what differs: the document, the settings or the prompt
- 2Add your questions
- 3Press Compare and read what changed
Document A
Document B (new version)
Settings
Settings B
Prompt A
Prompt B
Questions
?Optional second box: words the right passage must contain. With it, the result says better or worse; without it, only whether the top passage changed.
File format for uploading questions
Text file (.txt): one question per line.
How many days do I have to return a bike? How long is the frame warranty?
CSV file (.csv): the question in the first column; the words the right passage must contain in the second column (optional). A first row of question,must is skipped. Put a question that contains a comma inside double quotes.
question,must How many days do I have to return a bike?,days of delivery "Is assembly refunded, and how much?",40 dollars
Up to 200 questions. Uploading replaces the questions on the page. to edit.
Result
What changed in the document
Text added or removed early in a document shifts where later passages are cut, so those can show as edited too. Splitting by heading keeps sections stable.
Optional. With a model connected, each question is also answered from A and from B, and the model checks each answer against its passages. Requests go straight from your browser to this address.
What it is for
- Check that an updated document still answers the questions the old one did, before you replace it.
- See whether a change to passage size, overlap or matching makes retrieval better or worse on your own questions.
- Compare two prompts on the same retrieved passages, using a model you connect.
- Keep a list of test questions and re-run it whenever something changes.
How it works
A RAG system has three things you can change: the documents, the retrieval settings and the prompt. Changing any of them can fix one question and quietly break another. The only way to know is to run the same questions before and after, and compare.
This page does that side by side. Side A is what you have now and side B is the change. Every question is run against both, and the result shows where the right passage was found, at what position, and whether that is better, worse or the same. When the documents differ, a second panel lists the passages that were added, removed or edited, which usually explains why a question changed.
Retrieval is compared entirely in your browser, with no model. If you connect your own model, each question is also answered from both sides, and the model checks each answer against the passages it was given. A model checking answers is helpful but lenient, so the passages are shown next to every verdict for you to judge.
Questions
How do I test whether a new version of a document is better?
Choose "The document", load the current version as A and the new one as B, add your questions and press Compare. Each question is marked better, worse, same or changed, and the lower panel shows which passages differ between the versions.
What are the "must contain" words for?
They tell the tool which passage is the right one. With them it can say whether the right passage was found and at what position on each side. Without them it can only say whether the top passage changed.
Can I upload a list of questions?
Yes. Upload a text file with one question per line, or a CSV with the question in the first column and the optional "must contain" words in the second. Press Sample file to download an example you can edit, and open "File format for uploading questions" for the details.
Are both versions kept?
Both are held in memory while the page is open, so you can keep running questions against them. Save comparison downloads a file with both documents, the settings and the questions, which you can load again later or send to someone else. Nothing is stored on a server.
Do I need an AI model?
No. Retrieval is compared without one. Connecting your own model adds the written answers and an automatic check of each answer; those calls go to your model and are counted before they run.
What is the difference from the Tune tab?
Tune tries many settings on one document and ranks them. Compare looks closely at two specific setups, question by question, and also handles two document versions and two prompts.
Is my text uploaded?
No. Everything runs in your browser tab. Your documents and questions never leave your device.
More
Try RAG on your own document · Chunking Visualizer · Token Counter & Cost Estimator · Graph RAG Playground · RAG Cost Calculator · What is RAG? · How to choose a chunk size for RAG · What are embeddings? · What are tokens, and what do they cost? · Hybrid search: words plus meaning · How to test RAG retrieval · Seven common RAG mistakes · What is GraphRAG?