RAG Battle: Compare RAG Setups

Put RAG setups head to head. A free online RAG A/B testing playground. Run the same questions against two versions of a document, two or three sets of settings, or two prompts, and see what got better or worse. Nothing is uploaded.

What is different between A and B?

Document A

Document B (new version)

Settings

Questions

?

Optional second box: words the right chunk must contain. With it, the result says better or worse; without it, only whether the top chunk changed.

File format for uploading questions

Text file (.txt): one question per line.

How many days do I have to return a bike?
How long is the frame warranty?

CSV file (.csv): the question in the first column; the words the right chunk must contain in the second column (optional). A first row of question,must is skipped. Put a question that contains a comma inside double quotes.

question,must
How many days do I have to return a bike?,days of delivery
"Is assembly refunded, and how much?",40 dollars

Up to 200 questions. Uploading replaces the questions on the page. to edit.

Result

Press Compare to run every question against A and B.

Optional. With a model connected, each question is also answered from A and from B, and the model checks each answer against its chunks. Requests go straight from your browser to this address.

Examples

What it is for

How it works

A RAG system has three things you can change: the documents, the retrieval settings and the prompt. Changing any of them can fix one question and quietly break another. The only way to know is to run the same questions before and after, and compare.

This page does that side by side. Side A is what you have now and side B is the change. Every question is run against both, and the result shows where the right chunk was found, at what position, and whether that is better, worse or the same. When the documents differ, a second panel lists the chunks that were added, removed or edited, which usually explains why a question changed.

Retrieval is compared entirely in your browser, with no model. If you connect your own model, each question is also answered from both sides, and the model checks each answer against the chunks it was given. A model checking answers is helpful but lenient, so the chunks are shown next to every verdict for you to judge.

Questions

How do I test whether a new version of a document is better?

Choose "The document", load the current version as A and the new one as B, add your questions and press Compare. Each question is marked better, worse, same or changed, and the lower panel shows which chunks differ between the versions.

What are the "must contain" words for?

They tell the tool which chunk is the right one. With them it can say whether the right chunk was found and at what position on each side. Without them it can only say whether the top chunk changed.

Can I compare more than two settings?

Yes, up to three. Choose "The settings", then "+ Add settings C". C starts with the auto-tune recommendation if you have run the auto-tuner on the RAG tool, or else with a different chunking method; change anything you like. Each question shows where the right chunk was found under A, B and C, and B and C are each marked better or worse than A.

Can I upload a list of questions?

Yes. Upload a text file with one question per line, or a CSV with the question in the first column and the optional "must contain" words in the second. Press Sample file to download an example you can edit, and open "File format for uploading questions" for the details.

Are both versions kept?

Both are held in memory while the page is open, so you can keep running questions against them. Save comparison downloads a file with both documents, the settings and the questions, which you can load again later or send to someone else. Nothing is stored on a server.

Do I need an AI model?

No. Retrieval is compared without one. Bringing your own model (BYOM), such as Ollama, Gemini or Claude, adds the written answers and an automatic check of each answer; those calls go to your model and are counted before they run.

What is the difference from the Evaluate tab?

The Evaluate tab auto-tunes: it tries many settings on one document and recommends one. Compare looks closely at two specific setups, question by question, and also handles two document versions and two prompts.

Is my text uploaded?

No. Everything runs in your browser tab. Your documents and questions never leave your device.