Evaluate your RAG output
Bring what your own RAG system retrieved and answered, as a CSV or JSON file, and get retrieval and answer scores, the worst questions first, and a comparison with an earlier run. Ragas column names work as they are. Nothing is uploaded.
- 1Upload a file with your questions, retrieved chunks and answers
- 2Check how right chunks are judged
- 3Read the scores and the worst questions, then compare with another run
Your RAG output
File format: one row per question
Use the column names of Ragas, so a Ragas dataset loads as it is. Older names in brackets also work.
| Column | What it holds | Needed |
|---|---|---|
user_input (question) | The question | Yes |
retrieved_contexts (contexts) | The chunks your system retrieved, best first: a JSON list, a Python list, or one chunk | Yes |
response (answer) | The answer your system wrote | For answer scores |
reference (ground_truth) | The correct answer | At least one, to judge the chunks |
reference_contexts | The passages that hold the answer | |
must_contain | Words the right chunk must contain: ; between passages, + between phrases in one chunk | |
retrieved_context_ids + reference_context_ids | Chunk IDs, when your system has them: the most exact way to judge | |
tags | Optional, comma-separated, to see scores by kind of question | No |
Any other column of numbers, such as Ragas scores your pipeline already computed, is averaged and shown as well.
How to judge the chunks
Results
What it is for
- Score a RAG system that already exists, from its logs or a test run, without rebuilding it here.
- Find the questions where the right chunk was not retrieved, and see what was retrieved instead.
- Check whether a change to your pipeline helped, by comparing two runs question by question.
- Read a Ragas dataset, or a file of Ragas results, without writing code.
How it works
The rest of TryRAG retrieves from documents you load into it. This page does not retrieve anything. It scores what your own system retrieved and answered, so the numbers describe your pipeline, with its chunking, embedding model and prompt.
Each row is one question with the chunks your system retrieved, best first. To know which of those chunks are right, the page needs one label per row: chunk IDs, words the right chunk must contain, the reference passages, or the reference answer. Chunk IDs are exact; the others are matched by words, which stands in for relevance and can be fooled by a chunk that shares the words without answering.
Retrieval is scored with hit rate, MRR, precision, recall, nDCG, a rank-weighted context precision in the style of Ragas, and the share of retrieved words that are in right chunks. When the file has answers and reference answers, it also gives exact match, string presence, token F1 and ROUGE-L, and the share of answer words found in the retrieved chunks. Every average comes with a 95% range from resampling the questions.
None of these use a language model, so they are free and repeatable. They do not judge whether an answer is faithful or helpful in the way a model judge such as Ragas faithfulness does; that needs a model, and the answer word scores are only a rough stand-in.
Questions
Can I upload a Ragas dataset?
Yes. A Ragas EvaluationDataset saved as CSV, JSON or JSON Lines uses the same column names: user_input, retrieved_contexts, response, reference and reference_contexts. Lists written by pandas as Python lists are read too, and so are the older names question, contexts, answer and ground_truth.
Are these the same numbers Ragas gives?
Not exactly. Ragas context precision with a model asks the model whether each chunk is useful; here a chunk is right when it matches the label in your file. The formula for context precision is the same rank-weighted average, and ID-based precision matches Ragas, but text matching differs from the string distance Ragas uses. Treat the scores as Ragas-style, not Ragas.
What if my file has no reference answers?
Retrieval can still be scored from chunk IDs or expected words. Without any label, a row is counted as not scored and listed apart.
How do I compare two versions of my pipeline?
Run the same questions through both, load the first file, then use "Compare with another run" for the second. Each question is matched by its text, and each score shows the change with a 95% range, so you can tell a clear change from one that could be chance.
Does it judge the answers with a model?
Not yet. The answer scores compare words with the reference answer and with the chunks, which needs no model. Checking faithfulness claim by claim with your own connected model is planned.
Is there a size limit?
Up to 5,000 rows and 20 MB, which a browser handles in a second or two.
Is my text uploaded?
No. Everything runs in your browser tab. Your documents and questions never leave your device.