Evaluate your RAG output

Bring what your own RAG system retrieved and answered, as a CSV or JSON file, and get retrieval and answer scores, the worst questions first, and a comparison with an earlier run. Ragas column names work as they are. Nothing is uploaded.

Your RAG output

Drop a CSV, JSON or JSON Lines file here, or choose one. Nothing is uploaded: the file is read and scored in your browser.
File format: one row per question

Use the column names of Ragas, so a Ragas dataset loads as it is. Older names in brackets also work.

ColumnWhat it holdsNeeded
user_input (question)The questionYes
retrieved_contexts (contexts)The chunks your system retrieved, best first: a JSON list, a Python list, or one chunkYes
response (answer)The answer your system wroteFor answer scores
reference (ground_truth)The correct answerAt least one, to judge the chunks
reference_contextsThe passages that hold the answer
must_containWords the right chunk must contain: ; between passages, + between phrases in one chunk
retrieved_context_ids + reference_context_idsChunk IDs, when your system has them: the most exact way to judge
tagsOptional, comma-separated, to see scores by kind of questionNo

Any other column of numbers, such as Ragas scores your pipeline already computed, is averaged and shown as well.

What it is for

How it works

The rest of TryRAG retrieves from documents you load into it. This page does not retrieve anything. It scores what your own system retrieved and answered, so the numbers describe your pipeline, with its chunking, embedding model and prompt.

Each row is one question with the chunks your system retrieved, best first. To know which of those chunks are right, the page needs one label per row: chunk IDs, words the right chunk must contain, the reference passages, or the reference answer. Chunk IDs are exact; the others are matched by words, which stands in for relevance and can be fooled by a chunk that shares the words without answering.

Retrieval is scored with hit rate, MRR, precision, recall, nDCG, a rank-weighted context precision in the style of Ragas, and the share of retrieved words that are in right chunks. When the file has answers and reference answers, it also gives exact match, string presence, token F1 and ROUGE-L, and the share of answer words found in the retrieved chunks. Every average comes with a 95% range from resampling the questions.

None of these use a language model, so they are free and repeatable. They do not judge whether an answer is faithful or helpful in the way a model judge such as Ragas faithfulness does; that needs a model, and the answer word scores are only a rough stand-in.

Questions

Can I upload a Ragas dataset?

Yes. A Ragas EvaluationDataset saved as CSV, JSON or JSON Lines uses the same column names: user_input, retrieved_contexts, response, reference and reference_contexts. Lists written by pandas as Python lists are read too, and so are the older names question, contexts, answer and ground_truth.

Are these the same numbers Ragas gives?

Not exactly. Ragas context precision with a model asks the model whether each chunk is useful; here a chunk is right when it matches the label in your file. The formula for context precision is the same rank-weighted average, and ID-based precision matches Ragas, but text matching differs from the string distance Ragas uses. Treat the scores as Ragas-style, not Ragas.

What if my file has no reference answers?

Retrieval can still be scored from chunk IDs or expected words. Without any label, a row is counted as not scored and listed apart.

How do I compare two versions of my pipeline?

Run the same questions through both, load the first file, then use "Compare with another run" for the second. Each question is matched by its text, and each score shows the change with a 95% range, so you can tell a clear change from one that could be chance.

Does it judge the answers with a model?

Not yet. The answer scores compare words with the reference answer and with the chunks, which needs no model. Checking faithfulness claim by claim with your own connected model is planned.

Is there a size limit?

Up to 5,000 rows and 20 MB, which a browser handles in a second or two.

Is my text uploaded?

No. Everything runs in your browser tab. Your documents and questions never leave your device.