RAG auto-tuner: find the best settings for your documents

By TryRAG · Updated

Every RAG system has the same handful of settings: how the documents are cut into chunks, how big the chunks are, which search finds them, and how many are passed to the model. The right values depend on your documents and your questions, and the only reliable way to find them is to test. The auto-tuner in the RAG tool does that testing for you, in your browser, and tells you which setup to use and why.

Run the auto-tuner on the example Open it with your own document

What it needs from you

A few questions, each with words that the right chunk must contain. That is the evaluation set described in How to test RAG retrieval. Six questions are enough to try it; twenty or more give a result you can lean on. Add questions your documents cannot answer, with the expected words left empty, and the tuner will also set the similarity threshold.

What it tries, step by step

Trying every combination would mean thousands of runs, so the tuner works in steps and carries the winner of each into the next:

  1. Chunking method and size. Fixed size, by sentence and by heading, at 25, 40, 60, 90 and 130 words, plus your current setting.
  2. Chunk overlap. 0, 10, 20 and 30 percent of the chunk size.
  3. Search method. Keyword, semantic, hybrid at three mixes, and hybrid by rank fusion. Only when semantic search is switched on, since it needs the embedding model.
  4. Top-k and diversity. One to six chunks, and two levels of diversity.
  5. Re-ranking. On or off, when you have switched it on.
  6. Similarity threshold. Six minimum scores, when some questions have no answer. Here the score also counts how many of those questions were correctly left unanswered.

How it chooses

Each setup is scored by the mean reciprocal rank (MRR): 1 when the right chunk comes first, 1/2 when it comes second, 0 when it is missed, averaged over the questions. Then three rules decide:

With twenty or more questions, every third question is held back. The tuner never sees those, and the result is checked on them at the end, so you can tell whether the setup fits your questions in general or only the ones it was tuned on.

A worked example

The example in the RAG tool is a short bike shop handbook with six test questions. With keyword search and the starting settings (fixed size, 45 words, overlap 10, top 3), five of the six questions find their answer, and MRR is 0.75.

SetupHit rateMRRWords sent per question
Starting settings: fixed size, 45 words, overlap 10, top 383%0.75119
Recommended: by sentence, 130 words, overlap 15, top 1100%1.00120

The tuner tried 26 setups in three steps. Large sentence-based chunks put the answer to every question in the first chunk; once that was true, one chunk was enough, so top-k dropped from 3 to 1 and the words sent stayed the same. The maintenance question, which keyword search missed at every smaller size, is found because its chunk now also holds the heading words the question uses.

The tuner also says what to be careful about. Six questions are too few to call the difference clear, and none could be held back, so it reports that the gain "could be chance". The chunks are large, so it warns that a large chunk contains the expected words more easily, and that the model then has to find the answer in more text. Those notes are part of the result, not small print.

Reading the result

What it does not do

It tunes retrieval: whether the right chunk is found and how high. It does not judge the written answer, which needs a language model to read it. A chunk counts as right when it contains the expected words, which stands in for relevance. And it searches in steps rather than trying every combination, so it can miss a combination that only works when two settings change together. For a final check of one specific change, compare the two setups side by side in RAG Battle.