RAG auto-tuner: find the best settings for your documents
By TryRAG · Updated
Every RAG system has the same handful of settings: how the documents are cut into chunks, how big the chunks are, which search finds them, and how many are passed to the model. The right values depend on your documents and your questions, and the only reliable way to find them is to test. The auto-tuner in the RAG tool does that testing for you, in your browser, and tells you which setup to use and why.
Run the auto-tuner on the example Open it with your own document
What it needs from you
A few questions, each with words that the right chunk must contain. That is the evaluation set described in How to test RAG retrieval. Six questions are enough to try it; twenty or more give a result you can lean on. Add questions your documents cannot answer, with the expected words left empty, and the tuner will also set the similarity threshold.
What it tries, step by step
Trying every combination would mean thousands of runs, so the tuner works in steps and carries the winner of each into the next:
- Chunking method and size. Fixed size, by sentence and by heading, at 25, 40, 60, 90 and 130 words, plus your current setting.
- Chunk overlap. 0, 10, 20 and 30 percent of the chunk size.
- Search method. Keyword, semantic, hybrid at three mixes, and hybrid by rank fusion. Only when semantic search is switched on, since it needs the embedding model.
- Top-k and diversity. One to six chunks, and two levels of diversity.
- Re-ranking. On or off, when you have switched it on.
- Similarity threshold. Six minimum scores, when some questions have no answer. Here the score also counts how many of those questions were correctly left unanswered.
How it chooses
Each setup is scored by the mean reciprocal rank (MRR): 1 when the right chunk comes first, 1/2 when it comes second, 0 when it is missed, averaged over the questions. Then three rules decide:
- Ties are measured, not guessed. Two setups are tied when the 95% range of their per-question difference, from resampling the questions 1,000 times, includes zero. With few questions, most setups tie.
- A cheaper setup may replace the best only if it is close. It must be tied with the best, score within 0.05 of it, and find at least as many questions.
- Among those, the fewest words wins. Words sent to the model per question decide the cost and how much the model has to read, so the smaller setup is preferred.
With twenty or more questions, every third question is held back. The tuner never sees those, and the result is checked on them at the end, so you can tell whether the setup fits your questions in general or only the ones it was tuned on.
A worked example
The example in the RAG tool is a short bike shop handbook with six test questions. With keyword search and the starting settings (fixed size, 45 words, overlap 10, top 3), five of the six questions find their answer, and MRR is 0.75.
| Setup | Hit rate | MRR | Words sent per question |
|---|---|---|---|
| Starting settings: fixed size, 45 words, overlap 10, top 3 | 83% | 0.75 | 119 |
| Recommended: by sentence, 130 words, overlap 15, top 1 | 100% | 1.00 | 120 |
The tuner tried 26 setups in three steps. Large sentence-based chunks put the answer to every question in the first chunk; once that was true, one chunk was enough, so top-k dropped from 3 to 1 and the words sent stayed the same. The maintenance question, which keyword search missed at every smaller size, is found because its chunk now also holds the heading words the question uses.
The tuner also says what to be careful about. Six questions are too few to call the difference clear, and none could be held back, so it reports that the gain "could be chance". The chunks are large, so it warns that a large chunk contains the expected words more easily, and that the model then has to find the answer in more text. Those notes are part of the result, not small print.
Reading the result
- Questions that changed lists every question that got better or worse. Read it before switching: a better average can hide one important question that got worse.
- How each step decided gives the reason for each choice in one line.
- Every question against the main setups is a grid: one row per question, one column per setup (your starting settings, the best of each step and each chunking method, the recommendation, and any other you add), each cell the position of the first right chunk. It shows which questions every setup gets, which only some get, and which none do.
- Quality against words sent plots every setup tried. Points high and to the left are good and cheap.
- Use these settings applies the recommendation and evaluates it, so you see the same numbers in the normal results.
- Compare in RAG Battle opens your starting settings as A and the recommendation as B on the same questions. Add a third set as C, and connect your own model there to compare the written answers too.
- The report download (Markdown, JSON, CSV or PDF) includes every setup tried and the pick at each step.
What it does not do
It tunes retrieval: whether the right chunk is found and how high. It does not judge the written answer, which needs a language model to read it. A chunk counts as right when it contains the expected words, which stands in for relevance. And it searches in steps rather than trying every combination, so it can miss a combination that only works when two settings change together. For a final check of one specific change, compare the two setups side by side in RAG Battle.