RAG interview preparation: questions you can try

By TryRAG · Updated

Preparing for an AI engineer interview? These are RAG interview questions that come up again and again, about retrieval, evaluation and serving models. Each has a short answer, the points worth making, and a button that opens a tool on this site where you can see the answer working on a real document. Trying it once makes the answer much easier to give.

The list keeps to questions a tool here can show. Agents, memory, structured output and fine-tuning runs are left out. More questions will be added as the tools for them arrive.

Retrieval

Your RAG system gives wrong answers. How do you debug it?

In short: Find out first whether the right chunk was retrieved at all, because that splits the problem in two: retrieval or generation.

  • Look at the retrieved chunks before touching the prompt. If the passage with the answer is not among them, the model never had a chance.
  • If it was not retrieved, check in order: is the answer in the extracted text at all (scanned pages and tables often lose it), was it cut across two chunks, does the question use different words from the document, or did it rank just below the top-k?
  • If it was retrieved, the model answered from something else. Look for a similar passage ranked above it, too many chunks, or a prompt that does not insist on using the context.
  • Find the passage the wrong answer came from. Very often the model faithfully used the wrong chunk.
  • Change one thing, then re-run a set of questions, not just the one that failed, so the fix does not break others.

Try it: Debug a wrong answerSeven common RAG mistakes

Choose a chunking strategy for legal contracts.

In short: Split at the contract's own structure, numbered clauses and headings, so each clause stays whole and carries its number; then measure.

  • Fixed-size windows cut clauses mid-sentence and separate obligations from their conditions. Clause or heading boundaries keep a clause intact.
  • Keep the clause number and heading on each chunk, so "Section 12.3" is searchable and the model can cite it.
  • Definitions matter across the whole contract: give each definition its own chunk, or add the relevant ones to the context.
  • Very long clauses still need splitting; repeat the clause title in each piece, and check the pieces fit the embedding model's input limit.
  • Decide with numbers: a set of questions, each with the clause that answers it, and compare strategies on hit rate and MRR.

Try it: Chunking visualizerHow the auto-tuner compares strategies

BM25 vs dense vs hybrid search: when would you use each?

In short: Keyword search for exact terms, dense search for meaning, and hybrid as the usual production default because it catches both.

  • Keyword search (BM25) matches exact words: product codes, names, error numbers, rare terms. It needs no model and is fast, but misses paraphrases.
  • Dense (semantic) search matches meaning, so "maintenance" finds "service the bike". It can miss exact codes and depends on how well the embedding model knows your domain.
  • Hybrid runs both and merges the lists, by blending scores or by reciprocal rank fusion. It costs little more and covers both failure modes.
  • Tag your test questions by type (keyword, paraphrase, exact code) and compare the methods per tag; the averages hide where each one fails.

TryRAG's keyword search is a simpler word-count match than full BM25, which is enough to show the difference.

Try it: evaluate with Keyword, then Semantic, then HybridHybrid search explained

You add a re-ranker. How do you measure its impact?

In short: Run the same questions with and without it and compare ranking scores (MRR, nDCG), question by question, along with the latency it adds.

  • A re-ranker re-scores the top candidates from search, so it changes order more than what is found. MRR and nDCG show that; hit rate at a large k barely moves.
  • Measure at the k you actually send to the model, and vary how many candidates the re-ranker sees.
  • Compare per question: list the questions it fixed and the ones it made worse, and check the difference is larger than chance with a paired test.
  • Report the cost: added latency per query and any model calls.

On TryRAG: turn on Re-rank under Settings on the RAG tool, then run Auto-tune. Its re-ranking step compares on and off on your questions.

Try it: Auto-tune with re-rankingRe-ranking explained

Optimisation

Reduce LLM costs by 70% without losing quality.

In short: Find where the tokens go, send less of them, reuse answers where you safely can, route easy questions to a smaller model, and prove quality held with an evaluation set.

  • In RAG the retrieved context is usually most of the prompt, so start there: fewer or smaller chunks often keep the same quality.
  • Trim the prompt's fixed text, compress the context to the sentences that matter, and use the provider's prompt caching or batch discounts.
  • Cache answers to repeated questions; with a semantic cache, set the similarity threshold carefully, because "return a bike" and "return a helmet" look alike.
  • Route simple questions to a smaller model and keep the large one for hard cases.
  • "Without losing quality" needs proof: the same question set before and after, with the scores and their ranges.

Try it: RAG cost calculatorToken counterThe auto-tuner picks the cheapest setup that is as good

Evaluation

Build an evaluation suite for a customer support bot.

In short: A golden set of real questions with the passages that answer them, retrieval and answer scores with ranges, run on every change and compared with the last run.

  • Take real questions from support tickets, grouped by intent, each with the passage or passages that answer it. Add questions the documents cannot answer, which the bot should decline.
  • Score retrieval (hit rate, MRR, recall, nDCG) separately from answers (correct, grounded in the chunks, declined when it should be).
  • Give every score a range: with fewer than about 30 questions, two versions often cannot be told apart.
  • Run the suite on every change to documents, chunking, prompt or model, compare with the previous report, and list the questions that got worse.
  • Keep feeding it: sample production conversations and add the failures to the set.

Try it: Evaluate RAG retrievalEvaluate your RAG outputHow to test RAG retrieval

LLM-as-a-judge: how do you make the evaluations trustworthy?

In short: Give the judge the evidence and a narrow rubric, check it against human labels, and test it for consistency and bias before trusting its numbers.

  • Show the judge the retrieved chunks and, if you have one, the reference answer, and ask a narrow question ("is every claim supported: yes, partly, no?") with a short reason.
  • Prefer checking claim by claim over one overall score.
  • Calibrate on a sample labelled by people and report how often the judge agrees.
  • Test consistency: run it twice, swap the order in pairwise comparisons (judges favour one position), and avoid letting a model judge its own answers.
  • Treat its scores as estimates with ranges, and keep spot-checking.

On TryRAG, RAG Battle's answer check shows every verdict next to the chunks, so you can judge the judge.

Try it: RAG Battle with your own model

How do you detect and reduce hallucinations?

In short: Detect them by checking the answer's claims against the retrieved chunks; reduce them by fixing retrieval first and letting the model say it does not know.

  • Detect: check each claim against the context (faithfulness), flag numbers and names that appear in none of the chunks, and verify any citations.
  • Missing context invites invention, so fix retrieval first: a model given the right passage hallucinates far less.
  • Tell the model to answer only from the context and to say when the answer is not there; set a minimum score so questions with no good match get no chunks.
  • Measure: the share of answers fully supported, and how often unanswerable questions are correctly declined.

Try it: Debug a wrong answerTest declining: leave "must contain" empty and set a minimum score

Fine-tuning vs RAG vs prompting: choose one and defend it.

In short: Prompting first; RAG when answers depend on your documents or change often; fine-tuning to change behaviour or format, not to add facts.

  • Prompting is cheapest and fastest to change; it is enough when the model already knows the task and the facts.
  • RAG fits when answers come from your documents, the facts change, or you need citations and access control. Knowledge stays outside the model and updates when the documents do.
  • Fine-tuning changes behaviour: tone, format, a narrow skill, or getting a small model to do a task cheaply. It is a poor way to add facts that change.
  • They combine: RAG for the knowledge, light fine-tuning for the format.
  • Defend the choice with numbers: an evaluation set comparing the options, cost per query, and how often the knowledge changes.

What is RAG?Self-hosting vs an API: the cost side

Production

Serve an open-source model at scale using batching, KV caching and quantization.

In short: Make the weights and the KV cache fit, then keep the GPU busy with continuous batching, and check what each choice does to cost per token and latency.

  • Memory first: weights take parameters × bytes per weight; the KV cache takes 2 × layers × KV heads × head size × bytes for every token of every request in flight.
  • Quantization (8-bit, 4-bit) shrinks the weights and leaves room for more KV cache and larger batches; check the quality loss.
  • Continuous batching refills slots as requests finish, and a paged KV cache stops memory being reserved and wasted; watch throughput and p95 latency together.
  • Speculative decoding can speed up generation when the draft model's guesses are usually accepted.
  • Cost per million tokens depends mostly on how busy the GPU is; compare it with an API at your real volume.

Try it: KV cache for Llama 3 70B on an A100All five inference playgrounds, in order

How do you roll back a bad prompt change in production?

In short: Version prompts like code and ship them behind a switch, test old and new on a fixed question set first, and when it goes wrong, roll back first and diagnose second.

  • Keep prompts in version control and deploy them behind a flag or a gradual rollout, so going back is a switch, not a deployment.
  • Before release, run the same questions through the old and the new prompt and compare them question by question.
  • In production, watch refusals, answer length, user feedback and judge scores, and alert when they move.
  • Roll back first, then find the cause with the regression set, and add the failing cases to it.

On TryRAG, RAG Battle's "The prompt" mode runs the same questions through prompt A and prompt B with your own model.

Try it: RAG Battle, prompt A vs prompt B