RAG Interview Questions and Answers

When the knowledge isn't unstructured text that people search by meaning. If the corpus is small and stable, I paste it into the prompt: no retrieval step means no retrieval misses. If the question is structured, like "orders over $500 last month," that's a SQL query behind a tool, because embeddings are fuzzy and the question is exact. If the data changes per request, like an account balance or the user's current plan, I call the live API instead of indexing a snapshot that goes stale. And if it's general, stable knowledge the model already handles well, retrieval only adds latency.

My two sorting questions are: is it structured, and does it change per request? A yes to either usually means a tool, not an index. RAG's lane is semi-stable documents people search by meaning, like a handbook or a help center.

Add keyword search and run hybrid retrieval. An embedding squeezes meaning into one fixed-length vector, and exact surface forms like ERR_CONN_5023 or a SKU are exactly what that compression throws away. BM25 matches those tokens directly.

So I run BM25 and vector search in parallel and merge the lists with reciprocal rank fusion, which combines ranks instead of scores, since BM25 scores and cosine similarities aren't on the same scale. Then a reranker orders the fused top candidates. Before building it, I'd audit the query log for the share of identifier-shaped queries, and I'd slice the eval set by query type. An overall recall number can look fine while identifier queries sit far lower, and that slice is what tells you hybrid is the fix.

Usually the retrieved set is relevant but not sufficient. Recall asks whether a relevant chunk made the top k. It doesn't ask whether the chunks together contain the answer. Three common causes: a multi-hop question where each passage is relevant but the combination is incomplete; the rule was retrieved but its exception wasn't; or the top k is five near-copies of the same half-answer.

Chunks that are too small cause this too. The model gets fragments and invents the connective tissue between them. So I add an answerability metric: on a sample, a human or a calibrated LLM judge marks whether the answer is fully derivable from the selected passages. When retrieval metrics and answer quality disagree like this, chunking is the first thing I look at.

As a pre-filter at retrieval time, bound to the authenticated user from the session. Every chunk carries tenant and permission metadata stamped at indexing, and the search only runs over what this user may see. Filtering after retrieval is weaker: content the user can't see has already affected the ranking and can leak via scores or snippet text. Post-filtering also empties the top k. If nine of ten neighbors belong to other tenants, the user sees one result, which shows up as "search barely works for small customers."

Testing splits in two. Access correctness is pass/fail: assert zero leakage across tenants and roles. Quality is measured per slice with realistic permission contexts, never as a superuser. Caches need the same authorization scope in their keys, or a cached answer becomes the leak.

No prompt reliably stops it, so I limit what a successful injection can do. This is indirect prompt injection: whoever can write to the corpus can, in effect, prompt the model. I fence retrieved text, label its source, and tell the model to treat it as data. That reduces accidents but won't stop a deliberate attack.

The real controls sit downstream. If generation has tools, they're least-privilege, and anything that sends, pays or deletes needs a human confirmation showing the exact action. I validate output before rendering: no auto-fetched images or links to hosts outside an allowlist, because a markdown image URL can carry data out with no tool call at all. For high-risk sources, a reader model with no tools can turn raw documents into structured summaries for an actor that never sees the raw text.