RAG vs Fine-Tuning: How to Choose (and How to Answer It in an Interview)

RAG and fine-tuning solve different problems. RAG (retrieval-augmented generation) puts relevant documents into the model’s context on each request, so it fixes missing, private or outdated knowledge. Fine-tuning keeps training the model on your examples, so it changes behavior: tone, output format, task patterns. If the model doesn’t know something, retrieve it. If it knows enough but responds the wrong way, prompt it, and fine-tune only when prompting can’t get there.

That one sorting question, “is this a facts problem or a behavior problem?”, settles most real decisions and most interview versions of this question. The rest of this post covers the mechanics behind it, the comparison table, the cases where you need both or neither, and how to say it out loud. If you’re preparing for an interview, the 50 free LLM interview questions cover this and the neighboring concepts.

What each one changes

A trained model has two places information can live: its weights, fixed when training finishes, and its context window, the tokens you send with each request.

Fine-tuning works on the weights. It’s the same training process used to build the model in the first place (next-token prediction on examples), aimed at a smaller dataset of your own. Instruction tuning, the step that turns a base model into an assistant, is supervised fine-tuning on prompt and response pairs, and it mainly shapes format and behavior. Your own fine-tune does the same thing at smaller scale.

RAG works on the context. At request time, the application searches an index for passages that match the question, inserts them into the prompt with source labels, and tells the model to answer from them and cite them. The weights never change. Remove the retrieval step and the knowledge is gone, which is the point: you can update it, audit it and filter it per user.

Why fine-tuning is the wrong tool for facts

The common request is “fine-tune the model on our docs so it knows our business.” It sounds reasonable and usually disappoints, for mechanical reasons:

• Recall stays unreliable. Training compresses patterns into the weights; it doesn’t build a lookup table. A model tuned on your handbook picks up its vocabulary and style, then still gets specific numbers and clauses wrong, the same way a base model hallucinates.
• No citations. The answer comes from the weights, so there’s no passage to point to and nothing a reviewer can check.
• No per-user permissions. Everything in the training set is available to everyone who can call the model.
• Slow updates. Changing a policy means another training run and another evaluation, versus re-indexing one document.

The same reasoning explains why a model has a knowledge cutoff in the first place: once training ends, the weights are frozen, and the fix for missing facts is to supply them at runtime through retrieval or a tool.

Why RAG is the wrong tool for behavior

The reverse mistake is less common but just as real. If the problem is that answers are too long, in the wrong format or in the wrong voice, retrieved passages don’t help. You’d spend context tokens on evidence when the fix is a clearer instruction, a few examples in the prompt, structured outputs, or, if those plateau, a fine-tune on examples of the behavior you want.

Comparison table

When to use both, and when to use neither

Mature systems often combine them: a model tuned on your team’s style and task shape, answering from retrieved current documents. A support model that writes like your best agents and quotes this week’s refund policy is that combination. The two don’t compete; they answer different questions.

Often you need neither. Several cheaper options come first:

• Put it in the prompt. If the content is small (tens of pages), stable and visible to every user, include it directly. With large context windows, that’s simpler than a pipeline and can’t miss a passage.
• Query a database. “Orders over $500 last month” is a SQL query behind a tool, not a semantic search.
• Call an API. Live data such as an account balance belongs to a tool call; an index would be stale.
• Write a better prompt. Most behavior problems yield to clearer instructions and two or three examples.

Long context has limits worth knowing for the follow-up question. Stuffing a large corpus into every request costs tokens on every call, adds latency, raises the chance of contradictory passages, and models can miss material placed in the middle of a long prompt (lost in the middle). Selection usually beats volume.

A decision rule you can apply in a minute

1. Is the gap knowledge (missing, private, stale, needs a citation)? Go to step 2. Is it behavior (format, tone, task pattern)? Go to step 4.
2. Does it fit in the prompt, stay stable and apply to every user? Put it in the prompt.
3. Otherwise: structured data goes to a database query, live data to a tool, and searchable documents to RAG.
4. Fix the prompt first. Fine-tune when a well-built prompt with examples still misses your quality bar on an eval set, and you have enough good examples.
5. Need both? Combine them, and evaluate retrieval and generation separately so you know which half to fix.

How to answer it in an interview

Interviewers ask this to see whether you understand where knowledge lives in a model, and whether you’ll talk a team out of an expensive mistake. A strong answer leads with the distinction, gives an example, names a tradeoff and mentions evaluation. Here is a version you can say in about a minute:

“They fix different problems. RAG supplies facts at request time: you retrieve relevant passages and put them in the context, so the knowledge is current, citable and filtered by what each user may see. Fine-tuning changes behavior in the weights, like format, tone or a task pattern. It’s a poor way to add facts, because the model still recalls specifics unreliably, can’t cite, and every update is a training run.

So I sort the request first. If someone says ‘fine-tune it on our docs so it knows our policies,’ that’s a facts problem, and I’d build RAG with citations and an abstain instruction. If the answers are right but in the wrong format, I’d fix the prompt and only fine-tune if that plateaus on our eval set. Many production systems use both: a tuned model answering from retrieved documents. And if the docs are small and stable, I’d skip both and put them in the prompt.”

That’s about 150 words. Expect a follow-up such as “when would you fine-tune anyway?” (consistent output format at high volume, a narrow, stable task where a smaller tuned model can beat a larger general one on cost and latency, or a style that examples in the prompt can’t hold) or “how would you evaluate the RAG version?” (retrieval recall on labeled queries, whether each cited passage supports its claim, and abstention on questions the corpus can’t answer).

Practice next

• RAG interview questions: pipeline, chunking, retrieval and evaluation questions with model answers.
• AI system design interview: where this choice shows up inside a full design round.
• 50 LLM interview questions: the free set covering tokens, context, training, RAG and agents.