RAG vs Fine-Tuning — The Interview Answer

"When would you use RAG versus fine-tuning?" is a filter question. It separates candidates who have shipped AI features from candidates who have read headlines. You need a 90-second answer with a clean decision rule, honest costs, and a domain example.

The 90-second version

The decision rule: knowledge goes in retrieval; behavior goes in weights.

• If the model needs to know facts — your policy manuals, current Subject Matter Requirements, this candidate's transcript — use RAG: retrieve the relevant text at query time and put it in the prompt. RAG answers are citable (you can show the exact SMR paragraph the match rests on), fresh (update the document store, behavior changes today), and access-controlled (retrieval can respect permissions).
• If the model needs to behave differently — a consistent report format, domain phrasing, a classification style that prompting can't reliably hold — fine-tuning adjusts the weights with examples. It buys tone and format consistency, and sometimes shorter prompts.

The corollary you should say explicitly: volatile or authoritative facts never go into weights. A fine-tuned model can't cite its source, can't be selectively updated when one SMR is revised, and can't forget on demand. For a state agency whose requirements change by regulation, baking them into a model is an auditability and freshness disaster.

Costs and the honest trade-offs

RAG's costs are engineering costs: an ingestion pipeline, chunking, embeddings, a retrieval index, and relevance tuning — plus longer prompts, which means more tokens per call. Fine-tuning's costs are data and lifecycle costs: hundreds-to-thousands of curated examples, training runs to pay for and repeat, evaluation before each rollout, and a custom model to redo every time the base model is deprecated. Prompting alone is cheaper than both — say that you'd exhaust prompt engineering with few-shot examples before either.

And they combine: the strongest setups often retrieve facts with RAG while a light fine-tune (or just rigorous prompting) holds the output format steady.

Anchor it in the transcript service

"Which would you use for our transcript-review system?" — RAG for the SMR text and policy handbooks, because analysts must see cited evidence for every suggested match and the requirements get revised. No fine-tuning at the start; if the standardized report's structure proved hard to hold via prompts and schemas, fine-tuning for format only is the one place it might earn its keep. Facts stay retrievable, citable, and versioned.

Red flags

• "Fine-tune it on all your documents" — treats fine-tuning as a knowledge store; it's a behavior store, and a leaky one.
• No mention of citations or freshness when the domain is regulatory.
• Presenting the two as rivals rather than tools that answer different questions.

Practice prompts

1. Deliver the 90-second answer aloud, with the SMR example, without notes.
2. The interviewer pushes: "Our requirements only change yearly — why not fine-tune?" Respond.
3. Name one thing fine-tuning does that RAG genuinely cannot, and when this agency might need it.