The question is usually asked backwards
"Should we fine-tune a model on our data, or do RAG?" is the second question in most enterprise AI conversations, and it is asked as if the two were rival ways to do the same thing. They are not. They change different parts of the system, they fail in different ways, and they answer different questions. Asked the right way round — what has to be true about the answers this system gives, and what does the model actually lack — the choice is usually obvious within an hour, and it is usually retrieval, sometimes both, occasionally a fine-tune, and more often than people expect, neither.
This is the reasoning we use in the first week of a RAG development or private LLM engagement, before anything is built. It is written for the people who have to decide — the head of technology, the compliance lead, the product owner — rather than for the engineer who already knows what LoRA is.
What each one actually changes
Retrieval-augmented generation leaves the model alone and changes what it sees. When a user asks a question, the system searches your own content — policies, contracts, manuals, tickets, circulars — for the passages most likely to contain the answer, hands those passages to the model with the question, and has the model answer from them, citing where each point came from. The model's job shrinks from "know everything" to "read these pages and answer", which is a job current models do very well. The knowledge lives in your documents, updates the moment a document changes, and can be shown to the user as a citation. The original RAG paper framed it as a way to give a model non-parametric memory; in practice it is search engineering with a language model on the end.
Fine-tuning changes the model. You take a base model — for enterprise work usually an open-weight one such as Llama, Qwen or Mistral — and continue training it on examples you curated, so that its weights shift towards the behaviour those examples demonstrate. Modern fine-tuning almost never touches the whole model: LoRA and QLoRA train a small set of adapter weights alongside the frozen base, so a fine-tune is a file you ship next to the model rather than a new model, and the base can be upgraded later with the adapter re-trained. What a fine-tune is good at is behaviour: producing a format consistently, adopting a tone or house style, handling domain vocabulary the base model mangles, or running a narrow task well at a smaller size than prompting would allow. What it is poor at is adding facts, and it is worse at keeping them current.
There is a third option that the question usually skips: prompting and context. Current models take long inputs, and if the material a question depends on is a handful of documents, the cheapest correct system is one that sends them. No index, no training. It is worth naming because a surprising number of "we need RAG" and "we need to fine-tune" requests are solved by it.
The five questions that decide it
1. Is the gap knowledge or behaviour?
Write down three answers the system got wrong in a prototype, or three it would have to get right in production, and ask why the base model cannot produce them. If the reason is "it does not know our policy / our product / this year's circular", the gap is knowledge, and knowledge belongs in retrieval, where it can be cited, permissioned and updated. If the reason is "it knows, but it answers in the wrong shape / the wrong tone / mangles our terminology / will not reliably fill our schema", the gap is behaviour, and behaviour is what a fine-tune fixes. Most enterprise assistants have a knowledge gap and a small behaviour gap, in that order, which is why most of them end up as RAG at the core with, at most, a small adapter on top.
2. How often does the truth change?
A fine-tune captures the world as it was on the day the training set was frozen. A regulator's circular published next Tuesday, a price list revised on the first of the month, a policy amended after an audit — none of those reach a fine-tuned model until someone curates new examples, retrains, re-evaluates and redeploys. A RAG system sees the new document on its next sync. If the material behind the answers changes weekly or monthly, that alone decides it. If it is stable for years — a coding convention, a document template, a classification scheme — a fine-tune is at least admissible.
3. Does the answer need a citation, and a permission check?
The first question a bank, hospital or government department asks about an assistant is not accuracy but leakage: will it quote an HR document to someone who cannot open it? A fine-tuned model cannot answer that question, because once a fact is in the weights it is available to every user of the model and there is no document to check a permission against. Retrieval can: each document's access list is stored with its chunks, the user's groups are resolved from the identity provider at question time, and the index is filtered before the model sees anything. The same structure is what makes citation possible — the answer points to the passage it came from, so a human check is a glance rather than a search. The data and permissions engineering underneath is familiar work: Engineers India Limited's repository of 200,000+ engineering documents, with metadata classification, role-based access aligned to project hierarchies and Elasticsearch search that took retrieval from hours to seconds, is exactly the kind of corpus a RAG assistant sits on, and the metadata and access model are most of the work. If citations or per-document permissions are a requirement, fine-tuning is not the tool for the facts.
4. What does failure look like, and can you find it?
When a RAG assistant gives a wrong answer, the model is usually blameless: the right passage was never retrieved, or it was retrieved with six irrelevant ones and lost in the middle. That is a diagnosable failure — every question, retrieved passage, prompt and answer is traced, so a bad answer can be tracked to a parsing gap, a retrieval miss, a permission filter or the model, and fixed at that stage. When a fine-tuned model gives a wrong answer, there is nothing to trace; the fact was wrong in the weights, or the training set taught it a pattern that does not hold, and the fix is another training run. For anything a compliance team has to sign off, the ability to explain a wrong answer is worth more than a few points of benchmark score.
5. What do you have to train on, and where may it run?
A fine-tune needs examples, and not many people have them. A few thousand high-quality, reviewed examples of the behaviour you want beat a hundred thousand noisy ones, and "high-quality" means an expert has read them. If what you have is a document repository rather than a set of demonstrated answers, you have the raw material for retrieval, not for training. Then there is where it runs. Fine-tuning an open-weight model means GPUs — the same ones that serve, in off-peak windows, or rented capacity for a day — and it means your training data sits on them; for regulated clients that keeps the whole exercise inside the on-premise or in-country tier they already chose. Retrieval has its own residency question, since embeddings and the index hold your content too, but it can be answered with the same three tiers: your GPUs, your cloud VPC, or an in-region managed endpoint that does not train on your data.
When RAG wins
- The answers must be current. Circulars, policies, product documentation, price lists, support histories — anything that changes faster than you would retrain.
- The answers must be traceable. Regulatory, legal, clinical or customer-facing answers where "where does it say that?" is part of the answer.
- Different users may see different things. Per-document permissions enforced at retrieval time, from the identity provider you already run.
- The corpus is large. Hundreds of thousands of documents do not fit in a context window and should not be baked into weights.
- You need to explain a wrong answer. Tracing from answer to passage is what turns a demo into something a compliance team signs.
When fine-tuning wins
- Consistent structured output. Extraction into your schema, classification into your categories, at a reliability prompting cannot reach.
- Tone and house style. Drafts that sound like your organisation without a page of style instructions on every call.
- Domain vocabulary. Terminology, abbreviations and languages the base model handles badly, where showing it examples fixes it.
- A narrow task at a smaller size. A fine-tuned 8B model doing one job well is cheaper to serve at volume than a 70B model prompted into it, and for a private deployment that difference is measured in GPUs.
- The behaviour is stable and you have the examples. Thousands of reviewed demonstrations, a held-out evaluation set, and no expectation that the underlying truth changes monthly.
When you need both
The most common production shape in our work is retrieval at the core with a small fine-tune around it. Retrieval supplies the facts — cited, permissioned, current — and a LoRA adapter on the serving model fixes the behaviour that prompting could not hold: the exact output schema a downstream system parses, the register a bank wants in customer-facing drafts, the vocabulary of a specialised engineering domain. The adapter is trained on examples that themselves came from a working RAG system, which is the honest way to get training data — it demonstrates the behaviour on real retrieved context rather than on invented questions. The same division holds when the assistant becomes an agent that acts rather than answers: retrieval over policies and past cases supplies the knowledge, and no facts are fine-tuned into the model, so the agent's knowledge can be audited document by document.
The order matters. Retrieval first, measured; prompting tightened, measured; and only then a fine-tune for what is left, measured against the same golden set and against the base model. Fine-tuning early is the most common way to spend a month teaching a model facts that retrieval would have handed it for free.
When you need neither
If the material a question depends on is a handful of documents, send them. If the task is reasoning over one contract, one report or one ticket thread, the model's context window is the retrieval system. And if the corpus is small and well structured, the right retrieval may not be a vector database at all. The assistant on this website is retrieval in its smallest honest form: a scoring function picks the four most relevant case studies for each question, they go into the prompt, and a hosted open model answers from them with instructions not to invent prices or timelines. There is no vector index because, for a corpus of a few hundred documents, a scored keyword search is faster, cheaper and easier to debug. Retrieval should fit the corpus. At the other end of the scale, the same principle says that for large, deeply structured documents the standard chunk-and-embed approach throws away the structure that made the document navigable, which is the argument of our chunkless RAG post — keep the tree, and let the model navigate it.
The costs people forget
RAG is sold as the cheap option and mostly is, but the cost is not in the model. It is in retrieval quality — parsing PDFs with tables and scans so headings and tables survive, choosing chunking per document type, running hybrid vector-plus-keyword search with reranking, and tuning all of it on your data rather than a tutorial's — and in operations: connectors that sync incrementally and honour deletions, re-indexing jobs, an index to run, and per-question token or GPU cost that we estimate during the pilot so the business case is arithmetic. Fine-tuning's cost is front-loaded and easy to underestimate: curating and reviewing the training set is expert time, not engineering time; the held-out evaluation set has to be built before the first run or you cannot tell whether the fine-tune helped; each base-model upgrade means re-training the adapter and re-evaluating; and the GPU hours, while modest for LoRA on a mid-size model, are real. Both share the cost that is never in the first estimate: evaluation. A system without a golden question set is a demo, whichever technique built it.
How we measure the choice
We do not decide this on argument. In the two-week discovery we build a golden set with your subject-matter experts — a few hundred real questions with the documents and passages that answer them, or, for a behaviour problem, a few hundred inputs with the outputs an expert would accept. Retrieval is then scored on its own before a model is connected: is the right passage in the top five, the top ten, at all? Every change to parsing, chunking, embeddings, filters or reranking is measured against that set. Only then is the whole system scored — faithfulness (is every claim supported by a retrieved passage?), answer relevance (does it answer the question?) and context precision (how much of what was retrieved was needed?) — with an LLM judge for scale and human review for the cases that matter. A candidate fine-tune is held to the same standard: evaluated on a held-out set against the base model with the same prompts and, where it exists, the same retrieval, so the improvement it claims is the improvement it delivers. The pilot that follows — four to six weeks on one corpus and one user group — ends with those numbers next to the bar you set, and the production build, typically three to four months to a permissioned system across all sources, is scoped from them. The pipeline is Python, on LangGraph or plain code, and the model is whichever one fits your residency rules: an API model, an in-region managed endpoint, or an open-weight model served on your own GPUs.
The short version
Use retrieval when the gap is knowledge: answers that must be current, cited and permissioned, from a corpus too large or too changeable to bake into a model. Use fine-tuning when the gap is behaviour: a format, a tone, a vocabulary or a narrow task, stable enough to train once and evaluated against a held-out set. Use both when a working retrieval system still cannot hold the output shape or register you need — and train the adapter on that system's own examples. Use neither when the documents fit in the context window or the corpus is small enough that a scored search beats an index. And in every case, decide on a golden set, not in a meeting.
If you want the choice made on your own questions and documents, send us the ten questions your teams ask most, the systems the answers live in and where the data is allowed to run, and we will come back within a week with a discovery proposal that says which of the four it should be. The wider practice — RAG, agents, private deployment and the responsible-AI work around them — is on the enterprise AI services page, and the adapter methods discussed here are documented in the PEFT library, which is what we train with.
