Retrieval adds knowledge at question time and can cite it. Fine-tuning changes the model’s weights and is best at shaping how it behaves.
Teams often reach for fine-tuning to teach a model their documents — the job retrieval does better — and overlook it for the job it does well: holding a format, a tone or a narrow task. This guide shows what each technique changes, what controlled studies found, how the costs play out over a system’s life, and a decision tree that starts from the problem you actually have.
Updated · 6 sources
Retrieval augmented generation searches your sources at question time and puts the relevant passages in the prompt.
Further training on your examples adjusts the model itself, so it responds in a new way without being told each time.
RAG (retrieval augmented generation) leaves the model unchanged and gives it relevant passages from your sources with each question. Fine-tuning trains the model further on your examples so its weights change. In RAG vs fine-tuning, the rule of thumb is: retrieval for knowledge that must be current, cited and permissioned; fine-tuning for behaviour — format, tone, a narrow task — that prompting cannot hold. Many systems use both.
Every answer a model gives is shaped by four things, top to bottom. Prompt engineering and RAG work on the upper layers at question time; fine-tuning rewrites the bottom one ahead of time.
Parametric and non-parametric memory. The 2020 paper that introduced RAG framed it exactly this way: a model’s weights are parametric memory, an external searchable index is non-parametric memory, and RAG combines the two. Fine-tuning writes to the first; retrieval reads from the second.
Two Microsoft research papers from 2023 and 2024 tested the question directly, and an earlier one explains why fine-tuning became cheap enough to try.
The pattern across studies: fine-tuning is a weak way to add facts — Ovadia and colleagues found models learned new facts better when training showed many variations of the same fact — but a good way to adapt a model to a domain’s language and tasks. The agriculture study found the two gains add up, which is why the decision is rarely either-or.
Name what is wrong with today’s answers before choosing a technique. Each symptom points to a different fix — and the first fix is almost always a better prompt.
What is wrong with the answers today?
The first version is the cheap part. The difference shows when a policy changes, a person leaves, an auditor asks for a source, or the base model is retired.
Fine-tuned models follow their base model. OpenAI’s model-optimization guide states that fine-tuned models stay available for inference until their base models are deprecated — so a fine-tune is a recurring project tied to the provider’s model schedule, rather than a one-off cost.
RAFT (retrieval augmented fine-tuning, 2024) trains a model in the “open book” setting it will face in production: a question, some passages that help and some that do not, and an answer that quotes the right one.
Real questions from your domain, each with the document that answers it.
Pair each question with passages that look relevant but do not answer it.
Target answers quote the supporting passage verbatim.
Often with an adapter such as LoRA, so training stays affordable.
The tuned model reads live retrieved passages; facts still come from the index.
Score against the untuned model with the same retrieval, on the same questions.
On Botminds, enterprise knowledge lives in knowledge collections that agents search — so it stays current, permissioned and citable — and the model behind each agent is a setting.
Read agentic RAG and the LLM gateway, and see indexing in the Academy lesson search indexes and vectorization.
Neither in general. In RAG vs fine-tuning, RAG is better for knowledge — facts that change, must be cited or are restricted by person. Fine-tuning is better for behaviour — output format, tone, a narrow task, or moving that task to a smaller model. Studies find the gains from each can add up.
When the problem is how the model responds rather than what it knows: a strict output format it keeps breaking, a house style, a specialised classification, or a high-volume task you want a smaller, cheaper model to handle. Try better instructions and examples first.
Poorly. A 2023 Microsoft study found RAG consistently outperformed unsupervised fine-tuning for knowledge, and that models learned new facts better when training showed many variations of each fact. Fine-tuned facts also cannot be cited or updated without retraining.
Prompt engineering changes the instructions, RAG changes the context, fine-tuning changes the weights. Start with the prompt, add retrieval when the model lacks knowledge, and fine-tune only when behaviour still falls short at volume.
Low-Rank Adaptation trains a small set of extra weights beside a frozen model instead of updating every weight. Its authors report 10,000 times fewer trainable parameters and 3 times less GPU memory than full fine-tuning of GPT-3 175B, with no added inference latency.
Upfront, usually: RAG needs an indexing pipeline, fine-tuning needs labelled data and training runs. Per call, RAG prompts are longer because they carry passages. Over time, fine-tunes must be redone when facts change or the base model is retired.
Retrieval augmented fine-tuning: training a model on questions paired with helpful and distracting passages, with answers that quote the right passage verbatim. It prepares a model to use retrieval well in a specific domain.
Knowledge on Botminds lives in knowledge collections that agents search, so answers stay current, permissioned and cited. Your documents never train shared or third-party models. Agents can use any model added to the gateway, including models you host.
Bring one process and its documents. We’ll show you the agents, the controls and the record, running on your data.