Fine-tuning, retrieval or a better prompt: how to choose
Three ways to make a model behave the way you need, with very different costs and lifespans. A decision guide based on what is actually wrong, rather than on which technique sounds most serious.
Kurze Antwort
Should I fine-tune a model, use RAG, or improve my prompt?
Diagnose the failure first. If the model does not know something, that is a knowledge gap and retrieval fixes it. If the model knows but answers in the wrong shape, tone or format, that is a behaviour gap and prompting fixes it first, fine-tuning second. If the model fails at a specialised skill after both, fine-tuning is the remaining option. Prompting is cheapest and reversible, retrieval is the right default for facts, and fine-tuning is the most expensive and least reversible of the three.
Das Wichtigste
- The three techniques address different failures. Picking one before diagnosing the failure is the most common and most expensive mistake.
- Fine-tuning does not reliably add facts. It shapes behaviour — format, tone, style, a narrow skill — and facts learned this way are hard to update and hard to cite.
- Retrieval is the correct default for anything that changes, anything that needs a citation, and anything subject to per-user permissions.
- Prompting is not the junior option. A well-structured prompt with examples resolves a large share of problems that teams try to solve with training.
- Fine-tuning has a maintenance cost that estimates rarely include: every base model upgrade re-opens the question.
A model is not doing what you need. Three techniques are on the table, and the decision is usually made backwards — someone picks the approach first, then constructs a justification. Fine-tuning sounds the most substantial, so it gets proposed most often and works least often.
The reliable order is the other way round. Diagnose the failure, and the failure names the technique.
Three failures, three tools
Almost every complaint about model behaviour is one of three things.
A knowledge gap. The model does not know something — your product catalogue, last week's policy change, this customer's order history. It cannot know it, because it was not in training and is not in the prompt.
A behaviour gap. The model knows enough but responds in the wrong shape: too long, wrong tone, missing a required section, using terms your industry does not use, answering when it should decline.
A skill gap. The model performs a specialised task poorly even with the right information in front of it and clear instructions — a domain classification with subtle boundaries, a niche formal notation, a judgement that depends on conventions not well represented on the public web.
Why fine-tuning does not solve knowledge gaps
This is the most consequential misunderstanding in the area, so it is worth being precise.
Fine-tuning adjusts weights on example pairs. Facts contained in those examples do become more likely to appear in output — that part is real. What does not follow is any of the properties you need from a knowledge system:
- You cannot enumerate what was learned. There is no query that returns which facts made it in.
- You cannot update one fact. Correcting a single price means assembling a new dataset and training again.
- The model cannot cite. It produces the fact with no traceable source, which is unusable in any regulated or auditable context.
- Permissions cannot be applied. A model trained on all customers' data will answer about all customers' data, to anyone.
- Confident errors get worse, not better. Fine-tuning on facts tends to increase fluency and confidence about a domain without proportionally increasing accuracy, which is the least useful combination available.
Retrieval has the inverse profile on every line. Change the document, and the next answer changes. Every passage has an id. Permissions are a filter applied before the model sees anything.
Why prompting is underrated
Teams often treat prompting as the beginner's option that serious systems graduate from. In practice a large share of behaviour problems are solved there, and the reason is that most weak prompts fail in the same three ways.
No examples. Two or three demonstrations of the desired output shape are worth more than several paragraphs describing it. Show, do not specify.
No structure. Instructions buried in prose get followed unevenly. Sections, numbered constraints and an explicit output format are followed far more reliably.
No permission to decline. A prompt that never states what to do when the answer is unknown is implicitly demanding an answer. One sentence — "if the documents do not contain the answer, say so and stop" — removes an entire class of fabrication.
Prompting also has a property the other two lack: it is reversible in seconds. While requirements are still moving, that is worth more than any efficiency gain.
The honest limit is cost and consistency. A 2,000-token system prompt is billed on every request, and very long instruction sets get followed unevenly. When a behaviour is settled and the volume is high, moving it into weights is the correct optimisation — but that is an efficiency decision made after the behaviour is known, not a way of discovering what the behaviour should be.
What fine-tuning is genuinely for
Three cases justify it clearly.
Format and style consistency at volume. When every response must follow a house structure and prompting gets you to 95%, training on a few hundred consistent examples can get you to a much higher number while removing the instruction tokens from every request.
Narrow skill acquisition. A classification with subtle domain boundaries, an internal notation, a house convention that is not represented on the public web. This is the only one of the three techniques that changes what the model can do rather than what it is told to do.
Making a small model sufficient. This is the most economically interesting case. A fine-tuned small model that matches a frontier model on one narrow task can cut cost and latency substantially, and it is often the mechanism behind a viable self-hosted deployment.
Parameter-efficient methods — training a small adapter rather than the full network — have made all three far cheaper than they were, which shifts the calculus but does not change the diagnosis. A cheap solution to the wrong problem is still the wrong solution.
The costs estimates leave out
Dataset assembly. Several hundred to a few thousand consistent, correct examples, which usually means labelling work. Contradictory examples are worse than no examples — they teach the model to be inconsistent.
Evaluation. You need a held-out set and a way to compare before and after, or you have no way to know whether the tuned model is better. This is the step most often skipped and the one that makes the difference detectable.
The base model treadmill. Every base model upgrade re-opens the question: retrain on the new base, or stay on the old one and forgo the improvement. This recurs indefinitely, and prompting does not have it — a better base model simply arrives.
Regression risk. Training for one behaviour can degrade others. Without a broad eval set, this is discovered by users.
The order that works
- Write down the failure in a sentence, and classify it: knowledge, behaviour, or skill.
- Build the eval set first, from real failing cases. Without it, none of the three can be judged.
- Fix the prompt. Add examples, structure, and an explicit way to decline. Measure.
- Add retrieval if the failure is factual. Verify retrieval separately from generation.
- Fine-tune only if a behaviour is settled and expensive to prompt, or a genuine skill gap survives steps 3 and 4.
Most teams that follow this order stop at step 3 or 4. That is not a failure of ambition — it is the correct outcome, and the months not spent building a training pipeline are the return on doing the diagnosis first.
Häufige Fragen
- Can fine-tuning teach a model my company's data?
- Only unreliably. Training will make some of it more likely to appear, but you cannot query which facts were learned, you cannot update one without retraining, and the model cannot cite a source. For company knowledge, retrieval is the correct tool.
- What is fine-tuning genuinely good at?
- Making a model consistently produce a specific format, adopt a house style, follow a domain's conventions, or perform a narrow classification well enough that a smaller and cheaper model can replace a larger one.
- How many examples does fine-tuning need?
- Far fewer than people expect for style and format work — a few hundred well-chosen, consistent examples often suffice. Consistency matters more than volume; contradictory examples teach the model to be inconsistent.
- Is a long system prompt worse than fine-tuning?
- It costs tokens on every request, which matters at scale, and very long instructions get followed unevenly. But it is instantly editable and instantly reversible, which is worth a great deal while requirements are still moving.
- Can I combine them?
- Yes, and mature systems usually do: a fine-tuned model that reliably produces the right format, retrieval supplying current facts, and a compact prompt handling the request-specific instructions.