Fine-tuning vs RAG vs prompting: the decision, and the honest costs
"Should we fine-tune?" is the most over-asked question in applied AI, usually asked by a team that actually needs RAG, or just a better prompt. The three techniques solve genuinely different problems, and reaching for the wrong one wastes months. Here's the decision, plain, with the costs nobody puts on the slide.
The decision in one table
| You need to… | Reach for | Why |
|---|---|---|
| Inject knowledge that changes | RAG | You edit a document, not a model |
| Teach a behavior / format / style | Fine-tune | Bake in how to respond |
| Just get better instructions / examples | Prompting | Most problems. Try this first. |
| Reason over one big artifact at once | Long context | No retrieval needed if it fits |
They compose: a fine-tuned model with RAG and a good prompt is common. But knowing which knob your problem turns is the difference between a week and a quarter.
The rule that prevents most mistakes
Fine-tuning teaches behavior. RAG injects knowledge. Confuse the two and you'll fine-tune a model to "know" facts, then get a confident, expensive, stale hallucination machine.
If your problem is "the model doesn't know our products / docs / current data," that's RAG (or long context), full stop. Fine-tuning facts in is the classic, costly error: the facts go stale, the model blurs them, and you can't update without retraining. Knowledge lives in documents you can edit; behavior lives in weights you train.
Why fine-tuning is rarely the first answer
- The cost is the data, and the data is expensive. Curating a high-quality training set is the real work, far more than the training run itself. Then you own a model you must re-create every time the base model improves.
- Base models moved faster than fine-tunes paid back. Through 2024–2026, many carefully fine-tuned models were obsoleted by the next base release before they recouped their cost. A better prompt on a newer base often beat last quarter's fine-tune.
- Prompting got further than people expect. A clear instruction plus a few good examples (few-shot) solves a surprising amount of what teams reach to fine-tune for, at zero training cost and with instant iteration. Try it first, always.
When fine-tuning genuinely pays
It's the right tool. Just rarely the first one:
- A narrow, stable, high-volume task where you can distill a big model into a small cheap one. This is the strongest case: use a frontier model to generate training data, fine-tune a small/local model to do that one job at a fraction of the cost and latency. The economics are real at volume.
- A consistent format or behavior that prompting can't reliably pin down: a rigid output structure, a specific domain voice, a skill the base model genuinely lacks.
- On-device / local: a small model fine-tuned for a specific job, runnable on your own hardware. LoRA/QLoRA make this cheap enough to do overnight on a single machine.
Distillation: the underrated middle path
The move worth knowing is distillation. Let a big model generate the training data for a task, then fine-tune a small model to replicate it. You get the small model's cost and speed with much of the big model's quality on that narrow task. For the local tier of a cascade (where you want the free local model to clear as much as possible), a distilled specialist can widen what local handles by a lot.
The lazy ladder
Climb only as far as the problem forces you:
- Prompt + examples. Free, instant. Solves most things. Always first.
- RAG (or long context) when the issue is knowledge.
- Fine-tune when behavior/format/cost provably can't be solved above, and prefer distillation into a small model when the goal is cost.
That order is YAGNI for model customization: the expensive, high-maintenance option is the last resort, not the first instinct, and most teams never need to reach it.