Prompt, retrieve, or fine-tune?

Someone fine-tunes a 7B or 13B model on the company wiki, ships it, and weeks later is debugging why the assistant quotes a pricing page that changed after the training run finished. The facts got baked into weights nobody can patch without training again.

The unit that matters is not the model alone, it's a completed task on a machine that stays responsive: evidence goes in, an answer comes out, a quality gate passes it. A lot of expensive fine-tuning is an attempt to store knowledge that belongs in an index or a tool instead.

Format, facts, and tone are three different bugs

Take a local app that fails three ways at once: it ignores the output schema, gets domain facts wrong, and answers in a tone nobody asked for. Those are three separate failures with three separate fixes, not one bad model, and conflating them is expensive. A wrong schema is usually an unclear instruction, fixed with a better prompt or template. Wrong facts mean missing evidence: the model never had the current answer in context, so retrieval fixes it, not more training. Wrong tone is learned behavior, the kind of pattern that sticks only after repeated exposure, the one case fine-tuning earns its cost.

Before touching any of the three, decide what you're trying to move: first-token latency, accepted jobs per hour, models kept resident, energy per finished task, or how often a human corrects the output by hand. Make it faster protects nothing and never tells you when you're done.

Freeze the rig before you start the clock

You need a baseline you can reproduce: pin the model artifact, tokenizer, prompt template, runtime build, launch command, and sampling settings, tested on real workload inputs, including the awkward edge cases. Run once cold, storage untouched and model unloaded, if users hit that path, then run warm long enough to expose cache effects, queueing, memory pressure, and thermal throttling.

Time the phases separately instead of trusting one total number:

• queue wait
• model load or activation
• prompt processing and time to first token
• decode rate and completion time
• peak RAM, VRAM, and swap
• whether the quality gate passed, retried, or had to abstain

Raw tokens per second is a fine diagnostic and a bad product metric. For automation, count valid completed records per hour. For coding assistance, count review and correction time. For chat, look at p50 and p95 time-to-first-token across realistic conversation lengths. A short warm run at batch size one can win the benchmark and lose the day once model swaps and long prompts show up.

Nothing crashes, so nothing looks wrong

The mistake from the opening recurs most: fine-tune on a document set, then act surprised when facts go stale the moment those documents change. That's a retrieval problem wearing a training-shaped disguise, and no amount of further fine-tuning fixes it: the model can only be as current as its last training run.

Local runtimes make this worse by being resilient. They offload layers, page memory, silently miss a cache, queue a request, or fall back to a slower generic kernel instead of failing loudly, so a badly configured setup just looks slow, not broken. Check startup logs, device placement, and OS-level counters, and confirm the requested optimization is active for the tensor shapes and context length you're running.

Change one variable at a time unless comparing two whole systems, and keep the outputs, not just the metrics: quantization, context compression, different sampling, or a model swap can make an answer faster while making it wrong, so run the quality gate on every candidate. A shorter answer needs a completeness check, not applause, and a faster extractor needs a count of valid records, not a count of strings that merely parse.

Weigh operational cost too: startup, upgrades, observability, rollback, rebuilding the box after a disk dies. A five percent win from an undocumented patch, or a cache warmed by hand every morning, is a bad trade for anything shared. Boring configurations age well.

The cheapest fix first, in writing, with an expiry date

The rule I use is plain: apply the cheapest reversible fix that targets the real failure mechanism, and only move up the cost ladder once the cheaper thing has failed. Prompt first, retrieval second, fine-tuning last, saved for behavior a document store cannot represent. Doing that on your own hardware changes the cost calculus, worth its own look in a piece on fine-tuning at home.

Write the conclusion down next to the workload, the date, and the reasoning, and write down what should trigger a retest: new model family, driver update, longer contexts, different traffic mix. Skip that and your benchmark calcifies into infrastructure folklore nobody can defend or wants to re-run. The retrieval half needs the same discipline: an index isn't a one-time ingest job, it needs an ongoing maintenance story.

Leave headroom once you hit the target instead of packing every resource to the ceiling. Free memory absorbs prompt variance you didn't test for. Spare queue capacity keeps a batch job off an interactive user. Thermal and power margin lets the box run for a week, not an afternoon. None of that shows up on a benchmark chart, and all of it is the difference between a demo and a service.

What I still don't have a clean answer for is the boundary case: a model that has to describe an evolving product line in a consistent house voice while the product line itself keeps changing. Retrieval handles the facts, fine-tuning would hold the voice, and gluing the two together without one overriding the other is still more trial and error than method.