Commercial vs free models for RAG systems: the real cost per completed task
You are pricing a RAG pipeline. Retrieval works, evaluation passes, and now someone wants a decision: route generation to a paid API, or run a local 14B-class GGUF and call it free. Neither half of that sentence means what it sounds like.
same label, three different bills
A paid API is inference bundled with a service: rate limits, uptime, someone else's GPUs. A free hosted tier is a scaled-down version of that service, not a gift. An open-weight model is neither: an artifact, sometimes open source, sometimes not, and the compute to run it is somebody's line item. Local inference doesn't remove cost. It swaps a token invoice for hardware, electricity, maintenance, and capacity planning.
For RAG, what you're buying is an answer the source passages can support. Feed it a question, ranked passages, metadata, citation rules; the characteristic failure isn't silence. It's a fluent answer that ignores, merges, or invents evidence. A cheap generation that fails validation and eats ten reviewer-minutes can cost more than the pricier call that lands first try.
count the whole path, not the sticker
Total cost is inference, idle hardware, engineering, retries, validation, and human review; drop any one and the comparison lies to you. Run both routes, commercial API against local model plus local embeddings, on the same evidence and acceptance bar, and log this per accepted answer:
| track | tells you |
|---|---|
| route + model revision | what produced the answer |
| token/energy, queue to completion | unit cost and latency |
| pass / retry / abstain / repair | where failures land |
| engineering + incident time | cost nobody invoices |
Tag every failure by mechanism: missing evidence, bad instruction-following, malformed output, wrong reasoning. Those labels make routing possible later, instead of guessing. The local model often nails the easy majority and only falls over past some context or ambiguity threshold. That's a routing decision, not a verdict on the model.
pick a lane, but leave the door unlocked
Commercial wins when traffic is spiky and a bad answer is expensive, not because a bigger model is available. Put a cheaper commercial tier with capped output in the mix. Local wins when volume is steady, the task fits a smaller model, and you have hardware and staff to run it. That advantage evaporates the moment the model barely fits or needs a babysitter. The classic error is dividing a GPU price tag by theoretical lifetime tokens. Anchor every number to a workload and a time window instead, same discipline as 99-percent-cost-architecture.
Put provider-specific request handling behind one adapter, keep prompts in version control, and make the fallback explicit: degrading to local should never mean shipping sensitive data to a remote endpoint unannounced, the exact failure mode local-first-cascade is built to catch.
Before committing to either lane, pull last month's traffic shape and check whether it's still steady enough to justify running local. That number goes stale first.