Commercial vs free models for RAG systems: latency, throughput, and queues

Route by shape, not by price sheet. A resident local model earns its keep on traffic that's steady and narrow, sized to hardware you own. A commercial API earns its keep on bursty or global traffic, where someone else's elastic capacity beats a queue behind your one GPU. Never decide either way off a single tokens-per-second number.

the three products hiding under two labels

"Commercial" and "free" aren't one axis, they're three products. A paid API bundles inference with a service: uptime, an on-call rotation. A free hosted tier is usually that service with a discount and a leash, still commercial, just unbilled to you. An open-weight model is neither: an artifact whose license may or may not be open source, its compute still somebody's expense. Local inference doesn't remove cost, it swaps a token invoice for hardware, electricity, maintenance, and capacity planning.

For a RAG system, the thing you're buying is an answer supported by the retrieved passages, not fluent text in general. Feed it a question, ranked passages, metadata, and citation requirements, and watch for the failure that matters: a confident answer that ignores the evidence, merges two sources, or invents a citation that reads fine and is wrong. A cheap generation that fails validation and burns a retry, or ten minutes of reviewer time, can cost more than the pricier call that finishes once.

measure the whole path, not the token price

The comparison sits on five separate properties: cold start, first token, sustained generation, concurrency, and tail latency. Replay the real request distribution against both routes and log p50, p95, queue delay, first token, completion time, and timeout rate for each, respecting each side's own prompt templates. Run it on real model IDs and today's prices, then keep those out of the permanent rule, because catalogs change.

tokens per second is a spec-sheet number; cost per accepted task is the one that survives contact with a budget

The quality gate is answer-bearing retrieval, citation correctness, willingness to abstain, and end-to-end latency, taken together. Keep every failed output and label why: missing evidence, an ignored instruction, malformed structure, bad reasoning, a tool error, an unsafe action. A local model often clears the easy cases and fails only once ambiguity crosses a threshold those labels finally name.

That's the shape: commercial for the bursty or global load, a cheap capped-output tier before the biggest model on the list; local for the narrow, steady work a resident model clears without a queue, an edge that shrinks once the model barely fits or needs a runtime only one person understands. The recurring mistake is quoting warm batch-one speed against an API average and calling it latency. What survives is cost per accepted task and p95 completion latency.

build the off-ramp before you need it

Put provider-specific handling behind one adapter: prompts and schemas in version control, job state somewhere that isn't a vendor's thread object. Contract-test streaming, tool calls, structured output, errors, cancellation, and token accounting on every route, because the untested path breaks mid-incident.

Fallback has to be explicit. A local model stepping in when the commercial API goes down works as a degraded mode, as long as users know capability changed. Quietly routing to a remote provider when local capacity runs out is different: it breaks the system's privacy promise, so that rule runs before capability routing gets a vote.

Re-run the comparison when traffic, model versions, prices, licenses, hardware utilization, or review practice shift underneath you. What was right at ten thousand tasks a month can be wrong at a hundred thousand, and a model that failed last year may pass today on a better quant while a free tier just disappears. None of this needs a universal winner: the durable setup is the smallest route set that clears the gate, keeps cost and data movement visible, and fails safely. "Free" and "commercial" are procurement labels; reliable completed work is the engineering metric.

I'd be lying if I said the adapter-and-contract-test version is what I'd actually build for a side project doing a few hundred queries a day. At that scale, pick whichever route is already wired up, ship it, and save the off-ramp for after a vendor change has burned you once.