Commercial vs free models for RAG systems: privacy and data control

Somewhere a team wired their retrieval pipeline into a hosted model and later found out every query, contract text included, had been sitting in the provider's debug logs the whole time. Nobody stole anything. Nobody even looked at it, probably. It just sat there, outside the retention window security review assumed, because "call the API" and "send our data somewhere else" turned out to be the same action, and nobody said so in the design doc.

That's the failure this piece tries to head off, and why "commercial versus free" is the wrong axis to plan around. Under that phrase sit three arrangements: a paid API bundling inference with a service, support, uptime, someone to call; a free hosted tier, usually a smaller version of that same service; and an open-weight model, whose cost just moves into hardware, electricity, and someone's time planning capacity.

the one failure that actually matters in a RAG answer

For a retrieval-augmented system the unit that matters is one answer, supported by the passages pulled back: a question, ranked passages, metadata, a citation format. The failure isn't a crash. It's a fluent paragraph that ignores the evidence, merges passages that don't agree, or invents a fact that reads like the real ones next to it.

The cheapest model per token isn't the cheapest model per answer: a generation that fails validation or eats ten reviewer-minutes already costs more than the pricier call that nailed the citation first time. Privacy is wider than "where does inference run": collection, transit, retention, who sees the logs, backups, whatever tool reads the output next. One side is a local open-weight generator with local embeddings; the other a commercial API, maybe with managed search bolted on. Use real model names and prices in the experiment, but keep them out of the permanent rule: catalogs change, quants improve.

run the same test twice and write down what moves

A fair experiment maps every copy the data makes and every operator who can see it, checks retention and residency against what's actually configured, and strips sensitive fields before anyone argues about accuracy. Give both routes the same evidence and acceptance bar, but let each keep its own prompt template and API. If one side ships a native tool the other has to hand-build, count that gap, don't average it away.

route + exact model revision
input / cached input / reasoning / output tokens
queue time / first token / completion / timeout
hardware energy draw and idle allocation
outcome: pass / retry / abstain / human repair
engineering and incident time spent
privacy, license, and fallback constraints

The quality gate is answer-bearing retrieval, citation correctness, willingness to abstain, and latency. Every failed output gets kept and tagged with why:

• missing evidence
• instruction not followed
• malformed output structure
• reasoning that doesn't hold up
• tool or integration error
• an unsafe action taken

Those tags make routing defensible later instead of a hunch. A local model often nails the easy cases and falls over once context or ambiguity crosses a threshold you can find in the tagged failures.

when to pay someone else and when to run it yourself

The commercial side wins when contract terms, certifications, residency, and managed security clear a bar the local setup can't reach alone, not because that API is the most capable this week. Put a cheaper commercial tier into the comparison and cap output length and reasoning effort. Managed capacity earns its cost when it removes work the team would otherwise do by hand; a dashboard is not a reason.

The local, open-weight side wins when data has to stay on hardware you control and the org can secure the whole lifecycle, not just inference. Its advantage compounds with steady utilization and narrow, repeatable tasks, and shrinks once the model barely fits in memory or depends on a runtime only one engineer understands. Spare hardware isn't free either, if another workload loses its slice while your job runs.

The recurring mistake is calling something private just because the weights run on a box in your rack. Attach every number to a specific workload and time window, not the model in general. Price per million tokens and tokens per second are diagnostics, not outcomes, the same trap as in subscription-wrong-for-ai once tokens become seats. Report cost per accepted task, p95 latency, failure severity, and how much human attention the workflow still eats.

the adapter you wish you'd written first

Put every provider-specific request and response shape behind one narrow adapter, and keep prompts and schemas in version control, not a vendor console. Keep the authoritative job record outside any thread object living only inside one vendor's system, or that vendor has quietly become an untestable dependency. Contract-test streaming, tool calls, structured output, errors, cancellation, and token accounting for every route you support.

Fallback has to be explicit, not implicit. If the commercial API goes down, a local model can be a reasonable degraded mode, similar to the local-first-cascade pattern, if users know capability changed. If local capacity runs out, quietly routing sensitive data to a remote provider defeats the point of the system. Privacy and authorization rules run before capability routing, always.

Re-run the comparison when traffic, versions, prices, licenses, or review practice change: a route right at ten thousand tasks a month can be wrong at a hundred thousand. A model that choked last year might clear the bar now on a better quant, and a free tier you relied on might vanish.

None of this needs a universal winner, and I stopped looking for one. In practice I run narrow, repeatable retrieval on local weights, and pay full commercial price for the messy, high-stakes queries where a wrong answer is expensive. That means giving up the vendor's support line and its lawyered data-processing agreement on the local half, on purpose, for customer text that never leaves the building. I know what I'm not getting from that trade, and knowing it is the point of choosing it instead of drifting into it.