Commercial vs free models for RAG systems: long-context economics

So you've decided a bigger context window is the safe default, but "free versus commercial" hides four things behind two labels. A paid API bundles inference with a service you don't run. A free hosted tier is a commercial product with a leash on it. An open-weight model is an artifact whose license may not be open source, and whose compute is still someone's expense; local inference swaps that invoice for hardware, electricity, and unbudgeted maintenance.

the unit that actually matters

For a RAG system the unit worth measuring isn't tokens accepted, it's an answer the retrieved passages support: a question, ranked passages, metadata, a citation rule. The failure that matters: a fluent answer that ignores, merges, or invents evidence. A cheap model that does that and triggers a retry or ten reviewer minutes ends up pricier than the call that finished it once. Evidence density beats maximum context, every time. Compare a local generator plus local embeddings against a commercial API, model IDs and prices staying inside the experiment, never the rule. Bucket prompts by length, weigh prefill against answer quality, and pit retrieval-plus-compression against full-context stuffing on each provider's template; count native-tool gaps as cost too.

per task, not per token:
  route + exact model revision
  input / cached / reasoning / output
  queue, first token, completion, timeout
  hardware energy + idle allocation
  outcome: pass / retry / abstain / human fix
  engineering, privacy, license, fallback

Grade every run on one gate: answer-bearing retrieval, correct citations, abstention, tail latency. Tag every failure: missing evidence, bad instructions, bad structure, tool error, unsafe action. A small local model can be flawless until context or ambiguity crosses a real threshold.

swap cost is the real variable

Commercial earns its keep when a task needs a window you'd never keep warm yourself, so managed capacity replaces headcount, not a dashboard; test a cheaper tier in the same run. Local wins on routine, repeatable prompts that fit a controlled window after retrieval and stable-prefix reuse (the local-first cascade logic), and that edge shrinks once the model barely fits, needs constant babysitting, or borrows hardware another job needed. Stop feeding it every document because the window accepts them: price per token and tokens per second are diagnostics, not outcomes, so track cost per accepted task, p95 latency, and attention spent. Put both routes behind one adapter, version-control prompts and schemas, keep job state outside vendor threads, and contract-test streaming and token accounting so swapping vendors is a config change. Fallback stays explicit: degraded local is fine once users know capability dropped, but an exhausted queue must never ship sensitive data to a remote API. Privacy rules run first. Full stop. Re-run this whenever traffic, versions, prices, or licenses move: right at ten thousand tasks a month can be wrong at a hundred thousand. No universal winner: the durable choice is the smallest route set that clears the gate, keeps cost and data movement visible, and fails safely; "free" and "commercial" are procurement labels, not engineering ones.

What I'd check next: pull last month's retrieval logs, bucket them by prompt length, and see how many fit a local window after compression, before renewing anything.