Commercial vs free models for coding assistants: quality ceiling versus sufficient quality
Pick whichever route clears your test suite for the least total cost, and only pay more once you can show, on your own workload, that the extra money buys fewer failures. That's the rule. Stated flat it sounds obvious, and almost nobody applies it consistently: teams pick a vendor once and never revisit it.
one line item, four different products
"Commercial or free" sounds like one axis. It isn't. A paid API sells inference bundled with a service: uptime, support, rate limits someone else tunes. A free hosted tier is usually that same service with the volume dial turned down, not its own category. An open-weight model is an artifact you can inspect and run yourself, and its license may or may not qualify as open source, which matters more than procurement admits. Local inference is the fourth thing hiding under the same two words: you're not avoiding cost, you're converting a token invoice into hardware, electricity, and your own maintenance time. Mix these up and the "quality" gap you're measuring is really a services gap.
score the diff, not the demo
For a coding assistant the only unit that counts is a reviewed change that passes the repository's own test suite, fed by repo context, issue text, tool output, and a diff. The failure that costs money is never the obviously broken patch. It's the plausible one: code that compiles and reads fine locally while quietly breaking a contract nothing in the prompt mentioned. A cheap generation that fails validation and eats ten minutes of review time can cost more than the pricier call that finishes the job once, so judge quality where a human or test suite accepts the output, not by how articulate the model sounds in chat.
Run the free or self-hosted candidate, a local open-weight coder or a capped free tier, against a metered commercial or frontier API on identical evidence and acceptance criteria, respecting each one's own prompt templates. If one side ships a native tool the other must reimplement, count that cost instead of pretending the endpoints are interchangeable. Log the model revision, token counts across input, cache, and reasoning, queue and completion time, hardware energy where applicable, the disposition, pass, retry, abstain, quiet human repair, and the engineering and incident time nobody invoices. Blind-score against tests, static checks, and a real diff review, and label each failure: missing evidence, a misread instruction, malformed output, wrong reasoning, a tool error, something unsafe. Those labels make the routing decision defensible later: a local model can be excellent on most traffic and fall apart only past a threshold you can now name. Use real names and today's prices here, just don't bake them into the architecture: catalogs change, allowances get cut, and mediocre open weights improve on their own schedule.
where the money actually buys something
The commercial route earns its premium when hard cases are common enough that frontier capability measurably cuts severe errors or review time. That's not a license to route everything to the priciest API: put a cheaper commercial tier in the same comparison and cap the reasoning effort you're paying for. Managed capacity earns its cost when it removes work your team would otherwise do by hand, not because someone likes the dashboard.
The free or open-weight route earns its keep when an evaluated model clears the quality gate on most traffic, and that advantage compounds with steady, narrow, repeatable use, the shape local-first-cascade argues for. It evaporates once the model barely fits, reloads constantly, needs daily hand-patching, or depends on a runtime only one person understands. Borrowed GPU time isn't free either if another workload loses access while yours is busy.
The mistake I keep seeing, the same one behind subscription-wrong-for-ai, is paying for a benchmark lead that never changes a production decision. Fix that by attaching every number to a real workload and time period: price per million tokens and tokens per second are diagnostic inputs, not business outcomes. Cost per accepted task, p95 latency, failure severity, and human attention required, that's the real scoreboard.
Free and commercial are procurement labels. A passing diff is the only category that pays rent.
keep the vendor swappable, keep the state yours
None of this matters if you can't act on it. Put every provider's request and response handling behind one narrow adapter, keep prompts and schemas in version control instead of a vendor UI, and keep conversation or job state in your own database, not a proprietary thread object. Contract-test streaming, tool calls, structured output, errors, cancellation, and token accounting for every route, because you want to know before you switch, not after.
Fallback should be a decision, not a default: if the commercial API goes down, dropping to a local model is a fine degraded mode, but tell users their assistant got weaker, don't let them find out from the output. If local capacity runs out, don't quietly route sensitive code to whatever remote provider is up. Privacy and authorization rules run ahead of routing, no exceptions.
Re-run the comparison whenever traffic, model versions, prices, licenses, hardware utilization, or your review practice changes. A setup that was right at ten thousand tasks a month can be wrong at a hundred thousand. The open model that failed your gate last year might clear it on a better quantization; the free tier you built around might vanish.
What I haven't solved is the re-run trigger itself. I'm supposed to redo this the moment a model version or quantization changes underneath me, but nothing pings me, so it gets rerun only once something breaks, the lagging indicator this exercise was meant to replace. No clean fix yet.