Commercial vs free models for coding assistants: the operational burden
One team carries a metered API key for a commercial coding model and a bill that moves with usage. Another runs a local open-weight coder on a GPU it already owns, plus a person who gets paged when the thing falls over. Same ticket queue. Same acceptance tests. Different week entirely.
For a coding assistant the unit that counts is a change a human approves and the test suite accepts, not a token count. Feed either route the same repo evidence, an issue, tool output, a diff, and the failure you hit is a patch that compiles clean on a laptop and quietly breaks a contract nobody wrote down. A cheap completion that fails validation, gets retried, and eats a reviewer's afternoon can cost more than the pricier call that finishes the job once.
four things hiding under one price tag
"Commercial versus free" sounds like one decision. It's four, and they don't behave the same way under load. A paid API bundles inference with a service: uptime, rate limits, someone on the other end when it breaks. A free hosted tier is usually a limited version of that same commercial service, not a separate category. An open-weight model is an artifact, not a service; its license may or may not be open source, and the compute to run it is still an expense, just moved onto your own books. Local inference is the sharpest trade of the four, swapping a token invoice for hardware, electricity, upkeep, and somebody's time on capacity planning.
The comparison that matters: the free or open side doesn't come with reliable serving, observability, backups, or an on-call owner. Your real candidate there is a local open-weight coder or a capped free hosted tier; on the commercial side it's a metered coding API or a frontier model billed by the token. Use real model names and real prices for the experiment itself, but don't bake them into the permanent decision. Catalogs get reshuffled, free allowances get pulled, and last quarter's model gets a better quantization that changes the math.
Give both routes the same evidence and acceptance criteria, and respect whatever prompt template or API shape each one documents. Track setup time, monthly upkeep, failure recovery, and what an outage costs. If one side has a native tool the other has to fake, write down that integration cost too, instead of pretending the two APIs match.
Then log the path itself, not the sticker price:
| Track this | Because |
|---|---|
| Route and exact model revision | "the model" moves without a version pin |
| Token accounting: input, cached input, reasoning, output | these bill and behave differently |
| Timing: queue, first token, completion, timeout | latency is never one number |
| Hardware energy and idle allocation | idle GPU time still costs money |
| Outcome: pass, retry, abstain, human repair | the real failure taxonomy, not "it worked" |
| Engineering and incident time | the hour nobody puts on the invoice |
| Privacy, license, and fallback constraints | the reason you can't always pick cheap |
The quality gate is tests, static checks, diff review, and how long correction takes a human. When something fails, label why, not just that it did:
- missing evidence
- instruction failure
- malformed structure
- incorrect reasoning
- tool error
- unsafe action
Those labels make routing decisions defensible later. A local model often handles routine tickets cleanly and only falls over once context or ambiguity crosses some threshold you can find and name.
the number that actually decides it
I wouldn't trust list price per million tokens or a tokens-per-second figure to tell me which side wins; those are diagnostic inputs, not outcomes. The number that decides it is cost per accepted task, alongside p95 completion latency, how bad failures get, and how much human attention the route needs after the model is done.
Commercial wins when a team wants a supported API and someone else's capacity planning more than it wants infrastructure control. That's not an argument for routing everything to the priciest model: put a cheaper commercial tier in the comparison, cap output length, cap reasoning effort. Managed capacity earns its cost when it removes work you'd otherwise be doing yourself, not because there's a dashboard.
Local or open-weight wins when the org already runs accelerators and wants control over versions, scheduling, or where the data lives. That advantage compounds with steady utilization on narrow, repeatable work, and it evaporates when the model barely fits, reloads constantly, needs hand-holding, or only one person understands the setup. Spare hardware isn't free if some other workload loses access to it.
The mistake I see most is pricing engineering and incident time at zero when someone compares self-hosting against an API, because that's the number that makes self-hosting look free. Attach every figure to a workload and a time period, or don't bother writing it down.
Build the exit before you need it. Put provider-specific request and response handling behind one narrow adapter, keep prompts and schemas in version control, and keep the conversation or job state out of a vendor's thread object. Contract-test streaming, tool calls, structured output, errors, cancellation, and token accounting for every route you ship.
Fallback has to be explicit. If the commercial API goes down, routing to a local model as a degraded mode is fine, as long as the user knows the capability changed. Reverse that logic and it turns dangerous: if local capacity is exhausted, silently shipping sensitive data to a remote provider defeats the point of running local. That's the shape of a local-first cascade: privacy and authorization rules run before capability routing, never after it.
Re-run the whole comparison whenever traffic, model versions, prices, licenses, hardware utilization, or your review habits change. A setup that was right at ten thousand tasks a month can be wrong at a hundred thousand. A local model that choked last year might clear the bar now on a better quant. A free tier can just vanish overnight.
There's no universal winner, and it doesn't need one. What you're building is closer to a cost architecture than a procurement decision: the smallest set of routes that clears your quality gate, keeps cost and data movement visible, and fails safely when something upstream breaks.
The rule I'd keep, if I only got to keep one: never let free or commercial decide anything by itself. Only cost per accepted task, under a fallback you've actually tested, gets a vote.