Commercial vs free models for coding assistants: privacy and data control
A GPU you already own for another job doesn't become free just because no invoice shows up for the tokens it burns. Local inference doesn't remove cost. It relocates it: from a metered API line to hardware, electricity, maintenance, and whoever now owns capacity planning.
three things wearing the same "free" label
"Commercial versus free" hides three different products. A paid API bundles inference with a service. A free hosted tier is usually a limited version of that same service. An open-weight model is neither, an artifact whose license may or may not be open source, and the compute is still somebody's expense.
None of that is the axis that matters for a coding assistant working over repo evidence, issue text, diffs. Privacy is who collects the request, where it transits, how long it's retained, which admins can see it, and which tool gets called with your code. Test with real prices, kept out of the architecture doc: catalogs reshuffle, allowances vanish, quants improve.
count the whole route, not the token price
The unit that matters is a reviewed change that clears your test suite, not a diff that looks confident. The failure is a patch that compiles cleanly and quietly violates a contract nobody wrote down. A cheap completion that fails validation and burns a reviewer's afternoon can cost more than the pricier call that finishes the job.
Map every data copy and operator, verify retention and residency on paper, and test with sensitive fields stripped before trusting either number. Run both routes on identical evidence, in their own prompt templates and APIs. Per route, I track:
- exact model revision, not the family name
- input, cached, reasoning, and output tokens, kept separate
- queue time, completion time, and timeout rate
- pass, retry, abstain, or human repair
Tag every failure by mechanism: missing evidence, a misread instruction, malformed structure, bad reasoning, a tool error, an unsafe action. A local model nails routine cases and falls over once ambiguity crosses a line these labels expose. Price per token and tokens per second are diagnostic, not a verdict: what decides is cost per accepted task, p95 latency, and human attention on it.
pick a side, but leave a door in the wall
Commercial earns the work when its contract, certifications, and managed security cover something your team would otherwise build, not because a dashboard feels safer. Free or open-weight earns it when data must stay on hardware you control, compounding with steady work and fading once the model barely fits or needs one irreplaceable person; spare hardware isn't free once another job needs it back.
Calling something "private" just because inference happened on a box down the hall is the recurring mistake: tie every number to a workload and time window instead. Fallback runs both ways: local is a fine degraded mode if someone's told, and an exhausted local box never means quietly routing sensitive code out. No universal winner here, just the smallest route set that clears the gate and fails safely.
I run the low-stakes, routine generation on a local-first cascade and pay for the commercial API where I need the certifications behind it. What I give up: raw throughput and one system to reason about, in exchange for never explaining why a client's code sat on a server I patched myself at midnight.