Commercial vs free models for coding assistants: latency, throughput, and queues

And that's the trap: you pick "local" or "commercial" as a philosophy, then spend a long while discovering it was actually a latency problem in disguise.

"Commercial" and "free" aren't two products, they're three, and conflating them is where the argument goes wrong. A paid API sells inference wrapped in a service: uptime, rate limits, someone to call. A free hosted tier is usually that same service with a governor bolted on, not a gift. An open-weight model is neither, an artifact, license open or not, whose compute costs somebody electricity regardless. Local inference swaps the invoice for hardware, maintenance, capacity planning, not escape from paying.

the unit nobody prices correctly

For a coding assistant the thing you're buying isn't tokens, it's a reviewed change that clears the repo's test suite, built from repository context, issue text, tool output, diffs. The costly failure is a patch that compiles locally, looks plausible, and quietly breaks a contract the prompt never mentioned. A cheap generation that fails review and eats ten minutes of a reviewer's afternoon moved the cost somewhere your dashboard doesn't look.

five clocks, not one

Cold start, first token, sustained generation, concurrency, and tail latency are five separate numbers; treating them as one is how these comparisons go wrong. One side is a local open-weight coder, or a rate-limited free tier standing in. The other is a metered commercial coding model, or a frontier API. Use real model IDs and today's prices for the test, not the architecture: catalogs won't stay put.

Replay your real request distribution against both routes and log the same fields for each:

route + exact model revision
input / cached input / reasoning / output tokens
queue, first token, completion, timeout
hardware energy + idle allocation
pass / retry / abstain / human repair

Grade every run against one bar: tests, static checks, diff review, measured correction time. Keep the failures and tag why: missing context, ignored instruction, malformed output, broken tool call. That tagging makes routing possible, since a local model usually nails the boring bulk and falls over only once ambiguity crosses some threshold you can now name.

where the queue costs more than the wire

Commercial capacity earns its keep when traffic is bursty and users are scattered across time zones, because elastic capacity absorbs that, not raw model quality. That's no argument for routing everything to the flagship model: put a cheaper commercial tier in the same test and cap the output length instead.

The local route wins the opposite problem: steady, predictable, narrow tasks where a resident model clears your latency bar without touching a queue. That edge grows with utilization and shrinks once the model barely fits in memory, reloads constantly, or needs a runtime only one engineer understands. Borrowed GPU time isn't free if another job needed it.

the comparison everyone gets wrong on purpose

The classic mistake: quoting a warm, batch-of-one local benchmark next to an API's advertised average and calling that end-to-end latency. It isn't. Tie every number to a workload and a time window or throw it out. Price per million tokens and tokens-per-second are diagnostics, not conclusions, the way 99-percent-cost-architecture treats sticker price as one input, not the answer: what you want is cost per accepted task, p95 time to a finished change, and the human attention the invoice hides.

one seam, not five integrations

Put every provider's request and response handling behind one narrow adapter, prompts and schemas in version control, the discipline a local-first-cascade setup assumes. Don't let a vendor's thread object become the source of truth for job state. Contract-test streaming, tool calls, and structured output for every route, so a changed error shape doesn't surprise you mid-incident.

Fallback needs to be a decision, not an accident. Commercial API down, a local degraded mode is fine, but tell the user capability changed. Local capacity exhausted, quietly routing sensitive requests to a remote provider is a leak, not a fallback; privacy rules run ahead of routing, never behind. Re-run the comparison when traffic, prices, or licenses shift: what's right at ten thousand tasks a month won't hold at a hundred thousand.

None of which means you need a universal winner, or that any of this is worth doing at your current scale. If you're shipping a handful of PRs a week, skip the adapter, skip the logging pipeline, and call the API like everyone else does.