Commercial vs free models for tool-using agents: the real cost per completed task

Route the same support ticket through two agents: one hitting a commercial reasoning API with native tool calls, the other running an open-weight model on a box someone bolted under a desk. One of them finishes faster. Neither invoice tells you which.

"Commercial versus free" is really three deals wearing one label. A paid API bundles inference with a service you don't operate yourself. A free hosted tier is that same commercial service with the meter off for now, not a different category of thing. An open-weight model is an artifact: the license may or may not let you use it your way, and the compute to run it is still somebody's line item. Local inference doesn't remove cost, it swaps a token invoice for hardware, electricity, and capacity planning.

For an agent calling tools against a live system, the unit worth measuring is a completed task whose tool actions stayed inside policy, not a token count. That means an objective, tool schemas, a trail of observations, and execution state holding together across steps. The failure you're guarding against is a long expensive loop, a malformed tool call, or worse: a correct-looking action taken without the authority to take it.

A cheap generation that fails validation or eats ten minutes of a reviewer's afternoon can cost more than the pricier call that finishes once. Total cost includes inference, idle hardware, engineering time, retries, and human review, or you're comparing list prices, not costs. Run the experiment with real model IDs and this week's prices, just don't bake either into the architecture: catalogs get reshuffled, free allowances get pulled, and someone ships a better quantization of the open model right after you finish it.

Counting retries, idle GPUs, and reviewer minutes

Give both routes the same evidence and acceptance test, then let each work the way it actually works: its own prompt template, its own API shape, its own tool-calling convention. Don't force a shared harness that flattens the differences you're measuring. If the commercial side has a native tool the open-weight side must fake with a wrapper and retries, that gap is real, and it belongs in the number, not a footnote.

What needs logging, for every accepted result, not just the smooth ones:

The quality gate is task completion, tool-call validity, step count, recovery, and policy violations. Keep the failures instead of discarding them, and label why each happened: missing evidence, an ignored instruction, malformed structure, wrong reasoning, a tool error, or an action that never should have fired. Those labels make a routing decision defensible later instead of a guess in a chart. A local model can handle the boring bulk of a workload, then fall over once ambiguity crosses a threshold nobody flagged.

Build the adapter before you pick a winner

The commercial route tends to win when traffic is irregular, a failure is expensive enough that capability beats unit price, and the provider removes real operating work you'd otherwise carry. That's not an argument for routing everything to the single most capable API on the price list: put a cheaper commercial tier in the comparison too, and cap output length and reasoning effort first. Managed capacity earns its premium when it removes work your team would otherwise do, not because it ships with a dashboard.

The free or open-weight route tends to win when volume is steady, the task genuinely fits a smaller model, and the hardware and staff to run it are already sitting there. This is close to the local-first cascade idea: keep narrow, repeatable work on the cheapest thing that clears the bar, and reach past it only when the task demands it. That advantage grows with steady utilization and shrinks once the model barely fits, reloads constantly, or needs manual repair. Spare hardware isn't free if another workload loses it while your agent is busy.

The recurring mistake is dividing a GPU purchase by theoretical lifetime tokens, or comparing nothing but input prices. Fix it by attaching every number to a workload and a time period. Price per million tokens and tokens per second are diagnostic inputs, not outcomes. Report cost per accepted task, p95 latency, failure severity, and how much human attention it still eats.

Put provider-specific handling behind one narrow adapter, keep prompts and schemas in version control, and keep the authoritative job state outside any vendor-only thread object. Contract-test streaming, tool calls, structured output, errors, cancellation, and token accounting for every route you support, not just today's. It's the same instinct as an earlier piece on cost architecture: boring plumbing lets you change your mind without a rewrite.

Fallback has to be explicit. If the commercial API goes down, a local model can be a reasonable degraded mode, but the people relying on it should know capability changed. If local capacity runs out, quietly routing sensitive data to a remote provider defeats the point of running local at all. Privacy and authorization rules run before capability routing, always.

Re-run the comparison whenever traffic, model versions, prices, licenses, or review practice shifts. A choice right at ten thousand tasks a month can be wrong at a hundred thousand. A local model that failed last year might pass now on a better quant. A free tier that made the plan work can vanish with a blog post nobody reads until it's too late.

None of this needs a universal winner. "Free versus commercial" was always a procurement question dressed up as an engineering one. What I'd check next: pull last month's failed tool calls, sort them into the six failure categories above, and see which route produced fewer per completed task. That number beats any list price.