Commercial vs free models for tool-using agents: latency, throughput, and queues

Run steady, narrow, repeatable tool loops on a model you host yourself. Route anything bursty or spread across time zones to a commercial API and let someone else absorb the queue. That's the whole rule. The rest of this piece is about where it quietly stops holding.

Start with the label: "commercial versus free" is three deals wearing one name. A paid API bundles a model with a running service. A free hosted tier is the same service with a lower ceiling. An open-weight model is different, an artifact you feed your own compute, where license and power bill are separate questions. Local inference just swaps the token invoice for hardware, power, and upkeep.

For an agent that calls tools, the unit that matters isn't a reply, it's a completed task whose tool calls stayed inside policy the whole way through. The failure worth defending against is a long expensive loop, a malformed call, or a correct-looking action taken without authority. A cheap generation that fails validation and burns a retry, or eats ten minutes of a reviewer's time, can cost more than the pricier call that finishes the job once.

where the seconds actually go

Cold start, first token, sustained generation, concurrency, and tail latency are separate properties; folding them into one number is where most comparisons go wrong. One side is an open-weight agent model on your box, or a free allowance; the other a reasoning API with native tool support or managed agent features. Use real model IDs and today's prices, then drop them from the decision: catalogs change, free tiers vanish, and open weights get a better quant.

Replay your request distribution and record p50, p95, queue delay, first token, completion, and timeout rate for both routes, with the same evidence and bar. If one provider hands you a native tool the other side has to hand-roll, count that gap. Grade every failed run by mechanism: missing evidence, an ignored instruction, a malformed structure, bad reasoning, a tool error, an unsafe action. A local model can be flawless on most traffic and stumble only once ambiguity crosses a line you haven't found yet.

Tokens per second is a spec sheet. Cost per accepted task is a budget.

The mistake that keeps recurring is comparing a warm, single-request token rate against a vendor's average and calling that gap latency, the same trap the 99 percent cost architecture piece names for spend. Price per million tokens and tokens per second are diagnostic, not outcomes: report cost per accepted task, p95 completion latency, failure severity, and the human attention still baked in.

the part where the rule bends

Commercial earns its keep when concurrency is bursty and users are scattered across time zones, because elastic capacity is worth paying for, not because a dashboard exists. Put a cheaper commercial tier in the comparison too, and cap reasoning effort before reaching for the flagship model. Local earns its keep when traffic is predictable and a resident model clears your latency bar without standing in a queue, the local-first cascade idea again. That edge compounds with steady utilization and disappears once the model barely fits, reloads constantly, or leans on a runtime one person understands: borrowed hardware isn't free once another workload needs it too.

Keep provider handling behind one narrow adapter, prompts and schemas in version control, conversation state out of a vendor's thread object. Fallback should be a decision, not a side effect: local stepping in when the API is down is fine, as long as users know capability dropped, and an exhausted local box is never a reason to ship sensitive data to a remote provider. Re-run the comparison whenever traffic or prices move: a split right at ten thousand tasks a month can go wrong at a hundred thousand.

I'd still bet on running this test over trusting a vendor's benchmark page, but the test has a shelf life, and the tidy adapter you build today is the thing you'll end up defending once the next model generation makes half of this comparison beside the point.