Commercial vs free models for tool-using agents: tools and integration quality
Two routes get the same week of tool calls: an open-weight model running locally on fixed function schemas, and a commercial API with native search, computer use, and managed state on. Neither label survives contact: a free tier is a commercial service with a shorter leash, an open checkpoint still costs hardware and power to run, and only the paid API bundles inference and service into one line item.
three price tags wearing one label
Fix the unit first, the same mistake as metering versus subscribing. For a tool-using agent the unit is a completed task that stayed inside policy, not a token count. Feed it an objective, schemas, and observations, then watch what breaks. Rarely the model refusing outright. More often: a long expensive loop, a malformed call, or a correct-looking action taken without authority. A cheap generation that fails validation and eats ten minutes of a reviewer's time can cost more than the pricier call that finishes once.
the log that actually decides it
Reliability rides on schemas, streaming, argument precision, authorization, errors, and state handling. Run valid and adversarial fixtures against both the open-weight and commercial paths, then score complete task recovery, not tool-name matching. A native capability the open route must hand-roll is a real cost. Keep model IDs and prices in the experiment log, not the permanent rule: catalogs rot.
| track | log |
|---|---|
| identity | route, exact model revision |
| spend | input, cached, reasoning, output tokens |
| speed | queue, first token, completion, timeout |
| infra | hardware energy, idle allocation |
| outcome | pass, retry, abstain, human repair |
| people | engineering, incident time |
| constraints | privacy, license, fallback |
Label failures by mechanism: missing evidence, bad instructions, malformed structure, wrong reasoning, tool error, unsafe action. A local model can nail the easy majority and fail exactly where ambiguity crosses some threshold you won't see coming.
when the boundary actually holds
Commercial wins when native search, computer use, or managed state removes complexity you'd otherwise build. Put a cheaper commercial tier in the mix before assuming you need the flagship. Open weight wins on a small, stable tool set with steady use, where portability matters more than raw capability, the same shape as a local-first cascade. It stops winning once the model barely fits, reloads on every call, or needs a runtime only one person understands.
Put provider glue behind one adapter, version prompts and schemas, and keep job state out of any vendor's thread object. Contract-test streaming and tool calls per route, then re-run the comparison when traffic or prices move: at ten thousand tasks a month can be wrong at a hundred thousand.
I still don't have a clean answer for the commercial side's native tool with no open equivalent, computer use being the obvious one. The adapter keeps the boundary inspectable right up until one side of it can't be built at all.