Commercial vs free models for document extraction: tools and integration quality

Every team that has shipped document extraction has watched this: a call comes back fast and cheap, the JSON parses clean, the schema check passes, and the record is wrong anyway. A total transposed. A qualifier like "estimated" dropped. Nobody notices until someone downstream keys that number in, and the reviewer's ten minutes fixing it cost more than the pricier call would have.

"Commercial versus free" flattens several things into one argument: a paid API bundles the model with a service you don't operate, a free tier is that same service with a limiter switched on, and an open-weight model is an artifact you can inspect, license aside, whose compute is somebody's expense regardless. Running it yourself moves that bill to hardware, power, and capacity planning.

For extraction the unit that matters is one validated structured record: OCR text or page images in, a schema, field definitions, and source coordinates as the contract, a JSON object that matches the source or doesn't.

Count the reviewer's ten minutes

Run two candidates through the same tool-reliability rules: schemas, streaming, argument precision, authorization, state that survives a retry. One is a local open-weight model wired to your own validation code, the other a commercial multimodal API or managed document service with a native tool. Use real model IDs and dated prices in the experiment, then drop them from the permanent rule: catalogs change, allowances get pulled, quantizations flip verdicts.

Score both on the same fixtures, plain and adversarial, on whether the task recovers, not whether a tool call matched a name, respecting each provider's template. A native tool one side has that the other hand-rolls is a real difference: count it, don't pretend the endpoints match.

What I log per route, every run:

route + model revision
input / cached / reasoning / output tokens
queue / first token / completion / timeout
power draw + idle allocation
pass / retry / abstain / repair
engineering, incident, privacy, license, fallback

The gate is field-level accuracy, schema validity, abstention rate, and human repair time. Keep every failed output and tag why:

• missing evidence
• instruction failure
• malformed structure
• incorrect reasoning
• tool error
• unsafe action

Those tags make a routing call defensible months later. A cheap local model can nail most of a set and fail only once ambiguity crosses a threshold you can point to, not randomly.

Let the workload pick, not the vendor

Commercial usually wins when native search, computer use, file handling, or managed state removes real complexity you'd otherwise build yourself, not the flashiest API around. Put a cheaper commercial tier in the comparison, and cap output length and reasoning effort. Managed capacity earns its price by removing real work, not by having a dashboard.

Free or open-weight wins when the workflow needs a small, stable tool set and portability matters more than convenience. That edge grows with steady, repeatable work and shrinks once the model barely fits in memory or needs a runtime only one person understands. Spare GPU capacity isn't free if another workload loses it while yours runs.

The mistake I see most is treating API compatibility as proof tool-call behavior matches across providers. Attach every number to a workload and a time period: price per million tokens and tokens per second are diagnostic, not outcomes. Report cost per accepted task, p95 latency, failure severity, and remaining human attention.

Put provider-specific handling behind one narrow adapter, store prompts and schemas in version control, and keep authoritative job state outside a vendor-only thread object, much like a local-first cascade already does. Contract-test streaming, tool calls, structured output, errors, and token accounting for every route you support.

Make fallback explicit, not automatic and silent: a local model as degraded mode when the commercial API is down is fine, but the user should know capability changed, and routing sensitive documents to a remote provider once local capacity is exhausted defeats the point of running local. Privacy rules run before capability routing, no exceptions for a busy Tuesday.

Re-run the comparison whenever traffic, models, prices, licenses, or review practice change enough to matter: a route right at modest volume can be wrong tenfold, last year's failing local model can pass today on a better quant, and a free tier can vanish overnight. "Commercial" and "free" were always procurement labels, not engineering ones, the point behind why a subscription is the wrong unit for AI cost.

What I still don't have is a clean way to price engineering and incident time in the same currency as token spend. I can count retries, abstentions, and reviewer minutes, but hours nursing a flaky local deployment, or working out why a commercial tool call changed shape after a provider update, don't fold into a cost-per-record number without hand-waving. Until somebody solves that, every comparison I run carries an asterisk I can't shake.