Commercial vs free models for document extraction: a hybrid route instead of a winner

And that's usually where the argument stalls: someone says run the open model, it's free, someone else says pay for the API, and both have skipped the actual question, which is what happens on the request in front of you, not what happens in general.

free is not one thing

"Free" and "commercial" are procurement words, not technical ones. A paid API bundles inference with a service: uptime, support, someone absorbing the pain when the format shifts. A free hosted tier is the same service with the price hidden behind a rate limit. An open-weight model is an artifact, not a service: the license may or may not be open source, and the compute is still somebody's expense, moved from a vendor invoice to a power bill. Local inference swaps that invoice for hardware, electricity, maintenance, and capacity planning. Use real model names and prices in the experiment, then keep the numbers out of the architecture itself: catalogs change, free tiers vanish, open weights get new quantizations.

what actually breaks on a real document

Document extraction has one clean unit of work: a validated structured record from a real document. The input is OCR text or page images, a schema, field definitions, and coordinates back to the source. The failure mode isn't a crash, it's worse: JSON that validates cleanly and is simply wrong, a date transposed, a qualifier dropped. That's the whole game. A cheap model that produces confident garbage and costs a reviewer ten minutes hasn't saved you anything; a pricier call that nails the record once is often the actual bargain.

test both routes on identical evidence

The only fair experiment labels traffic, applies hard sensitivity filters up front, sends the same document down each route, and counts two numbers: escalation frequency and accepted cost with retries folded in. Give each side its own prompt template and API shape rather than forcing both into one interface. If one provider hands you a native tool the other has to hand-roll, count that integration tax.

route and exact model revision
input, cached input, reasoning, output tokens
queue, first token, completion, timeout
hardware energy and idle allocation
pass, retry, abstain, human repair
engineering and incident time
privacy, license, fallback constraints

Score against field-level accuracy, schema validity, abstention rate, and human repair time. Tag every failed output: missing evidence, ignored instruction, malformed structure, wrong reasoning, tool error, unsafe action. Those tags make routing worth trusting. A local model usually nails the boring cases and only trips once ambiguity crosses a visible threshold.

two thresholds, no single champion

Different request classes deserve different capability, privacy, latency, and cost boundaries, that's the real comparison, not a search for one winner. The free candidate is a local open-weight model plus ordinary validation code; commercial is a multimodal API or a managed document service. Commercial earns the harder cases, where extra capability repays its premium against getting it wrong, not a license to default to the flagship model out of habit. Put a cheaper commercial tier in the comparison too, cap reasoning effort and output length, and see if it clears the same bar. Managed capacity earns its cost only when it replaces work your team would do by hand, not because a dashboard exists.

Local earns the routine cases: high volume, low ambiguity, anything sensitive enough you don't want it leaving the building. It's the same instinct behind a local-first cascade: the advantage grows with steady utilization and narrow, repeatable work, and shrinks once the model barely fits, reloads constantly, or needs a runtime only one engineer understands. Spare hardware isn't free if another workload loses its slot to get it.

The mistake I see most: reaching for another expensive model to do the routing before plain rules and validation signals are exhausted. Attach every number to a workload and a time window, list price and tokens-per-second are inputs, not the decision, the same discipline behind a bare-bones cost architecture built for the common case. Cost per accepted task, p95 latency, failure severity, and how much human attention is still baked in, that's what you actually want.

keep the wiring behind one door

Put every provider's request and response shape behind one narrow adapter, keep prompts and schemas in version control, and don't let authoritative job state live inside a vendor's thread object. Contract-test streaming, tool calls, structured output, errors, cancellation, and token accounting for every route you support.

Fallback has to be visible, not implicit. Dropping to a local model as a degraded mode when the commercial API is down is fine, so long as whoever's watching knows capability changed. Shipping sensitive documents to a remote provider because local capacity ran out breaks the premise of the system: privacy rules run before capability routing, full stop. Re-run the comparison when traffic, prices, licenses, hardware utilization, or review practice change. A choice correct at ten thousand tasks a month can be wrong at a hundred thousand, and a model that failed last year might pass now behind a better quant, right as a free tier disappears.

If I had to keep exactly one rule out of all this: the routing decision stays visible and gets revisited on a schedule, never permanent, never assumed.