Commercial vs free models for document extraction: quality ceiling versus sufficient quality

Score the extraction on what happens after the model answers, not on how well it argues its case, and the commercial-versus-free question mostly answers itself: the route that clears field-level accuracy, schema validity, and capped human repair at the lowest cost per accepted record wins. Price per token is not that number. Neither is a benchmark score.

Four price tags wearing two names

"Commercial" and "free" hide more than they describe. A paid API bundles inference with a service you don't run. A free hosted tier is a limited commercial service. An open-weight model is an artifact whose license may or may not be open source, and the compute is somebody's expense. Local inference trades a token invoice for hardware, electricity, and capacity planning nobody puts on a roadmap.

For document extraction, the unit that matters is one validated structured record from a real document: OCR text or page images, a schema, field definitions, source coordinates. The failure you fight looks right but is quietly wrong: valid JSON, one value off or a qualifier missing. Judge quality where the app accepts or rejects the record, not by how fluent the model sounds.

Run both routes through the same door

Pick candidates: the free side is a local open-weight model wired to ordinary validation code, the commercial side a multimodal API or managed extraction service. Use real model IDs and today's prices in the experiment, then strip them from the rule, since catalogs and quantizations both move fast.

Blind-score representative outputs, record how bad each failure is, and count how often the stronger route changes the reviewer's decision. Give both sides the same evidence and acceptance bar, but let each keep its own prompt template and API shape. If one provider hands you a native tool the other must build by hand, charge for it.

A model that argues its case well is not the same thing as a model that gets the date field right.

Track the model revision, token and timing, and whether the record passed, retried, was abstained, or needed a human. Field-level accuracy, schema validity, abstention, and repair time are the gate. Label every failure: missing evidence, ignored instruction, malformed structure, bad reasoning, bad tool call, unsafe action. The local model often nails the easy cases and trips only past a threshold.

Where the money and the mistake live

The commercial route earns its price when hard cases are common and frontier capability lowers severe errors or review load, not because a dashboard exists. Test that: put a cheaper tier in the comparison too, cap its output and reasoning effort, and see if the gap holds.

The free route earns its keep when a compact, evaluated model clears the gate on most traffic, and the advantage compounds with steady utilization on a narrow, repeatable task. It shrinks fast once the model barely fits, reloads constantly, or runs on a box only one engineer understands.

The recurring mistake is paying for a benchmark lead that never changes a production decision. Pin every number to a workload and a time period: tokens per second is diagnostic, not a business outcome. What counts is cost per accepted task, p95 latency, failure severity, and human attention still baked in.

None of this survives without an exit built in from day one: a local-first cascade for extraction, default cheap, escalate only when the gate says so, keep it visible. Put provider handling behind one adapter, prompts and schemas in version control, job state outside any vendor object, contract-test each route. A local model standing in for a down commercial API is a fine fallback if visible; silently sending sensitive documents to a remote provider because capacity ran out is the failure this design exists to prevent.

What I don't have a clean answer for is drift on the free side. The local model's failure boundary isn't a wall, it's a fog line that moves with your document mix, and nobody upstream watches for regressions. OCR quality softens, templates shift, and repair time creeps for weeks before anyone notices. By then the free route may be worse than the fallback you turned down a year earlier, and short of re-running the comparison on a schedule nobody owns, I have no good trigger for catching it early.