Commercial vs free models for document extraction: latency, throughput, and queues

So no, the free tier isn't free, it's rate-limited commercial service, and the open model just moved that invoice from a token meter to your power bill.

Local inference trades that invoice for hardware, electricity, maintenance, and capacity planning. The unit here is one validated record from a page: OCR text or an image, a schema, source coordinates, JSON that passes or fails. The failure that hurts is confident garbage: valid output holding a wrong value or a dropped qualifier. A cheap call that fails validation and costs a reviewer ten minutes isn't cheap. Pin exact IDs and dated prices in the experiment, not the rule: catalogs turn, tiers vanish, weights improve.

Five separate clocks

Cold start, first token, sustained generation, concurrency, and tail latency are five different properties, not one number for a chart. Replay the real request distribution on both sides and log p50, p95, queue delay, first token, completion, and timeout, holding both to the same standard.

route + exact model revision
pass / retry / abstain / human repair
privacy, license, fallback constraints

Score both routes on field accuracy, schema validity, abstention, and repair time, counting a missing native tool as real integration cost. Keep failures and label them: missing evidence, instruction failure, malformed structure, bad reasoning, tool error, unsafe action. A model that nails easy cases and falls over past some ambiguity line isn't broken, just differently shaped.

Where the two routes trade places

Commercial wins on bursty concurrency, when elastic capacity replaces work you'd otherwise do by hand, not routing everything to the priciest model. Put a cheaper tier in the comparison, cap output and reasoning effort, before crowning the expensive one. The open route wins when requests are local and predictable enough for a resident model to clear latency with no queue, and grows with steady, repeatable work. It fades once the model barely fits or needs constant hand-holding. Borrowed hardware isn't free either.

The mistake everyone makes once: timing a warm, one-off laptop run against an advertised API average and calling it end-to-end latency. Attach every number to a workload and a time period. Price per million tokens and tokens-per-second are cost architecture inputs, not outcomes. What matters is cost per accepted task, p95 completion, failure severity, and remaining human attention.

Put provider handling behind one adapter, keep prompts and schemas in version control, keep job state out of vendor threads, and contract-test every route. Make the fallback explicit: a local-first cascade that drops to a resident model when the API is down is fine, if capability changes are visible. A queue overflowing into a remote provider is a privacy incident, not degraded service; privacy rules run first.

Re-run the comparison when traffic, versions, prices, licenses, or review practice moves. A route right at ten thousand tasks a month can be wrong at a hundred thousand. A model that failed last year might pass now; a free tier that made this easy might vanish. No universal winner.

None of that fixes the real risk on most teams: one engineer understands the runtime, everyone else trusts it. You're the bus factor now. Not the GPU.