Run OCR before reaching for a vision LLM

OCR first, vision model second, and only for the part of the page that earns it. Not a knock on vision-language models: they earn their cost on layout ambiguity and visual reasoning, a crossed-out field, a stamp over a signature, not on printed text a plain OCR engine reads correctly.

Most local-AI talk starts with a model name or a throughput number. The better start is the work on the other side of the request: a private chat used twice an hour, a coding loop, and a nightly batch of invoices can share one box and still want different things from it, the same logic behind local-first cascade.

confidence scores earn the GPU time

Before touching a config, write down a baseline: one named model, one fixed prompt set, the exact launch command, model artifact and prompt template attached; skip that and a rerun proves nothing. Let OCR read, then use its confidence and layout signals to pick what escalates: low-confidence regions, tables that won't parse. Everything else stays cheap, the same triage logic as an autorouter pointed at pixels.

Measure the whole request, not one favorite number: time to first token for loading and prompt cost, steady token rate for decoding, completion time for what the user feels. Add peak memory, queue delay, and wall power when they matter, valid jobs per hour on automated work, the waits people notice on interactive work. Log workload, fixed input, cold/warm p50/p95, failure and abstention rate, plus a decision with an owner and a retest date. A benchmark with no decision is trivia; a setting nobody owns turns into folklore.

the failure that never throws an error (ocr)

The mistake that survives longest: routing every full-resolution page to the biggest vision model, since the output still reads fine. Nothing crashes. Local inference fails quietly more than loudly: a model spills onto the CPU without saying so, a cache stops hitting, swap creeps up, a queue keeps serving requests whose client gave up, a fallback changes where the data goes. None of that throws an error.

Catch it by watching runtime logs and OS counters while the test runs, one variable at a time unless comparing whole configs, repeated enough to rule out a lucky sample. Read the actual output, since a faster wrong answer isn't progress, and write down the cost when a change hurts an important case. Simplicity matters too: a fix needing repair after every driver update isn't free, and a boring setup from a service file beats a clever one you can't reproduce. Keep raw artifacts out of the report, but keep the hashes and commands so a rerun months later still means the same thing.

The rule that holds: let deterministic extraction take the clear text, spend inference only on real ambiguity. Less exciting than picking the biggest model, but it leaves a stack with visible limits you can schedule around, route around, or price, not invisible ones that end in unexplained waiting and an emergency upgrade. Stop once quality and latency clear their targets with headroom left; that headroom absorbs a longer document, an extra user, a warm afternoon, or the next runtime release.

Check the routing log next, not the model. Pull last week's escalations and see how many pages were genuinely ambiguous versus how many tripped a confidence threshold set wrong. That ratio says whether you're paying for uncertainty or for a bad default.