OCR models for scanned archives: structured OCR with provenance

Here's the rule I use, flat: if a field in your output can't be traced to a specific patch of pixels on a page, don't trust it. That's the whole design brief. The rest of this is me earning that line, because an unreasoned rule is just a vibe, and vibes don't survive a stained old microfilm scan.

treat the scan as evidence, not a rough draft

Scanned archives means books, correspondence, reports, historical records off paper or microfilm. The target is text with page identity, reading order, and honest uncertainty intact, against skewed pages, bleed-through, stains, old typefaces, and scan quality that drifts by who ran the scanner.

Keep the original file exactly as captured, assign a document and page identifier before anything touches it, and log the chain: capture metadata, each preprocessing step's version, which model ran at what revision, and any reviewer change. The test: does it return coordinates and raw spans, not just a clean value.

Never let normalization overwrite transcription: an O and a 0, a decimal point, a unit, a diacritic, these turn ambiguous on a damaged page, and you store what the recognizer saw next to what the rule proposes, not one over the other.

Start with a conventional OCR engine, language packs, and layout segmentation, the stuff that already works cheaply, and bring a vision-language model in only where it earns its place: hard handwriting, irregular layouts, or the exceptions the deterministic stages bounce out. Error rate treats a missing comma and a wrong account number as the same mistake, so add exact-match accuracy on load-bearing fields and score table geometry separately.

the failure that looks exactly like success

Garbled output gets caught fast. What slips past review is clean, fluent prose that quietly swapped an unfamiliar surname, a damaged date, or a smudged part number for something more familiar: a language model continues plausible patterns, and a broken sequence is the gap it's built to fill convincingly. Store only the corrected value and the change is permanent and invisible.

Build an abstention path rather than trust the model to know when it's guessing: low-resolution regions, clipped pages, glare, unreadable handwriting, contradictory totals should raise an exception, not a confident answer. Model confidence is poorly calibrated, so pair it with image-quality checks, format validators, cross-field arithmetic, and dictionaries.

Provenance turns review into verification: store the page number and bounding polygon per field that matters, and show the reviewer the crop beside the proposed value. An uncertain digit resolves in seconds with the page in front of you; without coordinates it means searching a whole page, or reopening a physical document.

keep the model on a short leash, know when to skip it

A vision-language model earns its keep at specific jobs: linking a label to a distant value, reconstructing a fragmented table, explaining why two totals disagree. Keep raw observation separate from interpretation, and never let its output write to a database or call a tool without schema validation outside the model. Text printed in a document is data it's reading, not an instruction to follow.

Multi-page work needs the same discipline: preserve page order and repeated-header identity, split documents deterministically rather than letting the model sort a bundle into invoices on its own, and state whether a table may continue across a page boundary. Keep raw recognition, normalized value, confidence, and source region separate always, and replay a fixed set of clean pages, hard layouts, and unreadable cases whenever the pipeline changes.

Local deployment keeps documents under your control and gets cheap at high, steady volume, but you pay for storage, model memory, accelerator time, and upkeep yourself. Hosted APIs and frontier vision models buy elasticity and a higher exception ceiling, at the cost of upload time, retention terms, and a per-page bill. Hybrid routing, local first with visible escalation, suits strict privacy rules.

None of this changes the rule up top: the best system isn't the one producing the prettiest paragraph off a damaged page, it's the pipeline that keeps evidence, flags what it isn't sure of, and lets someone check any field without starting over. Honest caveat: for forty pages run once, this much provenance is more infrastructure than the job needs, and I'd probably just eyeball the scans with coffee instead.