OCR models for scanned archives: choosing an OCR-capable model

Archives teams have shipped "clean" transcriptions where a vision-language model swapped a smudged surname for a more common one, or read a stained "1" as a "7" because the sentence around it made more sense. Nobody caught it until someone checked the original scan, and by then the bad value had spread into an index and a trusted database column.

That's the real risk with OCR on scanned archives: not that it fails to produce text, but that it produces confident, readable text that's wrong in ways spellcheck won't catch. The material is books, correspondence, reports, and records off paper or microfilm, and the target is searchable text with page identity, reading order, and marked uncertainty. Skew, bleed-through, stains, and obsolete typefaces are the norm.

Pick a model for script, layout, handwriting, resolution, and output structure, not for how nice a demo sample looks. Start with a conventional OCR engine, language packs, and layout segmentation, cheap and deterministic. Bring in a generative model only where it earns its keep, and don't let it replace stages that already work.

Freeze the page before you touch it

Run the comparison first: a classical OCR engine, a layout-aware parser, and an OCR-capable vision-language model, on the same labeled pages, not a generic benchmark. Lock the file, hash it, assign a page and document id that survives every step. What you're building is a chain of custody:

Never let normalization overwrite what the recognizer saw. An O and a 0, an I and a 1, a decimal point, a minus sign, a unit, a diacritic: any of these can be ambiguous on a bad scan. Store what the model read and what the business rule proposed as two separate fields, always.

The metric that hides the expensive mistake

Character and word error rate are fine as a floor, not a ceiling. They score a dropped comma and a transposed digit in an account number as the same size of mistake, one cosmetic, the other a compliance problem. Add exact-match accuracy for the fields that matter: dates, amounts, warnings, units, identifiers. For structured documents, score field association and table geometry separately from transcription quality, because a model can nail every character and still put the number in the wrong cell. Round it out with page coverage and correction time, the number telling you if this was worth building.

Plausible is worse than garbled

Garbled OCR is annoying but honest. You can see it's broken. The failure that costs you is clean prose where an unfamiliar word, a date, or a name got silently swapped for something more common. That's the trap in picking a general vision-language model off a visual benchmark: it was trained to produce plausible continuations, so a damaged surname or total gets completed into whatever looks familiar, not what was printed. A high benchmark score says nothing about whether a model holds the line when a scan is ambiguous, and exact transcription is the whole gate here.

Give the system somewhere to say it doesn't know. Low-resolution regions, clipped pages, glare, unreadable handwriting, and totals that don't add up should raise an exception, not a guess. Confidence scores alone aren't enough, they're poorly calibrated: pair them with image-quality checks, format validators, cross-field arithmetic, and disagreement between independent passes. Store the bounding polygon for every field that matters so a reviewer sees the crop beside the value, which turns review into verification instead of retyping.

Reasoning without the keys to the car

A vision-language model earns its place doing what plain OCR structurally can't: tie a label to a value three inches away, reconstruct a table split across a fold, explain why two totals disagree. Nothing more. Keep raw observation and interpretation separate, and gate any downstream write behind schema validation outside the model. Text printed inside a document is data the model reads, not an instruction it should follow, and treating it otherwise gets you a model that "helpfully" rewrites its own extraction rules because a form field happened to look like one.

Multi-page batches need the same discipline: don't hand the model an arbitrary stack of pages and trust it to keep invoices separate. Split documents deterministically, carry identifiers through every stage, and state whether a table may continue across a page break. Run the cheapest recognizer that preserves text and layout, and escalate ambiguous regions rather than every page. Version the model, preprocessing, dictionaries, and validation together, and replay a fixed stratified set, clean pages, hard layouts, critical fields, the pages built to break it, whenever one changes.

Deployment is its own tradeoff, and I don't have a clean answer. Local keeps documents under your control and gets cheaper at high, steady volume, but you pay for storage, backup, accelerator time, and maintenance either way. Hosted document APIs and frontier vision models give elasticity and handle more exceptions, at the cost of upload time, retention rules, residency, and per-page charges. What I haven't found a satisfying answer for is where to draw the hybrid-routing line on a collection with mixed sensitivity: some folders can leave the building, some can't, and that boundary rarely matches how the files are organized on disk.