OCR models for scanned archives: handwriting mixed with printed text
An old ledger page has a printed header and a total scrawled in pencil beside it. One scan, two confidence problems: treat them the same and you get a wrong number that looks right.
Scanned archives (books, correspondence, reports, microfilm) aren't a clean image-to-text job: the system decides what belongs on the page and whether characters are trustworthy. Skew, bleed-through, and inconsistent scanning are normal here.
Split the page before you trust any of it
Segment print, handwriting, marks, and blank fields before interpreting how they relate. Start with a conventional OCR engine: language packs, layout segmentation. It's cheap, it works. Add a generative model only for hard handwriting or layout exceptions.
Keep the source file immutable, assign page and document IDs, and log the chain:
- source hash, page, capture device
- preprocessing versions: crop, dewarp, threshold, denoise
- model and runtime revision
- raw text, confidence, reading order, reviewer edits
Never let normalization overwrite the transcription: for an ambiguous O/0 or I/1, store what the recognizer saw next to what the rule proposed. Error rate alone misleads; add exact-match accuracy on identifiers and amounts.
The wrong answer that reads perfectly
The scariest failure is quiet: fluent prose that swaps an unfamiliar name or number for something more plausible and never looks broken, because that's what these models do. On mixed pages, a strong printed-text score can mask bad handwriting, or auto-link a note to the nearest label.
Give the pipeline a way to say it doesn't know. Low-resolution crops, glare, unreadable handwriting, totals that don't add up: raise an exception, not a guess. Confidence scores are often poorly calibrated: pair them with image-quality checks, format validators, and cross-field arithmetic.
Store a page number and bounding box for every important field, and show the crop beside the value. Skip it and review means hunting the whole page.
What the model is and isn't allowed to touch
A vision-language model earns its keep linking a distant label to a value, rebuilding a table, or explaining why two totals disagree. Keep it to observation: schema validation and business rules stay outside the model. The page's text is data, not permission to rewrite instructions or call a tool.
For multi-page batches, preserve page order and header identity. Split documents deterministically with real IDs instead of merging invoices from a bundled folder.
I wouldn't point a VLM at plain printed text a rule-based engine already reads cleanly. Local deployment stays under your control and gets cheap at steady volume; hosted APIs trade that for elasticity at a per-page cost. Hybrid routing wins when privacy rules are strict.
Version models, preprocessing, and validation together. Before trusting this on a real archive, replay a stratified batch: clean pages, noisy pages, tricky layouts, critical fields, a few impossible ones, and watch how often it defers. That deferral rate matters more than any accuracy score.