OCR models for scanned archives: layout and reading order

So a clean character error rate means nothing if the model read the footnote before the paragraph it belongs to, or merged two tables into one page.

The page is a graph, not a string

A scanned page is blocks in space that only mean something in the right order, and archival material fights that: skew, bleed-through, stains, inconsistent scanning. A vision-language model can untangle an ambiguous layout, or just as easily swap a doubtful character for smoother prose. Start with conventional OCR and layout segmentation, adding a generative model only for hard handwriting, odd layouts, or exceptions it kicks back, never replacing what already works.

Keep a record of what touched the page

Test on real multi-column pages, sidebars, and continuations, scoring structure apart from text, then freeze the file with a page and document identifier. Log the source hash, every preprocessing version, which model handled it, and the raw output, kept apart from review. If a character could be an O or a zero, a decimal point or a stray mark, store what the recognizer saw next to what the rule proposes, as two fields. Error rate alone treats a missing comma like a wrong account number; add named-entity and exact-match accuracy on critical fields, plus page coverage and correction time.

When the text looks cleaner than the page ever did

The most dangerous failure isn't garbled output, it's clean output: a damaged surname or part number completed into something plausible, because that's what a language model does with an unfamiliar gap.

A language model finishes a damaged name the way a good guesser finishes a crossword clue: plausibly, not correctly.

Give the pipeline permission to say it doesn't know: low resolution, glare, unreadable handwriting, or disagreeing totals should raise an exception, not a guess. Confidence scores are inconsistent and poorly calibrated; pair them with quality checks, format validators, cross-field arithmetic, and agreement between passes. Store the page number and bounding box for fields that matter, and show the crop beside the value someone signs off on, so a reviewer settles an ambiguous digit in seconds instead of hunting through the page.

The model doesn't get to grade its own homework

A vision-language model earns its place linking a distant label to a value, rebuilding a split table, or explaining disagreeing totals, but keep it observing, not deciding: writes and validation stay outside the model, and text on the page is data, never an instruction. Preserve page order and repeated headers, split documents on purpose, and state whether a table crosses a page boundary. Version the model, preprocessing, and validation together, and replay a fixed test set when any of them changes. Local gets cheaper as volume grows; hosted buys elasticity and a higher ceiling for weird cases, at a price.

So here's the one rule I wouldn't trade away: never let the cleaned-up value replace the evidence it came from.

Keep the crop and coordinates attached to every value for as long as it matters. Everything else is negotiable. That one isn't.