Vision models for chart and diagram understanding: preprocessing before the vision model
Swapping in a bigger vision model rarely fixes a wrong chart reading. Most of the time the defect happened before inference started: a crop that clipped an axis label, a resize that softened a decimal point, a heatmap JPEG-compressed into slightly different colors. Sometimes you do need more model. But that's true less often than the invoices suggest.
The scope worth caring about is narrower than "understand this image." For dashboards, plots, architecture diagrams, and slide visuals, the job is recovering visual relationships and answering one question without guessing at a label you can't read: feed the model the original image or vector export, the caption, legend, units, and the exact question.
check the crop before you check the model
Preprocessing should strip irrelevant variation without erasing evidence or shifting what it means. Build that step deterministically, then compare raw, normalized, cropped, and OCR-augmented inputs against each other, not intuition. Keep the original untouched and log every transform. The record needs more than a model name: runtime, prompt revision, dimensions, crop and resize, image count, OCR or metadata alongside pixels, token accounting, latency, peak memory, and outcome, grounded, unsupported, abstained, or repaired. If the API exposes an image-detail knob, log the value it actually used, not the default. Local setups add one more line: the vision projector, quantization, and context window. Mismatch the projector against the wrong base weights and the server keeps answering fluently while the grounding falls apart.
Grade against label transcription, relationship accuracy, numerical consistency, evidence citation, and abstention on anything unreadable, and include a few unanswerable examples. Start from the boring baseline: if the underlying chart data is sitting right there, use it and skip the vision call.
convincing and wrong look identical from here
The costly failure is a fluent answer describing a real-looking trend while reversing an axis, series, unit, or arrow. A close cousin lives in preprocessing itself: aggressive sharpening that makes text look crisper to a reviewer while it invents a character or paints over a defect.
An omission gets flagged for review. An invented number just becomes a decision.
Score those two differently. Require a cited region, frame, or label for every real conclusion, and show it to the reviewer, not bury it in a log. Split prompts into four layers, each with different evidence: what's visible, what it's associated with, what you infer, and what action follows. A red indicator is observation; that it belongs to the network panel is association; that the connection failed is inference; restarting the gateway is an action, and none should skip straight to the last. Multiple images add identity risk: label them, say whether they're the same object or a time sequence, drop duplicates, and test a shuffled control when order matters. A model telling the same causal story after the frames are scrambled isn't reading the sequence.
the operational tax nobody budgets for
None of this holds without replay. Keep every original retrievable, make each transform reproducible, and put that rule in code, not prompt text. Privacy sits inside the same job: crop out faces, screens, and reflections that don't belong, and set separate retention for originals, thumbnails, and debug captures. A local model cuts what leaves the building, not what sits unencrypted on disk.
Testing a hosted endpoint like Gemini CLI's multimodal mode means watching upload time, regional routing, and rate limits. Testing something local, like a vision model running through Ollama, means watching cold-load time, projector memory, and heat under concurrency. Either way, measure the whole path at p50 and p95, not just generation speed. Stop once the accuracy gate passes at a latency and abstention rate you can live with.
Next time a chart answer looks a little too smooth, skip the bigger model. Pull the logged crop, resize, and detail values, and check what reached the model before deciding what the answer was missing.