Vision models for chart and diagram understanding: reasoning across multiple images
A vision-language model never actually looks at the picture you upload. It works on whatever survives resizing, cropping, and tiling before the forward pass, at whatever resolution and token budget the pipeline allotted. The job here is narrower than 'understand this image': recover the relationships without guessing an unreadable label. Check the underlying chart data, caption, legend, and units first. Vision reasoning earns its keep only once layout or language defeats a rule you could write by hand.
log what the model actually saw
Build the path an image takes before the prompt. Label every image, keep timestamps and camera or page identity attached, and preserve the original alongside every transformation applied to it. For local setups, log the projector, quantization, and preprocessing code, not the defaults: a projector from a different checkpoint than the text weights can degrade quietly, unannounced.
| Record this | Why |
|---|---|
| dimensions, crop, resize, orientation | what reached the model |
| image count, ordering, timestamps | identity and sequence errors |
| actual detail/resolution setting | providers change defaults |
| visual, input, output tokens | budget, not one total |
| first response, total latency, peak memory | the whole path |
| grounded / unsupported / abstain / repair | wrong vs. made up |
Score label transcription, relationship accuracy, and evidence citation, counting correct abstention as a pass. Keep unsupported claims separate from missing information: an omission earns a second look, an invented number can trigger a wrong action. Include unanswerable cases; reward admitting it.
the shuffle test
The failure that survives review isn't garbled, it's fluent and wrong: a trend built on a reversed axis, a swapped series, or the wrong arrow direction. Hand it a pile of frames and it will merge different pages, moments, or objects into one description. Nothing flags it. Demand a region, frame, or label behind every conclusion that matters, and show it to the reviewer, not just a log. Split into evidence layers: 'a red indicator, upper right' is observation, 'it's the network panel' is association, 'the connection failed' is inference, 'restart the gateway' is an action that shouldn't ride on the first three alone. Label every image once you send more than one, note whether it's the same object or camera, and drop duplicates that just burn tokens. Where order matters, timestamp frames and run a shuffled control: a model telling the same story after a scramble isn't using it.
where the rule actually lives
Reduce the set, label each view, permit only labeled comparisons, and put it in code, not prompt wording a future edit will drop. Store the model and pipeline version with each result so upgrades can be replayed later. Privacy applies on both ends: crop rooms, faces, and screens irrelevant to the question, and set retention separately for originals and prompts. A local setup, run the way Ollama handles vision models, keeps pixels off someone else's server, not off an unencrypted disk. On a hosted path like Gemini CLI's multimodal mode, test upload time and rate limits; locally, test cold-load and concurrency. Compare the whole path at realistic load, not tokens per second, then stop at the gate.
None of this makes the bigger, higher-resolution model the safer default, though. More pixels mostly means more private detail extracted and more confidence behind an answer the pixels never supported, and I've yet to see a team resist turning it up anyway.