Vision models for chart and diagram understanding: prompts grounded in visible evidence
Someone points a vision model at a dashboard screenshot, asks what's wrong, and gets three confident sentences about a metric trending upward. The metric is trending downward: the crop cut the axis label off the edge, and the fix that followed hit the service that was fine. Nobody lied. It answered a question about pixels it never received.
That's the trouble with vision-language models on charts and diagrams. The picture looks concrete, but the model only gets what survived resizing, cropping, tiling, and a token budget, and answers as fluently either way. A confident sentence and a grounded one look identical from outside.
The job is narrower than "understand this image": recover the actual relationships in a dashboard, plot, diagram, or slide, and answer one question without guessing at a label the render never resolved. Feed the model the original image or a vector export, the caption, legend, units, and the exact question.
Check first whether you need a vision model at all. If the underlying data or source graph exists, start there: it can't hallucinate a trend it isn't generating. Reach for a VLM only once that baseline hits layout, ambiguity, or language a parser can't handle economically. It's the expensive fallback, not the default.
The crop already made the call
Keep more of a paper trail than a model name and a vibe. Track these fields every run:
- model ID, runtime, and prompt revision
- image dimensions, crop, resize, and orientation
- image count, ordering, and frame timestamps
- token accounting across visual, input, and output
- latency and peak memory for the full call
- grounded pass, unsupported claim, abstention, or repair
Log what an image-detail or resolution setting resolved to, not the default. Running locally, log the projector, quantization, and context length too: a mismatched projector degrades quietly while the server keeps returning plausible language regardless. Skip the record on a throwaway script; once a call feeds a real decision, the trail stops being optional.
Grade on label transcription, relationship accuracy, numerical consistency, and whether each claim points back at evidence. Score an unsupported claim separately from a missing one: an omission gets flagged for review, a fabricated serial number or trend triggers a wrong action unchecked. Put unanswerable examples in the set, and reward the model for admitting the picture doesn't show what you asked.
Say what's on screen before you say what it means
The failure that bites is a fluent trend running the wrong direction: axis flipped, series mislabeled, unit dropped, arrow backwards. Open questions let the model spin a story the image never proves, and it survives review because it matches what people expect. Split the prompt into four layers: "red indicator, upper right" is observation, "on the network panel" association, "the connection has failed" inference, "restart the gateway" action. Don't let it jump from observation to an irreversible tool call. Every conclusion needs a visible region or label behind it, not buried in a log.
More than one image adds an identity problem on top of grounding. State whether the frames show the same object, page sequence, camera, or time period, and drop duplicates that burn tokens for nothing. If order matters, attach timestamps and rerun it shuffled: a model giving the same story either way isn't using the sequence, it's pattern-matching a shape.
None of this holds if it lives only in prompt wording. Push the "every conclusion cites a visible region" rule into preprocessing, routing, or validation code, and log model and pipeline versions for replay. Crop out rooms, faces, and screens that don't belong, and set retention separately for originals, thumbnails, and embeddings. A model on your own machine cuts what leaves the building; it does nothing for an unencrypted disk or an overbroad account.
Hosted and local paths fail differently, so test them differently. Hosted: upload time, routed region, retention, rate limits, the sort of thing you'd track with Gemini's CLI multimodal path. Local: cold-load time, projector memory, CPU or GPU placement, heat under concurrency, the knobs you watch running vision models locally through Ollama. Compare the whole path at p50 and p95, not tokens-per-second.
Stop tuning once the grounding gate passes at a latency and cost you can live with, and an abstention rate you trust. A bigger model and a sharper crop aren't automatically safer; they just as easily expose more of the picture and tempt answers past the evidence.
What I still don't have a clean answer for is the review side. Once every conclusion carries its own citation, reviewers stop reading them, the way nobody reads a license agreement, and I haven't found where provenance turns from useful into noise skimmed past before approving. I suspect it differs by team, and I'd rather admit that than pretend a citation format solves attention.