Vision models for chart and diagram understanding: resolution and visual-token budgets

A screenshot of a dashboard goes into the model, and a paragraph of confident analysis comes back. Nobody in that exchange saw the same image: you saw the panel at full resolution, the model saw whatever survived resizing, tiling, and cropping before it reached the vision encoder. That gap is the entire problem with vision-language models on charts and diagrams. The picture feels concrete, so the answer feels earned, but the model is reasoning over a transformed representation with its own crop, resolution, and token budget, and a fluent answer can be fully grounded, partially grounded, or built around a detail that never made it through preprocessing.

The material here is dashboards, plots, architecture diagrams, schematics, slide visuals: the job is to recover a relationship between marks and labels, not to describe a scene. The right input is narrower than understand this image: the original image or a vector export, the nearby caption, the legend, the units, and the exact question someone wants answered. Narrow it that far and model choice, preprocessing, evaluation, and safe failure become things you can measure instead of argue about.

The camera's resolution is not the model's resolution

Useful pixels are set by the smallest piece of evidence the task has to resolve, not by how many megapixels the source image carries. A four-figure axis label needs enough resolution to read four figures; the rest of the chart can be mush. Before reaching for a vision-language model, check whether the underlying chart data or the source graph structure is sitting right there in the pipeline, because reading numbers back out of pixels is the hard way to get numbers you already had. Vision-language reasoning earns its place once that baseline hits layout, ambiguity, cross-references, or language a fixed rule can't handle without becoming its own maintenance burden. If you're piping screenshots through something like the gemini CLI's multimodal handling, that check happens before the prompt is written, not after the answer disappoints you.

What the eval log has to capture

The actual experiment is a sweep: vary image dimensions and crop strategy, and for every run record task accuracy, visual-token accounting, time to first response, and total cost. Keep the original asset untouched and log every transformation applied on top of it, because "the model got the image" means nothing without knowing which image it got. A useful eval record looks closer to this than to a model name and a score:

model ID, API/runtime, prompt revision
original dimensions, crop, resize, orientation
image count, ordering, frame timestamps
OCR or metadata supplied beside pixels
visual/input/output token accounting
first response, total latency, peak memory
grounded pass, unsupported claim, abstain, repair

If the provider exposes an image-detail or resolution control, log the value it actually used, not the default you assumed. Local models add their own way to fail quietly: the vision projector, the quantization, the context setting, and whatever preprocessing code sits in front of the weights all have to match, and a projector paired with the wrong text weights can keep answering fluently the whole time it's wrong. Running a vision-capable GGUF through Ollama makes that mismatch easy to miss: the server doesn't know it's wrong either, it just answers worse.

The acceptance gate is label transcription, relationship accuracy, numerical consistency, evidence citation, and abstention on anything genuinely unreadable. Score unsupported claims separately from missing information: they are not the same failure. A gap sends a human to look at the source. An invented serial number, control, or trend can trigger an action on its own with nobody checking first. Put genuinely unanswerable examples in the eval set and reward the model for saying so instead of guessing politely.

Confident, coherent, and wrong

The failure worth worrying about is a coherent description of a trend that has the axis, series, unit, or arrow direction flipped. Two traps are specific to this kind of input: downscaling a full page until the text stops being text, and uploading a whole sensor image to ask about one corner of it. Both survive a casual read. The answer matches what the reviewer already expected to see, so nobody stops to check.

Split the claim into layers and the trap gets easier to spot:

• "There is a red indicator at the upper right" is observation
• "It belongs to the network panel" is association
• "The connection has failed" is inference
• "Restart the gateway" is action

Each layer needs a different amount of evidence and authority, and the failure mode is letting a plausible observation walk straight into an action with nothing checked in between. Require a region, page, frame, or visible feature behind every important conclusion, and show that provenance to the reviewer instead of burying it in a log file nobody opens.

Multiple images multiply this risk. Label each one and say explicitly whether they show the same object, a page sequence, the same camera, or the same point in time. Drop duplicate frames, they burn tokens without adding evidence. If order matters, attach timestamps and test a shuffled version of the set: a model that tells the same causal story after you scramble the frames isn't using the sequence, it's pattern-matching on individual images.

Crop to the evidence, not the sensor

The rule that survives production is: crop to the evidence, keep enough pixels for it, cap everything outside it. That belongs in preprocessing, routing, or validation code, not prompt text somebody can edit without noticing the constraint was load-bearing. Store the model and pipeline version with every result so an upgrade can be replayed against the same images instead of trusted on faith.

Privacy applies on both sides of inference. Crop out rooms, faces, screens, addresses, and reflections before the image goes anywhere. Set retention separately for originals, thumbnails, OCR text, embeddings, prompts, and debug captures, because none of them need the same lifespan. And here's the part worth saying plainly: running the model locally doesn't solve this. It cuts external transfer, which is real, but it does nothing for an unencrypted disk or a dashboard account with more access than the job needs.

For hosted models, put upload time, regional routing, retention settings, rate limits, and provider errors into the same test as accuracy. For local models, test cold load time, projector memory, CPU or GPU placement, preprocessing time, heat, and concurrency. Compare the whole path at p50 and p95, not the tokens-per-second number a benchmark likes to lead with, because that number never includes the part where your image sat in a queue.

Stop tuning once the visual gate passes at an acceptable latency, cost, and abstention rate. More pixels and a bigger model aren't automatically the safer choice; they can expose more private detail per request and make the system more willing to answer past what the evidence supports.

Crop to what the question needs, log what got cropped, and let the model say it can't tell when the pixels run out.