Vision models for camera-event understanding: resolution and visual-token budgets

Every consumer camera app I've used has at some point told somebody a stranger was in their yard when it was a trash can and a long shadow. That isn't a hardware problem. It's what the model was actually looking at, and that's rarely the frame you assume it saw.

Point a camera at a doorway and everyone assumes the model judges the picture a person would see. It doesn't. Between the sensor and the model sits a pipeline of resize, crop, tiling, and a hard cap on visual tokens, invisible in the answer you get back. The model can be fully grounded, half grounded, or reasoning about a detail that vanished before inference started, and the output reads the same.

The workload worth building around is camera-event understanding: doorway, workshop, warehouse, garden, safety-camera frames, classifying a bounded event from selected frames without repeating the alert or exposing more of somebody's kitchen than necessary. Feed it motion-gated frames, timestamps, a zone label, and a short fixed vocabulary of events. Narrower than "describe this picture," and narrow is what makes it measurable.

The pixel budget nobody sets on purpose

My rule: useful pixels are whatever resolves the smallest piece of evidence the task cares about, not whatever the sensor outputs. A keypad needs enough resolution to make out the digits; a person-shaped blob crossing a driveway doesn't. More pixels than the task needs doesn't make the model smarter, just slower and hungrier for tokens and memory.

Before reaching for a VLM, run the boring stuff first: motion detection plus a small dedicated object detector catches most events cheaper than a VLM ever will. Bring the VLM in only for what that can't resolve: ambiguous layouts, object relationships, anything needing actual language reasoning.

Log the transform, not just the answer

That means sweeping image dimensions and crop strategies, recording accuracy, visual-token counts, time to first response, and total cost. Keep the original asset and log every transformation applied before the model saw it.

What needs to be in that record:

• model ID, API or runtime, prompt revision
• original dimensions, crop, resize, orientation
• image count, ordering, frame timestamps
• token accounting: visual, input, output
• first response time, total latency, peak memory
• grounded pass, unsupported claim, abstain, or repair

If the provider exposes an image-detail setting, log the value it used, not the default you assumed. Locally, log the vision projector, quantization, context length, and preprocessing, since mismatched text weights and projector fail quietly. Ollama's vision path is an easy place to get this wrong.

The gate for shipping is event precision and recall, duplicate-alert rate, latency, privacy retention, and clean abstention. Score an invented answer differently from a missing one: "insufficient evidence" costs a review, but an invented serial number, control, identity, or attribute can trigger a wrong action unchecked. Put unanswerable frames in the test set and reward the abstention.

From red light to restart command

The failure that bites is turning one uncertain visual clue into a confident identity, hazard, or security conclusion. The trap: someone downscales a page until text disappears, or uploads a whole sensor image to ask about one corner of it. It sails through review because the answer matches what people expected. Require a region, frame, label, or visible feature behind every conclusion, and show that provenance to the reviewer.

Keep four layers separate. A red indicator at the upper right is an observation. That it belongs to the network panel is an association. That the connection failed is an inference. Restarting the gateway is an action. Don't let a plausible observation jump straight to an irreversible one.

Multiple images multiply that risk. Label each one, and state whether they show the same object, a page sequence, the same camera, or different times. Cut duplicate frames burning tokens without adding evidence. If order matters, attach timestamps and run a shuffled control: the same story after scrambling the frames means it isn't using the sequence.

Where the crop happens matters more than the model

Put the actual rule into code, not prompt text: crop to the evidence, keep enough pixels to resolve it, cap everything outside that region. Prompt prose is a request; preprocessing, routing, validation, and authorization code is a rule that survives a model swap. Store the model and pipeline version with every result.

Privacy applies on both sides of inference. Crop out the neighbor's window, the face outside the event, the open inbox, the reflection in the glass door. Set retention separately for originals, thumbnails, OCR text, embeddings, prompts, and debug captures, since they leak on different timelines. A local model cuts transfer risk. It does nothing for an unencrypted disk or an overbroad dashboard account.

Hosted, put upload time, regional routing, retention, rate limits, and provider errors in the same test as accuracy, the way you would for Gemini's CLI multimodal path. Local, add cold load time, projector memory, CPU or GPU placement, preprocessing time, heat, and concurrency. Compare the whole path at p50 and p95, not the number that looks best in a slide.

Stop optimizing once the gate passes at acceptable latency, cost, and abstention. A bigger model and more pixels aren't automatically safer; mostly they expose more private detail per request and a system more willing to answer past what the evidence supports.

The rule I'd keep, if I had to keep exactly one: crop to the evidence, not to the sensor.