Vision models for camera-event understanding: preprocessing before the vision model

Preprocess before you upgrade the model. Rotation, cropping, contrast, frame selection, and the metadata already sitting beside the pixels usually buy more accuracy per dollar than a bigger vision model does, and that's the rule the rest of this earns.

A photo looks concrete, so the confidence problem is easy to miss: the model never sees the photo, only a resized, cropped, tiled, token-budgeted stand-in, and a fluent answer can be grounded in a detail cropped away before inference started. Camera-event work means doorway, workshop, warehouse, garden, and safety-camera frames: classify a bounded event from selected frames, protect privacy, stop firing the same alert twice, using motion-gated frames, timestamps, zone metadata, and a short event vocabulary, not a raw stream and an open prompt.

Run motion detection and a plain object detector first. Let a vision model in only once that baseline hits layout, ambiguity, object relationships, or language rather than geometry. Most events never get there, and narrowing inputs this far is what makes failure mode measurable, not argued about.

Log the crop, not just the model name

Version the pipeline like anything else that changes output. Compare raw, normalized, cropped, and OCR-augmented frames as separate conditions, keep the original untouched, and record every transformation. The evaluation record needs more than a model name and a score:

model, API/runtime, prompt revision
dimensions, crop, resize, orientation
frame count, ordering, timestamps
OCR/metadata supplied beside pixels
input, output, visual token counts
latency, peak memory
grounded / unsupported / abstain / repair

If the provider exposes an image-detail or resolution control, the way gemini-cli-multimodal does, log the value sent, not the default. Local through ollama-vision-local needs its own row: vision projector, quantization, context setting, and whatever preprocessing the runtime applies. Mismatched text weights and projector degrade quietly while the server keeps answering fluently: nothing crashes, it just gets dumber, caught only in a grounded-pass rate, not uptime.

The acceptance gate is event precision and recall, duplicate-alert rate, latency, privacy retention, and safe abstention. Score unsupported claims apart from missing ones: an omission gets caught in review, but an invented serial number or identity can trigger the wrong action unchecked. Put unanswerable frames in the set and reward the model for saying so.

Four kinds of claim, four different amounts of trust

The failure that costs you is one uncertain clue turning into a confident identity, hazard, or security conclusion. There's a trap specific to the lens too: aggressive enhancement that makes text look sharper while it quietly invents a character or papers over a real defect. Both survive casual review because the answer matches what's expected. Require a visible feature behind every conclusion, shown to the reviewer, not buried in a log.

Split conclusions into four layers: "red indicator, upper right" is observation, "belongs to the network panel" is association, "connection has failed" is inference, "restart the gateway" is action. Each needs its own evidence; a plausible observation isn't grounds for an irreversible tool call. Multiple images add identity risk: label inputs, state whether frames share one object, camera, or time period, drop duplicates, and test a shuffled frame order on purpose, since a model telling the same story after a scramble isn't using the sequence.

Keep the original retrievable and every transformation reproducible from code, not prompt prose, and store pipeline versions so an upgrade can be replayed. Crop irrelevant rooms, faces, and screens first, and set retention separately for originals, thumbnails, and debug captures; a local model stops external transfer, not an unencrypted disk. Test hosted paths on upload time and rate limits, local ones on cold loading and heat under concurrency, then compare at p50 and p95, not tokens per second.

Stop once the gate passes at acceptable latency, cost, and abstention. A bigger model isn't automatically safer; it exposes more private detail and more room to answer past the evidence, and I wouldn't chase one past the gate. What I'd check next: pull the transcripts where it didn't abstain and ask whether each conclusion has a labeled region behind it, or just a sentence that sounds right.