Vision models for camera-event understanding: prompts grounded in visible evidence
A vision-language model never gets the frame you saved to disk. It gets whatever survived resizing, cropping, tiling, and a token budget, and nothing in the reply says which pixels made the cut. A fluent answer can be fully grounded, half grounded, or built around a detail that vanished before inference started, and from outside those look the same.
For camera-event work the material is ordinary: doorway, workshop, warehouse, garden, safety-camera frames, footage nobody watches until something breaks. The job is narrower than "understand this image": classify a bounded event from motion-gated frames, hold onto privacy, and don't fire the same alert five times for one event. Feed it timestamps, zone metadata, a short fixed vocabulary, not an open "what do you see."
My rule: a VLM goes in last, not first. Motion detection and a plain object detector clear most of the volume cheaply. Only the residue a fixed rule can't classify economically reaches a model that reasons in language.
the log line that tells you if the answer is real
Ask for observable evidence, a location, a confidence or answerability signal, and a constrained conclusion, in that order. Keep the original asset and record every transformation applied before the call. A model name and accuracy score tell you almost nothing alone.
| What you log | Why it matters |
|---|---|
| Model ID, runtime, prompt revision | Ties a result to a config you can reproduce |
| Dimensions, crop, resize, orientation | Shows what the model actually received |
| Frame count, order, timestamps | Tests whether sequence changes the answer |
| Metadata passed beside the pixels | Separates text-grounded claims from visual ones |
| Tokens, first-response time, memory | Prices the accuracy, not just reports it |
| Grounded, unsupported, abstain, repair | The real outcome, not correct-or-incorrect |
Log the resolution value actually sent, not the default assumed; scripting the calls yourself makes that easy, the approach in gemini-cli-multimodal. The same stack on your own hardware has its own burden: vision projector, quantization, context length, whichever preprocessing path ran, covered in ollama-vision-local. Mismatch text weights and projector, pulled from two different releases, and the server keeps answering fluently while quietly getting worse.
observation is not the same claim as action
Split every prompt into four layers, don't let them collapse into one. "There's a red indicator, upper right" is observation. "It belongs to the network panel" is association. "The connection has failed" is inference. "Restart the gateway" is action. Each needs different evidence, and the failure I see most is a plausible observation walking straight into action with nothing checking between.
The trap is one uncertain clue turned into a confident identity, hazard, or security conclusion, because an open question rewards a polished story the frame never proves. It survives review because it matches what a person expected to see. Require a region, frame, or label for every conclusion that matters, and show that pointer to the reviewer, not a log nobody opens.
Score unsupported claims separately from missing information. "I can't tell" costs a review cycle. An invented serial number, UI control, or identity triggers the wrong action unnoticed. Put unanswerable examples in your eval set and reward abstention; a model that never says "not visible" hasn't been tested on the cases where the answer is no.
Multiple frames add a second failure axis. Label each, state whether they show the same object, camera, or time sequence, and strip duplicates that burn tokens without evidence. If order matters, run the set shuffled as a control: a model telling the same story regardless of order isn't using the sequence, it's pattern-matching content.
what local doesn't buy you, and what I still haven't fixed
The rule that every conclusion must point to a visible region, frame, or fact belongs in preprocessing and validation code, not prompt wording, and results should carry model and pipeline versions, so an upgrade can be replayed later.
Privacy applies on both sides of inference. Crop rooms, faces, screens, and reflections unrelated to the event before sending, and set retention separately for originals, thumbnails, and embeddings. Running the model on your own hardware cuts network transfer risk, but does nothing for an unencrypted disk or an overbroad dashboard account.
Measure hosted and local paths differently: hosted upload time, regional routing, retention, rate limits, error modes under load; local cold load, projector memory, GPU placement, preprocessing time, concurrency. Compare the whole path at p50 and p95, not just tokens-per-second.
Stop tuning once the event gate passes at acceptable latency, cost, and abstention rate. A bigger model or higher resolution isn't automatically safer: it can expose more of the frame than needed and make the model readier to answer past what it saw.
The part I haven't solved: the crop-before-inference step, meant to protect privacy, still needs a person to decide what's irrelevant before any model looks at the frame. At the volume this is meant to run at, that call doesn't scale, and I don't have an automated stand-in I'd trust yet.