Vision models for product-image analysis: local, hosted, and hybrid vision deployment

A catalog photo lands in the pipeline at whatever resolution the seller uploaded it in, and before any model touches a single pixel it gets resized to fit an encoder's fixed input size, tiled into patches, sometimes cropped to a center square, and packed into a token budget somebody else chose.

None of that shows up in the answer you get back. The model writes a confident sentence about the product regardless, and it can be fully grounded, half grounded, or built around a detail cropped out three transforms before inference even started. Nobody checks this by default.

For product-image work the material is catalog photos, user uploads, packaging shots, defect photos, and visual-search queries, and the job is almost always one of four things: describe an attribute, compare two items, flag a bounded defect, or retrieve something similar. Feed it multiple controlled views, existing catalog metadata, the allowed attribute values, and some signal about image quality. That's narrower than "understand this image," and narrower is the point: a bounded visual question is the only kind you can actually measure.

The resize happens before the reasoning does

Deployment should route on sensitivity, image size, task difficulty, where the hardware lives, and how much time you have. Start with the boring baseline: image embeddings plus deterministic catalog filters, no generation involved. A lot of product-image work is genuinely nearest-neighbor lookup with a rules layer on top, cheap and auditable in a way no language model output will ever be. Bring in a vision-language model only once that baseline hits layout, ambiguity, relationships between objects, or free-text language that fixed rules can't handle without turning into a pile of exceptions. Whether that's Gemini CLI's multimodal support hosted, or local vision through Ollama, the routing decision happens before either tool sees the image.

Write down what the model actually looked at

The real experiment is running the same visual gate against a local model and a hosted one, side by side, counting upload time and review as cost, with explicit escalation rules. Keep the original asset untouched and log every transformation before the model call:

model id, runtime or API version, prompt revision
original size, crop, resize, orientation applied
image count, order, any frame timestamps
OCR or metadata passed in alongside the pixels
input and output token counts
time to first token, total latency, peak memory
grounded / unsupported claim / abstain / repair

If the provider exposes a resolution knob, log the value you actually sent, not the default you assumed. For a local model, record the vision projector, quantization, context length, and preprocessing code path: text weights from one release paired with a projector from another can fail in a way that's easy to miss, because the server keeps returning fluent language right up until the content stops matching the picture. The acceptance gate is attribute precision, retrieval relevance, defect recall, and consistency across views, plus a metric people skip: unsupported-claim rate, scored apart from missing information. A model that says it can't tell from this angle is doing its job. One that invents a serial number or a material it has no basis for looks like a normal answer, so put unanswerable examples in the eval set and reward it for admitting that.

One sentence, four different claims

The failure that matters most is the model inferring material, authenticity, dimensions, or condition from pixels that can't support it, in the same confident voice it uses for things it actually can see. A second, more operational trap sits next to it: silently sending a sensitive image to a hosted model because the local one failed, or routing every thumbnail to an expensive frontier API because nobody wired up the cheap path. Both survive casual review, because the output usually matches what a person expects to see anyway. Require a region, a frame, a label, or some visible feature behind every conclusion that matters, and show it to the reviewer instead of burying it in a log.

It helps to keep four layers separate, in the prompt and in your head: observation, association, inference, action. "There's a red indicator in the upper right" is observation. "It belongs to the network panel" is association. "The connection has failed" is inference. "Restart the gateway" is action. Each needs a different amount of evidence and authority before it's allowed to happen. Guard against a plausible observation sliding straight into an action nobody can undo.

Multiple images add an identity problem on top. Label each input, say whether they're the same object, a page sequence, one camera, or one time period, and drop duplicate frames that just burn tokens. Where order matters, attach timestamps and run a reordering test: a model that tells the same causal story after you shuffle the frames isn't using the sequence, it's narrating after the fact.

The boundary belongs in code, not in the prompt

The rule that holds up is simple to state and annoying to build: keep privacy rules hard, make every remote escalation visible to a human, and compare accepted visual tasks per dollar and per second, not raw tokens per second. That rule has to live in preprocessing, routing, or authorization code, because prompt text is a suggestion the model can and will ignore under load. Store the model and pipeline version with every result so an upgrade can be replayed against the same images later.

Privacy work happens on both sides of inference, and it needs separate handling for each kind of artifact you're keeping around:

• originals, kept only as long as the business case needs them
• thumbnails and derived crops, a shorter window than the original
• OCR text and metadata pulled off the image
• embeddings, which encode more than people assume
• prompts and responses escalated to a hosted model
• debug captures, which quietly accumulate the most sensitive material of all

A local model cuts external transfer, full stop, but does nothing for an unencrypted disk or an over-permissioned dashboard account on top of it.

Hosted tests need upload time, regional routing, retention settings, rate limits, and provider error rates, not just generation speed. Local tests need cold-load time, projector memory, CPU or GPU placement, preprocessing time, heat under sustained load, and concurrency. Compare the whole path at p50 and p95: a model fast on the happy path and slow at the tail isn't the fast option.

Stop tuning once the visual gate clears at a latency, cost, and abstention rate you can live with. A bigger model and more pixels aren't automatically safer; they can expose more private detail per request and make the system more willing to answer past what the image settled. I'd rather ship the version that abstains more than I'd like and kicks a slice of cases to a human than the one that answers everything smoothly and gets the unanswerable ones wrong with a straight face. That trade costs real review hours I could spend elsewhere. I'm keeping that cost on purpose.