On-device and edge AI: running models where the cloud can't reach

Capable small models plus better runtimes (Core ML, ONNX, TFLite, llama.cpp) mean real AI now runs on phones, laptops, and embedded devices, not as a degraded cloud but as a fundamentally different deployment with its own rules. And for a whole class of products, "it runs on the device" is the feature.

Why on-device: the reasons worth the constraints

• Privacy. The data never leaves the device. For health, personal, or regulated data, that's sometimes the only acceptable architecture. There's nothing to leak because nothing was sent.
• Latency. No network round-trip means instant. For real-time UX (smart keyboard, live camera intelligence, voice), the cloud's latency is the difference between magic and unusable.
• Offline. It works with no connection. Planes, tunnels, factories, the field.
• Cost. Zero marginal inference cost: the device pays, not your API budget. At scale, that inverts the cloud's per-token meter entirely.

If one of those is your product, on-device is the answer. If none of them is, the cloud is easier. Be honest about which you're in.

The constraints: the hard part

• Memory and compute. A phone is not a workstation. You're in small-model territory (often sub-7B), heavy quantization (int4 and sometimes below), or a task-specific model, not the frontier.
• Battery and thermals. Sustained inference drains and heats a device. The fix is to run on the dedicated accelerator (Core ML scheduling onto Apple's Neural Engine, or the device's NPU), which does inference at a fraction of the power of the CPU/GPU (why Core ML exists).
• App size and download. Shipping a multi-gigabyte model inside an app is a real product constraint: download size, update size, storage. It shapes which model you can even consider.

How it's done

• Quantize to fit and to run fast: smaller weights mean both "it fits in device memory" and "it generates faster" (inference is bandwidth-bound).
• Convert to the device format: Core ML (Apple), ONNX / TFLite (cross-platform), and target the accelerator, not just the CPU.
• Distill a small specialist. The strongest move: use a big model to generate training data and fine-tune a tiny model for the one job the device needs. A small model that does one thing well beats a general model that doesn't fit.
• Go hybrid. A small on-device model handles the common, private, instant case; escalate to the cloud for the hard case. It's the cascade drawn across the network boundary. It gives you privacy and speed by default, with frontier capability on demand.

Where it shows up

On-device transcription and dictation, smart keyboards and autocomplete, camera and photo intelligence, offline assistants, translation, and embedded or industrial inference where connectivity isn't guaranteed. The common thread is always one of the four reasons above doing real work.

The honest take

Edge AI is a deployment where privacy, latency, offline, or cost is the point, not a smaller cloud, and where the engineering challenge is fitting a capable-enough model into a tight box: the right quantization, the right runtime, the accelerator, often a distilled specialist. When one of those reasons is your product, on-device stops being a limitation and becomes the differentiator your cloud-only competitors can't match. When none of them is, reach for the cloud and save yourself the constraints. Know which situation you're in before you start. That decision is the whole architecture.