Why the Neural Engine rarely runs your chat model

Apple's neural coprocessor is rated at a healthy multiple of trillions of operations per second, and on a Mac running llama.cpp or MLX for local chat it does almost nothing. That's not a defect in the chip. It's a plain fact about which software paths reach it, and you should know it before you buy hardware expecting the Neural Engine to quietly absorb your inference load the way a spec sheet implies.

The gap between what a machine can compute and what your runtime can reach is the whole story here. A private chat you poke at twice an hour, a coding loop firing on every keystroke pause, and a nightly batch extraction job can run on the same box and still want three different things from it.

Adding up tera-ops is how you get fooled

The easy mistake is adding the CPU peak, GPU peak, and ANE peak into one shared budget and assuming decoding draws from all three. It mostly doesn't. llama.cpp and MLX run on CPU threads and Metal kernels; reaching the ANE takes a CoreML conversion path with its own operator support and quantization limits, and most quantized weight formats people actually run never go near it. The chip sits there, rated and idle, while decoding happens somewhere else entirely.

This survives as folk wisdom because the output still looks fine. Nothing crashes. Nobody sees an error that says "ANE unused." Tokens stream, the answer arrives, and the assumption that all the silicon is pulling its weight goes unchallenged until someone profiles it. It's the same blind spot behind Ollama's GPU offload math: you can't add capacities together and expect a scheduler to spread work across parts never wired to cooperate.

What the stopwatch should actually catch

Before reconfiguring anything, or buying a bigger machine, write down a baseline: one named model, one fixed prompt set, the exact server command, plus the model artifact and prompt template, both easy to lose track of and both large enough to quietly invalidate a comparison you make weeks later.

Then measure the request as it moves through the whole system, not the theoretical ceiling of its parts. Time to first token catches loading and prompt work. Steady-state token rate describes decoding. Completion time is what the person on the other end actually experiences. Add peak memory and queue delay when they change the decision, wall power if the machine runs around the clock. For a batch job, count completed valid jobs per hour; for an interactive one, count the waits long enough that someone notices.

Keep the record plain:

• artifact, runtime, exact launch flags
• workload and a fixed input set
• cold start, warm start, p50, p95
• peak memory and wall energy
• quality failures and abstentions
• decision, owner, retest date

Drop that last line and the whole exercise turns into trivia. A number nobody acted on is not a benchmark. It's a screenshot with extra steps.

Soft failures don't ring any alarm

Local inference fails quietly more often than it fails loudly. A model partially offloads to CPU without telling you. A cache misses and nobody notices for a week. Swap creeps up under memory pressure. A queue keeps holding requests whose client already disconnected. A fallback path changes where your data actually goes, which matters if keeping it on the machine was the whole point. None of this throws an exception, and the text still comes out looking plausible.

Watch runtime logs and OS-level counters while you test, not just the number at the end. Change one variable at a time unless you're deliberately comparing two full configurations. Run it more than once, since a single fast run tells you nothing about luck, and read the actual outputs rather than trusting speed as a stand-in for correctness. Buying is a separate question from tuning, and general guidance on picking hardware for local models covers that side better than a spec sheet will.

Operational simplicity belongs in the same column as latency. An optimization that needs hand repair after every driver update or model refresh isn't free, whatever the benchmark said on measurement day. Favor the boring setup, the one that reproduces from a service file or a short script, over one that only works because you remember which flags you passed last Tuesday.

Buy and configure for the paths you've actually measured working, and treat any future ANE support as a bonus, not something you're counting on already. Visible limits can be scheduled around, routed around, priced around. Invisible limits turn into a late-night page and an emergency order.

Stop tuning once the workload clears its quality and latency bar with headroom to spare. That margin isn't waste; it's what absorbs the longer document, the extra user, the hot afternoon, the next runtime release. Spend model capacity only where it changes what the user gets back. Nothing else earns the cycles.