Metal-backed llama.cpp or MLX?

People pick a framework off a benchmark screenshot and find out weeks later that the model they need was never exported to that format. It happens with llama.cpp and Metal too: someone locks into GGUF because it's everywhere, then hits a repo that only ships MLX weights. Neither framework is wrong. The mistake happened earlier: benchmarking speed before checking which formats and operations the application needs.

Metal-backed llama.cpp and MLX are both solid ways to run models on Apple Silicon, and picking between them on throughput alone skips the question that bites you later: does the model you actually need exist, already converted, in a format your runtime reads?

The format decides before the benchmark does

Local AI conversations tend to start with a model name or a peak tokens-per-second number, the wrong end of the problem to grab first. Start with the work waiting behind the API instead: a private chat firing a couple of times an hour, a coding loop running nonstop through a session, a nightly extraction job chewing through a folder of documents. Same Mac, same GPU, three different jobs, each wanting something different from the runtime underneath it.

Before touching a config file, write down a baseline: one named model, one fixed prompt set, the exact server command you ran. Attach the model artifact and the prompt template to the record, not just a description. Those two get lost first, and losing them is enough to invalidate a comparison months later.

model + runtime + launch flags
workload + fixed prompt set
cold start / warm start / p50 / p95
peak memory + wall power
failures and abstentions
decision, owner, retest date

A benchmark without a decision attached is trivia. A setting without an owner turns into folklore by the time anyone asks about it again.

Three jobs, one Mac, three different right answers

Compare model conversion, prompt templates, API integration, memory footprint, and sustained speed on identical tasks, and follow the request through the whole system, not one stage of it. Time to first token covers loading and prompt processing. Steady-state token rate covers decoding once the model is warm. Completion time is what a person or a downstream job experiences, and it's the only one of the three that tells the full story alone.

Add peak memory, queue delay, and wall power when they affect the decision, and settle your hardware ceiling first. For the nightly job, what matters is completed valid jobs per hour. For the chat and the coding loop, what matters is the wait a person notices. Optimize the wrong one and you get a system that benchmarks well and still feels slow.

The slowdown that never throws an error

The tempting mistake is picking a framework off one benchmark number and rebuilding half the application around a feature the other quietly lacks. It survives code review because the system still produces plausible text. Nothing crashes. That's the trap: local inference fails soft far more often than it fails loud.

• a model partly offloads to CPU and latency quietly creeps up
• a cache quietly stops hitting
• swap grows underneath a process that still looks healthy
• a queue keeps holding requests whose clients already gave up
• a fallback path quietly changes where the data gets processed

None of those raise an exception. Watch runtime logs and OS-level counters while a test runs, not just the number at the end. Change one variable at a time unless you're comparing two complete configurations, and repeat the run more than once: one fast pass says nothing about whether an improvement is durable or just luck. Read the actual outputs, too. A faster wrong answer isn't progress, so if a change speeds things up but makes an important task worse, write that cost down next to the gain.

The runtime you can still explain in six months

Operational simplicity earns its own line on the sheet. A clever optimization that needs repair after every driver or model update isn't free, it's a subscription you keep paying. Favor the boring configuration you can reproduce from a service file, a container definition, or a short script someone else could read cold, teammate or future you. Keep raw artifacts out of the written report, but keep the hashes and exact commands, so a rerun later means the same thing it meant today.

The rule I'd carry forward: run the simplest runtime that covers your whole workflow, not the fastest one that covers most of it. That's less exciting than buying the bigger card, though plenty of people running MLX as a daily driver would agree. But it gives you a stack whose limits you can see. A visible limit gets scheduled around, priced, or handed to a bigger machine on purpose. An invisible one just shows up one afternoon as an outage nobody can explain.

Stop once the workload clears its quality and latency bar with real headroom left over, not the bare minimum. That spare room isn't waste; it's what absorbs a longer document, one more concurrent user, a hot afternoon when the fans are already working, or whatever the next runtime release changes underneath you. The tradeoff I take on purpose: give up some peak speed on paper for a setup boring enough to still explain to myself in six months. I'd rather lose an afternoon to benchmarking than a Tuesday firefighting a fallback path nobody wrote down.