A month of MLX as my daily local runtime
In late May I got annoyed enough at the pause before first token to actually do something about it. Long-context prompts through llama.cpp's Metal backend (a 9k-token code review payload, say) would sit for ten or twelve seconds on my 64 GB M3 Max before answering. A colleague had been running mlx-lm for months and was gently smug about it. I made it my default runtime for a month to see. It's still the default.
The mechanics are almost disappointingly small. One pip install gives you the two commands that matter: mlx_lm.generate for one-shots, mlx_lm.server for an OpenAI-compatible endpoint your existing tools can point at. The mlx-community org on Hugging Face carries 4-bit and 8-bit conversions of nearly everything popular, usually within days of a release.
pip install mlx-lm
mlx_lm.server --model mlx-community/Qwen3-Coder-30B-A3B-Instruct-4bit --port 8080the part nobody sells hard enough
Unified memory. On the 3090 box under my desk, every model decision starts with offload arithmetic: how many layers fit on the card, what spills to CPU, how much context I can afford before the whole thing slides off a performance cliff. On the Mac there is no arithmetic. The model file either fits in unified memory or it doesn't, and that's the entire decision tree.
I didn't realize how much energy the offload math was costing me until it was gone.
what actually got faster
Prompt processing, mostly. That 9k-token review prompt went from eleven-ish seconds to first token down to about seven. Generation moved less. The 30B MoE gained maybe 10 to 15% over its GGUF twin under Ollama, and a dense 8B was close to a wash. If your local use is chat with short prompts, you might shrug. My use is long prompts with short answers, so prefill is most of the wall clock, and MLX wins prefill on this hardware clearly enough that I stopped re-checking after the second week.
On a Mac the model either fits or it doesn't, and that one sentence quietly replaced a spreadsheet I'd maintained for a year.
the conversion tax
When mlx-community hasn't converted the model you want, you run mlx_lm.convert yourself with a quantize flag. It works, and I've come to dread it. It pulls the full-precision weights first (call it 28 GB for a 14B, far more above that), chews for a while, and leaves you with two copies of every model in different formats. My models folder crossed 340 GB in mid-June, and a depressing share of that is the same weights twice. On a 1 TB laptop that's a real line item.
the evening I blamed the wrong layer
A confession from week three. I loaded a 4-bit model whose file weighed about 39 GB (fits in 64 GB, on paper) and it thrashed so badly I wrote MLX off and went back to Ollama for two days. The real cause: macOS caps how much memory the GPU is allowed to wire, roughly three quarters of the machine by default, so my model straddled the limit and paged. There's a sysctl for it, iogpu.wired_limit_mb, and one line fixed everything. Two days of wrong conclusions because I'd skipped one paragraph of documentation.
what stays on llama.cpp
The GGUF side still has the deeper quant menu: the K-quant and imatrix gradations I walk through in the quant-picking piece simply don't exist for MLX, where you mostly get 4-bit or 8-bit, with little in between. Grammar-constrained output is more mature over there too. And GGUF is universal: the same file runs on the 3090 or a rented VPS. MLX weights serve exactly one hardware vendor. That's the trade, and I accepted it knowingly.
One more honest note: on battery this is brutal. A 40-minute refactoring session took about a third of my charge with the fans clearly audible, and sustained speeds sag as the chassis heats. The numbers are in my laptop thermals piece. Docked at a desk, a non-issue.
So the Mac now serves everything through MLX and the 3090 keeps llama.cpp, which is roughly where I guessed things would land when I wrote up the Apple inference stack. I just expected to be more annoyed getting here. A month in, the thing I notice most is other people's machines pausing before first token.