Read macOS memory pressure during inference
Size for stable green memory pressure under the actual workload, not for the one clean load that impressed you in a terminal window. That's the rule, stated up front, because everything below is just me earning it.
macOS treats RAM as something to spend, not hoard. It compresses aggressively, swaps without asking, and will happily report gigabytes of "free" memory that would evaporate the second you actually asked for them. On a unified-memory machine running a 14B-class GGUF next to an IDE and a browser with too many tabs open, that free number tells you almost nothing about whether the box is about to choke. What tells you something is pressure, compressed-memory growth, and whether swap keeps climbing while tokens are still streaming out.
The workload matters more than the model name. A private chat opened twice an hour behaves nothing like a coding loop hammering the same context window, and neither behaves like a nightly extraction job chewing through a folder of PDFs while nobody's watching. All three can share a machine, and none should be sized the same way. A long-running agent that keeps accumulating context instead of resetting between turns changes the math again, a separate problem I get into in agent memory architecture.
Pin the model, the flags, and the prompt set before you touch a knob
Before changing anything, write down what's running: the model artifact, the runtime, the launch flags, and a fixed prompt set you'll reuse every time. Skip this and your comparisons stop meaning anything within a week, once you've forgotten which quantization you tested and end up arguing with a past self who left no notes.
Then watch the whole session, not one number. Time to first token is loading and prompt work; steady token rate is decoding once warm; completion time is what the person waiting feels. Add peak memory and, when it matters, wall power. Automated work gets scored on completed valid jobs per hour; interactive work gets scored on the slow waits a person remembers days later. Log each run against the model, runtime, flags, cold and warm timings, memory, any failures or abstentions, and a decision with a name and a retest date on it.
A measurement nobody has to act on is just decoration.
A model that launches is not a model that survived the week
The trap is calling a setup comfortable because it started clean and produced plausible text. Low bar. Local inference fails softly more than loudly: a model partly offloads to CPU, a cache misses without complaint, swap creeps up through the afternoon, a queue holds onto requests nobody's waiting on, or a fallback path quietly changes where your data goes. None of that throws an error. It shows up in the counters, and only if you're watching them.
So watch the counters, not just the stopwatch. Change one variable at a time unless you're deliberately comparing whole configurations. Run it more than once, because a fast single pass proves luck, not durability. Read what the model actually produced instead of trusting speed as a stand-in for correctness, and if a change buys latency at the cost of quality on the task that matters, write that cost down next to the win.
Favor boring. A setup needing manual repair after every driver or model update isn't saving you anything, it moved the cost to a Tuesday you didn't plan for. A service file or short script you can rerun beats a clever one-off tuned by hand at midnight. Keep raw logs out of the report itself, but keep the hashes and exact commands, or the next person rerunning your numbers is rerunning a different experiment without knowing it.
I'll admit the catch: "stable green pressure" is a judgment call, not a threshold macOS hands you, and the first few times you read these counters you will misjudge what stable looks like. You get calibrated by living through a few bad weeks on the machine, not by reading a rule someone else wrote down.