Batch size on Apple Silicon is a memory decision

Crank the batch size up on an M-series Mac and you get free throughput, until you don't. That headroom is borrowed against the pool your browser, your IDE, and macOS draw from. A discrete GPU hits a wall and tells you. Unified memory doesn't: one number, and batching decides how much goes to inference before anything else gets a turn.

One pool, three tenants

What competes for that pool on an ordinary week: a private chat wanting a fast first token, a coding loop wanting the same plus consistency, a nightly job wanting only throughput while you sleep. Capture a baseline first: one named model, one fixed prompt set, the exact server command, the model artifact, the prompt template, the two things everyone forgets and big enough to wreck the comparison. Pin the runtime too (MLX or CoreML); swapping mid-comparison throws the exercise away. Sweep batch size against memory pressure, swap, throughput, and how the machine feels. Time to first token catches loading, steady rate describes decoding, completion time is what the user gets. A batch job wants valid jobs per hour; an interactive one wants the slow waits a person remembers. Keep the record plain: artifact, runtime, launch flags, workload, input set, cold/warm-start latency at p50/p95, peak memory, wall energy, quality failures or abstentions, an owner, a retest date. No decision, no point: that's trivia, and an unowned setting is folklore.

Plausible output hides the real cost

The mistake I see most: taking a higher jobs-per-minute number that quietly pushes macOS into sustained swap. It survives review because the output still reads fine, the model has no idea its host is starving, so it keeps producing plausible text while the disk grinds underneath it. Local inference fails softly: a model can partly offload onto the CPU, a cache can miss silently, a queue can keep holding requests from disconnected clients, or a fallback path can quietly change your privacy boundary. None of that trips an exception. Watch runtime logs and OS counters while it runs, change one variable at a time, and run it more than once, a single fast pass proves nothing. Read a sample of the outputs instead of trusting speed as proof of equivalence, and report any quality cost. Simplicity counts too: an optimization needing manual repair after every driver update isn't free, just deferred. I'd rather run a boring configuration I can rebuild from a service file or short script, artifacts out of the report, hashes and commands in.

Headroom is doing a job

The rule I keep coming back to: leave enough unified memory that the machine still behaves like a computer, not an inference appliance. Less exciting than picking the biggest model that fits, but it leaves a system whose limits you can see. Visible limits can be routed around or priced into a plan. Invisible ones show up later as unexplained waiting and an emergency order nobody approves. Stop tuning once the workload hits its target with headroom left over, and don't treat that margin as waste: it absorbs a longer document, an extra person on the machine, or whatever the next release of mlx-lm costs you on upgrade day. Efficient local inference is mostly spending model capacity only where it changes what the user gets back. What I still haven't worked out is separating the two kinds of swap Activity Monitor shows: memory compressed and sitting there harmlessly, and memory paging to the SSD. The counter conflates them, the model server can't tell which one it's causing, and until I find a lower-level way to watch that boundary, every sweep I run flags ordinary compression as the real problem.