Your laptop is lying about its tok/s
The demo was going fine for the first ten minutes. Late May, a client's conference room in Brno, my fanless MacBook Air running a 14B coder model through mlx-lm, me showing off that code review could happen without a single byte leaving their building. Around minute twelve the answers started arriving like someone was typing them by hand. By minute twenty I was narrating over silence.
Nothing was broken. A passively cooled slab of aluminium was doing exactly what physics says it must.
the first minute is a lie
Back home I measured it properly instead of guessing. Cool machine, 4-bit quant, one long generation looping for half an hour, throughput sampled every minute. First minute: 29 tok/s. Minute five: 24. Minute ten: 21. Minute twenty: 18.5, and it stayed pinned there for as long as I cared to keep the loop running. That plateau is the real number for any session longer than a coffee. Every figure I had ever quoted off this laptop was a first-minute number, because benchmarks are short and heat is slow.
The mechanism is boring. The SoC warms up, there is no fan to move heat anywhere, the chassis soaks, and firmware pulls the clocks down to keep the silicon inside its thermal budget. On my Air the sustained plateau lands somewhere between 60 and 70% of burst speed depending on the room. A fan-equipped MacBook Pro reportedly holds closer to 90%, but those are a colleague's runs, not mine, so treat them as hearsay.
A tok/s number without a duration attached is marketing, even when the only person you're marketing to is yourself.
the evening I blamed a model update
On battery the Air held up better than I expected; Apple silicon doesn't slash throughput unplugged the way my old ThinkPad did, where pulling the cord instantly cost about a third. What did hurt was Low Power Mode, still switched on from a flight two days earlier. It capped the whole curve roughly 25% lower from the very first token, and I spent an entire evening convinced that a quant I'd re-downloaded had regressed. I diffed checksums. I re-converted the model from safetensors. The fix was a checkbox in System Settings, and I found it by accident while looking for something else.
That one stung.
what helped, and what was cope
The most effective fix was the one I resisted longest: a smaller quant. Dropping from 6-bit to 4-bit cuts memory traffic per token, which cuts power, which raises the thermal plateau. I gave up a little sharpness and got back roughly 15% of sustained speed. Trimming context helped almost as much: prefill is the hottest phase, and I'd been re-feeding an 8k-token prompt every single turn out of laziness. Measuring it honestly is what convinced me: ten-minute warm runs, or the number doesn't count.
The cooling pad was cope. Four hundred crowns of whirring plastic moved my plateau by about one token per second, because the bottleneck sits between die and case, not between case and room. Propping the machine on two pencils did nearly as much for free.
For demos I stopped fighting physics. I warm the machine up for ten minutes beforehand and quote the warm number, and when a client asks why my figures look lower than some video's, I explain thermal soak. That conversation has built more trust than the missing ten tokens per second ever would have.
desktops don't have this problem
The old 3090 in my closet sustains within a few percent of its burst number indefinitely, because it has airflow measured in actual fans. That's the argument for dedicated hardware that never appears on a spec sheet: a desktop benchmark is a promise about every minute, a laptop benchmark is a promise about the first ninety seconds.
I still run models on the Air every day — mlx-lm is my daily driver for exactly the private-by-default work that client wanted. I've just stopped believing its opening act.