Huge pages are a measurable optimization, not a ritual

A CPU walks a multi-level page table on every TLB miss, and a file mapped into memory at gigabyte scale, 4 KB per entry, is going to miss a lot. That's the physical case for huge pages: fewer, bigger entries, fewer walks, less time spent translating addresses instead of running the model. It's a real mechanism. It's also nowhere near the biggest lever on your inference box, and flipping it once and never checking again is how a config line outlives anyone who remembers why it's there.

Name the metric before you flip a switch

Picture a Linux host serving memory-mapped GGUF models, several rotating through RAM as requests arrive, storage fast enough that loads aren't the bottleneck, one runtime process holding each mapping open while it's hot. That's the workload where address translation matters, because the mapping is huge and every fault on a cold load walks the table again. Before touching a sysctl or a launch flag, write down what you're actually trying to move: first-token latency, accepted jobs per hour, how many models stay resident at once, energy per finished task, or fewer corrections downstream. "Make it faster" doesn't say when to stop, and it will let through a config that's faster and wrong.

Freeze everything except the one thing you're testing

A baseline only means something if it reproduces, so pin the model artifact, tokenizer, prompt template, runtime build, exact launch command, and sampling settings, written down somewhere that isn't your head; treat it the way you'd document llama.cpp server flags for a teammate. Pull a small set of inputs from the real workload, including the annoying ones users actually send. Run once cold, storage untouched and model unloaded, since that's the path a real user hits first, then run warm long enough for cache effects and queueing to show up. Measure by phase, not one total that hides where the time went:

• queue wait
• model load or activation
• prompt processing and prefill
• time to first token
• decode rate and completion time
• peak RAM, VRAM, power, and swap
• quality pass, retry, abstain, or repair

Raw tokens per second is a fine diagnostic, but the number that matters is valid completed records per hour for automation, review time included for coding, and p50/p95 first-token latency across realistic conversation lengths for chat, worth instrumenting before you trust anything in measuring local tokens. A config that wins a short warm single-request run can still lose the day once model swaps and long contexts show up.

Check that the switch actually did anything

The mistake I see most is flipping huge pages on system-wide, then crediting it for a speedup that came from page cache warmth, a different quantization, or a shorter test prompt. Local runtimes keep working even when a setting isn't doing what you think: they offload layers, fall back to a generic kernel, page memory around, or miss caches quietly, and none of it throws an error. So verify it directly:

grep -i huge /proc/meminfo
cat /sys/kernel/mm/transparent_hugepage/enabled

If the pages aren't backing your mapping, if madvise never got called on it, or the kernel couldn't oblige the allocator, you're paying the configuration cost for a change that never took effect. Change one variable at a time unless you're deliberately comparing two whole systems, and keep the outputs, because quantization and sampling changes can shrink an answer while making it look faster. If it's shorter, check it's still complete before you call it a win.

Somebody still has to run this thing at 3 a.m.

Performance work has a bill outside the benchmark: startup time, upgrades, what you can observe in production, whether you can roll back, whether you can rebuild the box from nothing after a disk dies. A five-percent gain resting on an undocumented kernel patch or someone manually re-warming a cache every morning is a bad trade for anything shared. I wouldn't bother with static hugetlbfs reservations on a workstation that reboots daily; that memory doesn't come back for anything else, and in a month nobody remembers why it's carved out. Leave headroom after hitting the target: spare memory absorbs prompt-length variance, spare queue capacity protects interactive users from a batch job, thermal margin keeps the win from evaporating under sustained load.

Put a shelf life on the result

Keep a change only when repeatable evidence from the actual workload says to, and write the conclusion down with the workload, the date, and the reason, right beside the condition that should trigger a retest: a new model family, a driver update, longer contexts, a different traffic mix. Skip that boundary and last quarter's number quietly turns into house folklore nobody re-checks. This is the real advantage of running local: every layer is inspectable, so a speed claim doesn't ride on faith the way a hosted vendor's number does. So before you touch the sysctl next time, check whether your runtime ever calls madvise with MADV_HUGEPAGE on the model mapping at all. If it doesn't, transparent huge pages are decoration, and khugepaged is off doing compaction work in the background that nobody asked it to do.