Understand memory-mapped model loading

That three-second boot you're proud of only mapped the file and handed the runtime a pointer. Every real read got deferred to whatever moment the inference loop first touches that memory. mmap moves where the I/O happens, not whether it happens. Run a llama.cpp server off local NVMe, or off the network storage you settled for when picking hardware for local inference, and the deferred reads land on the first request after every restart or swap, on GGUF weights nobody pre-warmed.

cold first, then warm, then compare

Decide what number you're moving before touching a launch flag: first-token latency, jobs per hour, resident model capacity, energy per task, fewer corrections downstream. Pin the artifact, tokenizer, prompt template, runtime build, launch command, and sampling settings, then test a small set of real-workload inputs, awkward cases included. Run one pass cold, storage unloaded, if users hit that path, then warm, repeatedly, long enough for queueing, memory pressure, and thermal throttling to show up. Time the phases, not just the total: queue wait, load, prefill, first token, decode rate, peak memory, VRAM, power, swap, and the quality check. Raw tokens per second is diagnostic, not the product: automation wants valid records per hour, coding owes you correction time on top of generation, chat lives on p50 and p95 first-token latency, not a warm best case.

resilience hides the config you shipped

The recurring mistake is benchmarking once the OS has cached the model, then shipping that number as steady state. Local runtimes cope too well: they offload layers, page memory, miss caches, fall back to generic kernels without complaint, so a degraded config sits there quietly hitting every target you set. Read the startup logs. Check device placement. Watch the OS counters and confirm the optimization is active for the tensor shapes and context you're running. Change one variable at a time unless comparing whole systems, and save outputs, not just metrics: quantization, context compression, and sampling can buy speed by quietly changing the answer, so the quality gate runs on every candidate. Weigh operational cost too: startup, upgrades, rollback, rebuilding after a disk dies. A gain needing a hand patch or daily cache-warming is a bad trade for shared services. Boring configs age well.

Report the cache state you tested, not the cache state you hoped for.

That's the decision rule: pick storage for the cold behavior users encounter, write the workload, date, and reason into the result file, and flag what forces a retest, a new model family, a longer context. The same discipline behind squeezing the local tier also means leaving room once you hit the target: spare memory for prompt variance, spare queue so interactive users don't wait behind a batch job, thermal margin for the hot afternoon.

I still don't have a clean answer for the shared box: two model servers on one host, leaning on the same page cache, the kernel deciding under pressure whose pages get evicted first. You can set priorities. You can cgroup toward fairness. The eviction happens in a layer neither owns, and I haven't made that boring yet.