Should model files live on a NAS?

Keep the model files on the NAS as the one true copy. Keep fast local disk on every inference box as the thing that actually serves requests. That's the rule. I'm putting it first because everything below exists only to earn it, not to build up to it.

Immutable weights are easy to centralize: one NAS share holding every GGUF and safetensors file, one place to update, one place to back up. Inference doesn't care. It wants predictable local access, and even a fast network link adds latency variance a local NVMe drive doesn't have. You keep both, but only once the NAS stops being somewhere models run from and becomes somewhere they come from.

what a cold load actually costs you

Picture a handful of inference nodes pulling from one shared, curated set of models. Before touching a launch flag or buying switches, write down the number you're trying to move: first-token latency, accepted jobs per hour, models kept resident at once, energy per completed task, fewer corrections from whoever reviews the output. "Make it faster" tells you neither when you're done nor what you're allowed to break to get there.

Then build a baseline you can rerun without arguing about it later: pin the model artifact, tokenizer, prompt template, runtime build, launch command, and sampling settings the way you'd pin any dependency (see local sampling settings for the knobs people forget), and use real inputs from the workload, awkward cases included. Run once cold, storage untouched and model unloaded, since that's the path a real deploy takes, then run warm, long enough to expose cache effects and memory pressure. Record every phase separately: queue wait, model load, prompt processing, time to first token, decode rate, peak memory and power, and whether the output passed quality or needed a retry. One wall-clock number hides which phase the network is punishing.

Raw tokens-per-second is diagnostic, not the product. For a batch job, count valid completed records per hour. For coding, count review and correction time. For chat, watch p50 and p95 first-token latency across realistic conversation lengths. A config that wins a short warm demo can lose once model swaps and long contexts show up.

the failure that never throws an error (nas)

Here's the one that gets people: memory-mapping a large model over a flaky Wi-Fi link or an oversubscribed NAS. Local runtimes are almost too well-behaved about this. They offload layers, page memory, miss caches, queue requests, or fall back to a generic kernel rather than fail outright. That resilience is the trap: a degraded setup still returns an answer, just a slower or worse one, and nothing tells you. Read the startup logs, check device placement, watch the OS-level counters, and confirm the optimization you asked for is running against the tensor shapes and context length you use in production.

Change one variable at a time, and keep the outputs, not just the metrics, since quantization, context compression, and sampling changes can shift an answer while making it look faster. Weigh operational cost too: startup, upgrades, rollback, rebuilding the box after a disk dies. A five-percent win from a hand-applied patch or someone warming the cache every morning is a bad trade for anything shared. I wouldn't bother with a tiered-caching product here; a dumb copy-on-first-use script in front of local disk has covered every case I've hit.

So: NAS as origin, local disk as cache, the decision written down with workload, date, and reason, plus the condition that should trigger a retest: a new model family (see local model updates), a driver change, a longer context window, another concurrent user. Leave headroom once you hit the target. Free memory absorbs prompt variance, spare queue capacity protects interactive users from a batch job, and thermal margin protects anything running unattended for hours.

What I still don't have a clean answer for is the thundering-herd case: every node in the pool cold-loading from the same NAS at once, right after a deploy or a power event, hammering a link sized only for steady-state traffic.