Keep the model library on NVMe, not your home directory

The second drive in my box is a 2 TB NVMe stick, and it holds exactly one thing: model weights, shared by llama.cpp, Ollama, and whatever test harness I'm running that week. My home directory gets none of it. Every GGUF, every checkpoint, every quant I'm still deciding about lives on that volume and nowhere else, on purpose, from the day I built the machine.

Model files don't change once they land. They're big, often tens of gigabytes for a single quant, and every runtime you point at one wants its own private copy by default. Ollama keeps a blob store. llama.cpp reads whatever path you hand it. A benchmark script you threw together downloads its own copy because that was the path of least resistance at the time. None of these choices is wrong by itself. Stacked together, they're how a 2 TB drive quietly fills up with three versions of the same file.

Three tools, one file, three copies

The failure shows up as disk usage, not as an error message. You go looking for space and find the same 20 GB quant sitting under three different paths, because three tools each decided independently to manage their own cache. Nothing crashed. Nothing warned you. That's the annoying part: local inference is full of setups that technically run, and "it loaded" tells you nothing about whether the layout is sane.

"The answer looked fine" isn't an evaluation either, and it's a separate problem from storage, but the two habits come from the same place: trusting whatever the tool reports instead of checking underneath it. Check the runtime's own logs and the operating system's disk and memory counters before you believe a flag did what it claimed. A quantization flag that silently falls back to a slower kernel, or a cache path that silently duplicates a file, will not announce itself. You have to go look.

The fix is not a smarter cache eviction policy. It's a layout that never lets the duplication happen in the first place.

Centralize the file, not the tool's opinion of the file

My rule: originals live in one place, and every runtime gets a pointer, not a copy. In practice that means:

• One directory tree of original model files, downloaded once, checksummed once
• Symlinks (or hardlinks where the filesystem supports them) from each runtime's expected cache location back to the original
• A checksum kept alongside each file so you can tell a corrupted or partial download from a real one without re-downloading to compare
• Runtime-generated caches (converted formats, KV-cache scratch space, logs) kept physically separate from the originals, so wiping a runtime's cache never touches the source file

That's it. It's not clever. The originals are the one thing on that drive you can't regenerate without a slow download, so they get treated like data, and everything a runtime derives from them gets treated like disposable output. If you're picking quant sizes to fit a specific card, the layout question and the quant-picking question are really two sides of the same decision: how much of the drive, and how much of the GPU, a given model family is allowed to claim.

Verifying checksums matters more than it sounds like it should, because a model file that's been silently truncated mid-download will often still load. It just performs worse, or fails in a way that looks like a prompting problem well after you've forgotten you ever re-downloaded anything. I don't bother with a fancy dedup filesystem for this. Content-addressed storage is a real answer for teams with hundreds of models rotating constantly; for one workstation with a shelf of quants you actually use, checksums and symlinks solve the problem you actually have. Building anything more elaborate than that is effort spent on the wrong bottleneck.

What a cold start actually costs you

None of this matters if you can't tell whether a layout change helped. Define the workload before you touch anything: interactive chat, code completion, batch extraction, whatever you actually run, and decide what a good result looks like for that workload specifically. Then use the same inputs every time and write the launch command down next to the result, because later you will not remember which flags produced which number.

One warm run tells you almost nothing. Include a cold start, because that's the one your actual users hit first thing in the morning, and repeat enough runs to expose queueing or thermal drift rather than trusting a single lucky sample. The numbers worth keeping are the ones that could actually change your decision: time to first token, prompt-processing speed, generation speed, peak memory, and task success on whatever you're using as ground truth. Medians tell you about the typical run; a slow percentile tells you about the run that makes someone close the laptop in frustration. Keep both.

Change one variable at a time and have a reason for expecting the result you get. A move from a spinning disk to NVMe, or a change to how caches are shared, should have a mechanism you can state in one sentence before you run the test, not just a number you're hoping goes the right direction. That discipline is the same whether you're reasoning about storage layout or about GPU offload math: you want to know why a number moved, not just that it did.

The five-percent win that won't survive an upgrade

Here's the part that's easy to skip: a small win you found by tuning one setup decays. A driver update, a runtime point release, a new model you swap in, and your careful five-percent improvement is gone and you won't notice until performance quietly degrades and nobody remembers why. You don't need an observability platform for a single workstation. You need a small script, a handful of prompts that represent your actual workload, and a plain-text file of results you can diff after every upgrade.

Spare capacity on that drive isn't waste, it's the thing that absorbs the longer prompt, the background process you forgot was running, and the model you'll want to try next month without a panic about free space. A library that's packed to the edge has no room to be reproducible.

Treat the model library as a cache you could rebuild from checksums, never as primary data you'd mourn losing.