Fix model-cache ownership before the container starts

Your compose file mounts /root/.ollama or ~/.cache/huggingface into the container as a bind mount, the base image's entrypoint runs as UID 1000 or as root depending on who picked that image, and the first big model pull writes several gigabytes onto disk before anyone checks who owns the result. Nobody notices for weeks. Then a second container mounts the same path, runs as a different user, and either can't read what's already there or can't write next to it, and you're debugging a permissions error at eleven at night with a large download sitting half-finished.

That's the problem in one sentence: a model cache needs a predictable writer, a predictable reader, and a lifecycle that outlives whatever container happens to be running against it this week. Containers are disposable on purpose. The weights sitting in that volume are not, and the moment you treat the cache as though it were as ephemeral as the process reading from it, you've bought yourself a permissions incident with a multi-hour download attached.

One box will often have more than one thing wanting at that cache: a chat container someone opens a couple of times an hour, a coding loop pulling completions all day, maybe a nightly batch job that spins its own container up and down and expects the weights to already be there. They can share the same NVMe-backed directory fine. What they can't share is an unstated assumption about which UID is allowed to touch it.

Pin the UID before the first byte lands on disk

Decide the numeric UID before anything runs and write it into the compose file or pod spec explicitly. Don't lean on whatever the base image defaults to, that default was picked for local-dev convenience and it won't match the UID your serving container runs as in production. Pre-create the cache directory on the host with that UID as owner before the first container ever starts. Let the container create it on first run instead, and you inherit whatever the entrypoint was running as at that moment, and unpacking a multi-gigabyte archive is exactly the kind of job someone wrote to just work rather than to run unprivileged.

Then split the job that downloads and unpacks a model from the process that serves it. The pull needs write access. The server, almost all the time, only needs to read. Mount the cache read-only into the serving container once the weights are in place and you remove a whole category of accident: a server touching its own cache mid-request, a crash that leaves a half-written blob behind, two replicas racing to finish the same pull.

Confirming the fix didn't cost you anything means watching the request end to end, not just checking that the service came back up. Time to first token is loading and disk, not the model. Steady token rate is decode speed once the weights are resident. Completion time is what the person on the other end actually feels. Add peak memory, queue delay, and wall power on top when any of those are close to a limit. Track completed valid jobs per hour if the container runs unattended, it catches quota problems and cache misses that per-token timing won't. Track the slow waits a person will remember and complain about if someone's watching a terminal, because that's a different number from your averages.

Write down what you changed and why before you touch the mount options, not after:

That last row is the one people skip, and it's the one that matters. A benchmark without a decision attached is trivia. A permission change without an owner is folklore some future engineer inherits and is afraid to touch.

The workaround nobody schedules time to remove

The failure mode I've actually watched happen: someone hits the ownership mismatch, runs the container as root to make it go away, confirms it works, and moves on. It survives because the system keeps producing plausible output. Nothing crashes. The model answers questions. Nobody circles back, because there's no error demanding attention, just a container running as root that everyone has quietly agreed not to think about.

Local inference has a lot of these soft failures, and root-by-default is only the loudest one. A model partly offloads to CPU because a permission error silently blocked a GPU cache directory and nothing logged it as fatal. A read cache quietly misses because the reader can't see files the writer just created, so every request pays the cold-start cost and looks like a hardware problem instead of a mount problem. Swap grows. A queue holds requests from a connection that already dropped. A fallback path kicks in and quietly changes which data leaves the machine. None of that necessarily throws an exception you'll see on a dashboard.

So when you change a mount or a UID, watch it happen. Tail the runtime logs and the OS-level counters while the container starts and takes its first real request. Change one thing at a time unless you're deliberately comparing two whole configurations against each other. Repeat it, a single clean run tells you almost nothing about whether you got lucky. Read what came back, not just how fast it came back, because a broken cache can still hand you a fast, wrong answer, and if a change makes something worse, write that cost down next to whatever it gained.

Operational simplicity belongs on this list too, even though it never shows up in a latency number. A fix that needs manual repair every time the base image or the driver updates isn't fixed, it's deferred. I'd rather run a boring, slightly slower setup fully described in a service file or container definition than a clever one only one person remembers how to restore. Keep the model files out of the config repo, but keep the hashes and exact commands that produced them, so a rerun months from now still means what you think it means.

Stated as one rule: preparing an artifact and serving it are different jobs, with different UIDs and different mount modes, and they should stay that way. Less satisfying than picking a bigger model or a faster card. What it buys you is a stack whose limits are visible, and a visible limit can be scheduled around; an invisible one waits for a bad week and turns into an incident. Once the workload hits its target with room to spare, stop, that room is what absorbs the next runtime update or the extra user, not wasted capacity. What I still don't have a clean answer for is two container engines with different default UID mappings mounting the same Ollama cache from separate hosts. I've mostly just avoided setting that up rather than solved it.