Containers rarely cause the local inference slowdown

So no, it isn't the container doing that to you. Nine times out of ten when a Dockerized inference server runs slower than the same model launched natively, the real cause is dull: a driver mismatch, a storage mount turning weight reads into network round trips, a forgotten CPU limit, or a launch flag that quietly changed between runs. None of that is the container's fault. It just gets blamed because it's the newest thing in the stack.

The usual suspects aren't the container

GPU work does not get slow just because a process lives inside a namespace and a cgroup. That belief keeps people from finding the actual problem. Local inference talk tends to start with a model name or a throughput figure, the wrong altitude to look from. The better question is what's waiting on the other side of the API: a private chat opened twice an hour, a coding loop firing constantly, a nightly batch job. Those three can share a box and still want opposite trade-offs, and no container-versus-native number tells you which one you're optimizing for.

Holding everything still but one thing

Capture a baseline first: one named model, one fixed prompt set, the exact server command, written down with the model artifact and prompt template. Skip that and next week's comparison means nothing. Then swap the one variable you care about, container versus native, holding runtime build, flags, storage, and warm state identical on both sides. Testing Ollama under Docker, running Ollama in Docker covers the mount details that trip people up; mount type alone can swing numbers more than the container boundary ever will. Measure time to first token, steady decode rate, and completion time, each a different phase of the request, plus peak memory, queue delay, and wall power when they feed the decision. Batch work is judged on valid jobs per hour; interactive work on the pauses a person notices.

Six lines I keep next to the terminal

I write the same fields down every run, because I don't trust myself to remember what changed three benchmarks ago:

artifact + runtime + launch flags
workload + fixed input set
cold start / warm start / p50 / p95
peak memory + wall energy
quality failures and abstentions
decision + owner + retest date

Leave off the decision and the page is trivia. Leave off the owner and the setting turns into folklore repeated in standup without anyone knowing why it's true.

Failures that never throw an error (docker)

The tempting mistake is running two different binaries against two different cold caches, then blaming Docker for the gap. It survives because the output still reads fine. Local inference has soft failures that never surface as an error: a model partially offloads to CPU without saying so, a KV cache quietly stops hitting, swap creeps upward, a disconnected request sits in the queue anyway, or a fallback path changes where the data goes. None of that throws an error. If offload is your suspect, the arithmetic in GPU offload math for Ollama is worth running before trusting a vibe. Watch runtime logs and OS counters while the test runs, change one variable at a time, and repeat enough runs to tell a real improvement from a lucky sample. Read the output; speed alone isn't proof two setups produced the same answer, and a change that makes something worse belongs in the report beside whatever it gained.

Spending the headroom on purpose

Operational simplicity is a column too. A fragile tweak needing hand-repair after every driver update was never free, so rebuild from a service file or a short script rather than babysit something delicate. Keep raw artifacts out of the report, but keep hashes and exact commands so a rerun months later still means the same thing. Containerize for reproducibility, then investigate the measured phase differences instead of arguing about the boundary itself. Stop tuning once the workload clears its latency and quality target with real headroom left, since that margin absorbs a longer document, one more user, or the next runtime release. Spend model capacity only where it actually changes the outcome.

Next time a Dockerized model looks slow, don't open the Dockerfile first. Check the driver version inside the container against the host, check whether the storage mount is a bind mount or something copying on every read, and diff the cgroup CPU limit against the native process. Those three, before anything else.