Put authentication in front of the local API

Somebody on the same Wi-Fi as your workstation finds an open port, points a client at it, and starts running their own prompts through your GPU while you're asleep. This isn't a rare story: it's the standard way people find out that binding a model server to their LAN, or port-forwarding it for convenience, was never actually private.

I run an Ollama or llama.cpp server shared across a handful of trusted devices at home, which is about as ordinary a setup as this gets. No enterprise firewall, no dedicated network engineer, just a machine with a GPU and a few laptops that talk to it. The mistake that setup invites is assuming the model being private has anything to do with the network being private. It doesn't. An exposed inference endpoint leaks whatever gets typed into it, to anyone who finds the port. It burns compute you're paying for in electricity and wear. And on a bad day it becomes a way onto your network that has nothing to do with language models at all.

The model being private doesn't make the socket private

Before you lock anything down, write down who is actually supposed to talk to this thing. Interactive chat from your own laptop is a different threat model than a shared endpoint that a few housemates or coworkers hit for code completion, and both are different again from an overnight batch job nobody should be poking at manually. The common trap is forwarding port 11434 because typing a LAN IP address every time is annoying, and only later realizing every device on that network, sometimes beyond it, can submit unbounded work with zero authentication. Nobody sits down and chooses this on purpose. It happens one convenience shortcut at a time, and the shortcut works right up until it doesn't.

Four layers, not one flag

Four separate settings stacked on top of each other beat one clever setting, because any single failure shouldn't leave the whole thing open. Bind the server to a narrow address instead of 0.0.0.0. Put a reverse proxy in front of it that terminates TLS and actually checks credentials, instead of trusting the raw model server to do access control it was never built for. Segment the host itself, so a compromised inference box isn't sitting on the same flat network as everything else you own, the same isolation instinct behind any multi-tenant setup. And log who connects and when, without logging prompt bodies, so you have a record to check afterward without turning chat history into its own liability.

Answering requests is not the same as being locked down

A server that responds to curl isn't evidence that any of this works. "It loaded" was never a performance result, and it isn't a security result either. The proxy can be misconfigured, the auth check can be quietly bypassed by an internal route you forgot existed, and the only way to know is to check the runtime logs and the operating-system metrics instead of trusting whatever flag you think you set, the same instinct behind red-teaming your own endpoints rather than trusting a demo. There's a maintenance cost nobody budgets for, too. A config that only works because you happened to get it right once tends to quietly break the next time the model, the driver, or the runtime gets upgraded, and you won't notice until the next open-port surprise shows up in the logs. Keep it retestable: a small script, a handful of representative requests, and a plain-text file recording what happened, so a five-minute check after every upgrade replaces a nasty discovery months later.

Treat it like the internal service it is

My rule here is the one I use for any internal service that touches data and compute worth protecting: change the smallest plausible thing first, keep headroom instead of running the box at the edge of its limits, and stop once the setup meets its actual latency and access requirements rather than chasing a marginally tighter config nobody asked for. Spare capacity isn't waste. It's what absorbs the next long prompt, the background job you forgot was running, and whatever model you'll want to try next month without re-architecting the whole thing.

So the next thing I'd actually go check is not the model, it's the edge. Pull up the proxy logs and confirm the last several connections came from devices you recognize, then try hitting the raw port directly from a machine that was never issued credentials and make sure it gets nothing back.