A model-server health check should prove readiness

Every team running a local model behind a load balancer has watched some version of this happen: the health check is green, traffic keeps landing on a node, and the server on the other end never finished loading its weights, or its GPU context locked up an hour earlier and nobody told the socket. The port still answers politely. Requests pile into a queue. A person eventually notices generation stopped happening.

That gap exists because liveness and readiness are not the same question. A process that accepts a TCP connection is alive. Whether it can turn a prompt into tokens is separate, and a check that only measures the first tells you nothing about the second. Local setups make this worse than a hosted API: there is no vendor status page watching for you, and listening versus working can hinge on what the GPU driver did to a context handle overnight.

Split the socket check from the generation check

The fix is not clever, it just means refusing to let one cheap probe stand for two facts. Keep a liveness check for whether the process is running, and add a second, bounded probe for whether it can actually generate: a short, fixed prompt through the real path, tokens back inside a time budget. If you run something like vLLM or Ollama, this is the layer worth instrumenting first; see vllm-at-home. Expose model residency too: a server can be healthy on every metric and still be serving from weights partially offloaded to CPU because VRAM ran short.

Before anything else, capture a baseline: one named model, one fixed prompt set, the exact launch command and flags, artifact hash and prompt template kept with it. Follow the request through its full path rather than one number: time to first token catches loading and prompt processing, steady rate describes decoding, total completion time is what a person or a downstream job gets. Add peak memory, queue delay, and wall power when they'd change a real decision. Batch work wants valid jobs completed per hour; interactive work wants the slow waits a person will remember.

I keep this as a plain record, one line per field:

artifact + runtime + launch flags
fixed prompt set + model residency
first-token time / steady rate / total time
peak memory + queue delay + wall power (when it matters)
valid completions vs abstentions
decision + owner + retest date

That last line is the one people skip. A number without a decision is trivia; a setting without an owner becomes folklore.

The false greens that actually bite

The failure I'd flag first: restarting a server that's healthy but cold, because a heavy probe prompt timed out on a box that hadn't warmed up. Local inference produces plenty of soft failures that never raise an error, they just quietly degrade:

• a model partially offloads to CPU and nobody notices until latency doubles
• a cache misses and every request pays a cold-start cost again
• swap creeps up until the box slows for unrelated reasons
• a queue keeps holding requests whose client already disconnected
• a fallback path silently changes where a prompt gets processed

None of that trips an error handler; all of it gets blamed on the model instead of the layer that's broken.

When you test a change, watch runtime logs and OS counters while it runs, not just the number you're chasing. Change one variable at a time unless you're deliberately comparing whole configurations, and repeat the run: a fast result is often luck rather than a real gain. Read some of the output; speed alone says nothing about correctness. If a change hurts an important task, that cost belongs beside the gain, not buried in a footnote.

Operational simplicity earns a column too. A setting that only survives manual repair after every driver update was never free, whatever the benchmark said. A boring configuration in a service file beats a clever one you can't rebuild from memory. This is where the probe earns its keep beyond alerting: cheap signals let something like a llm-gateway-architecture route around a degraded node. A visible limit gets scheduled or priced into a promise; an invisible one turns into a stall and an emergency order.

Stop tuning once the workload clears its quality and latency bar with room to spare. I'd rather run a probe suite that's a little annoying than a stack whose only readiness signal is a socket with no idea what it's lying about. The tradeoff is one I take on purpose: a generation probe spends real inference cycles and adds latency to every health-check loop, cycles that aren't going toward a user's request. That's the price for a green light that means the model can actually produce a token, not just that the process hasn't crashed yet.