Make generation cancellation actually stop compute

Treat cancellation as a resource-control path and test it under load. That's the rule. Full stop. Everything from here is just me earning it.

Close the browser tab and the request doesn't necessarily die with it. Unless the disconnect signal reaches the runtime, the server keeps decoding tokens nobody will read, memory stays pinned, and the next real request queues up behind a ghost. None of that throws an error. It just quietly eats the headroom you were counting on.

That headroom matters because the same box rarely runs one job. A private chat used a handful of times an hour, a coding loop firing off completions all day, and a nightly batch extraction can all live on the same local runtime, each tolerating someone else's abandoned tokens sitting in the queue differently.

Watching the GPU instead of the spinner

Before you touch a config file, capture a baseline: one named model, one fixed prompt set, the exact server command you launched it with. Write down the model artifact and the prompt template too, because both are easy to lose and large enough to invalidate the comparison.

Then propagate the disconnect signal through every layer between browser and runtime, and confirm GPU activity falls within a second or two of the disconnect. Otherwise you've built a cancel button, not a cancel.

Time to first token catches loading and prompt processing. Steady token rate describes decoding once it's underway. Completion time is what the person on the other end actually experiences. Add peak memory, queue delay, and wall power when they change the decision, and for batch work count completed valid jobs per hour.

I keep the same short list on every run: artifact, runtime, and launch flags; the workload and its fixed input set; cold-start and warm-start numbers at p50 and p95; peak memory and wall energy; whatever quality failures or abstentions turned up; a decision with an owner and a retest date. A benchmark without a decision is trivia. A setting with no owner turns into folklore.

A cancel button that only hides the spinner is a UI feature, not an engineering one.

The failures that never throw an error (cancellation)

The tempting shortcut is to remove the spinner client-side while the backend keeps grinding through the full token budget. It survives review because the output still looks plausible. Local inference has plenty of these soft failures: a model partially offloading to CPU, a cache quietly missing, swap creeping upward, a queue holding onto requests nobody's listening for, or a fallback path that silently changes where your data goes. None of them raise an exception.

So watch runtime logs and OS-level counters while the test runs, not just the number the benchmark prints at the end. Change one variable at a time unless you're deliberately comparing two complete configurations. Run it enough times to tell a durable improvement from a lucky sample, and read the output instead of treating speed as proof of correctness. If a change makes an important task worse, put that cost next to the speed gain.

This isn't specific to a bare server wired straight to a websocket. Stream through something like LangChain's streaming, or coordinate multi-step generation through LangGraph's streaming events, and the abort still has to propagate through that layer too, not just stop at your reverse proxy.

Operational simplicity earns a column too. An optimization needing manual surgery after every driver or model update carries a cost, paid in on-call time. Favor a setup you can reproduce from a service file, a container definition, or a short script, and keep the commands and hashes after discarding the raw artifacts, so a rerun months later still means the same thing.

That's what makes cancellation boring: a resource-control path, tested under the load pattern of an ordinary Tuesday, not a clean single-user demo. Visible limits get routed around, scheduled, or priced. Invisible ones turn into unexplained waiting and a GPU bought in a panic.

Cancel is a resource-control path. Test it under load or it isn't one.