Stop sequences are a latency and safety tool

A nightly extractor working through a folder of invoices, told to emit one JSON object per document and then stop talking. The same box also runs a coding loop most of the afternoon and a chat window someone opens every half hour or so. Three workloads sharing one machine, and only the extractor actually cares whether the model closes the brace and shuts up.

Freeze the run before you tune it

Before you touch a stop sequence, capture a baseline with one named model, one fixed prompt set, and the exact server command you plan to run in production, then write it down before you change anything else.

Write down the model artifact and the prompt template alongside it, because those two details are the ones people forget, and forgetting either is enough to make the comparison worthless three weeks later when someone asks why the numbers moved.

I keep the record boring on purpose. A boring record is one I can still trust after I've forgotten the context.

Then pick stop sequences that respect the tokenizer's actual boundaries, test them against real output for accidental collisions, and keep a hard output-token cap in place regardless of how well the stops behave. The cap is the backstop for whatever the stop string doesn't cover.

The stop string that hides inside a good answer

The mistake I see most often is a stop string that also happens to occur naturally inside a valid answer. It survives testing because the system keeps producing plausible-looking text, just less of it, and nobody notices until an invoice total goes missing.

Local inference has other soft failures too: a model partly offloading onto the CPU, a cache quietly missing, swap creeping up over an hour, a queue holding requests from clients that already disconnected, or a fallback path that quietly sends a request somewhere it was never supposed to go. None of them throw an error.

You find these by watching runtime logs and operating-system counters while the test is actually running, not by reading a summary afterward. Change one variable at a time unless you're deliberately comparing two complete configurations against each other, and run it enough times to tell a durable improvement from a lucky sample.

Read the outputs. Speed is not proof that two configurations produced the same thing, and if a change makes an important task worse, that cost belongs right next to the gain, not buried in a footnote.

What the clock actually has to tell you

Time to first token catches loading and prompt processing. Steady token rate describes decoding once it gets going. Completion time is what the user actually experiences, and it's the one of the three that people remember a week later.

Add peak memory, queue delay, and wall power to the record when they change the decision, and skip them when they don't. I wouldn't bother logging wall power for a single box under someone's desk; that's a rack-scale number pretending to be relevant at laptop scale.

For automated work, count completed valid jobs per hour. For interactive work, count the waits long enough that a person notices and gets annoyed, because that's the number that actually drives complaints, not the one on the dashboard.

Operational simplicity earns its own column too. An optimization that needs a manual fix after every driver update or every model swap is not free just because the benchmark says it's faster: it's a maintenance bill with the due date left blank.

I'd rather run something boring that a service file, a container definition, or a short script can reproduce exactly, on a bad day, without anyone remembering the details. Keep the raw run artifacts out of the report itself, but keep the hashes and the commands, so the next person running the same test can be sure it's actually the same test.

Stop tuning when the margin is wide enough

The rule that holds up is combining structural stops with a validation pass, not trusting either one alone: the stop sequence bounds the generation, and the validator catches the case where the model stopped in the wrong place anyway.

If you're not writing that check by hand, something like LangChain's structured output tooling or Ollama's built-in structured output mode will do a reasonable version of it for you.

This is a less exciting rule than picking the biggest model or the newest card, but it produces a stack whose limits you can actually see. A visible limit gets routed around, scheduled for later, or priced into the plan. An invisible one turns into an unexplained wait and an emergency upgrade nobody budgeted for.

Stop adjusting once the workload clears its quality and latency target with real headroom left over, because that headroom isn't wasted capacity: it's what absorbs a longer document, one more concurrent user, a hot afternoon, or whatever the next runtime release changes without asking you first.

I know what that costs. Leaving real headroom on the table means the box does less work per dollar than it technically could, and a benchmark chasing the last measurable gain would probably beat mine on paper.

I'll take that trade anyway. What I want at two in the morning is a system whose failure modes I already know, not one that scores better on a chart and turns into a mystery the moment it breaks.