Tokenizer mismatch breaks budgets quietly

Count tokens with the tokenizer the request will run through, not a stand-in, and treat any mismatch as reason to distrust every number downstream of it. That's the whole rule. I trust it more than a model name or a peak throughput figure, because neither tells you what happens once a prompt nears the edge of what the model can hold.

Token boundaries are not a footnote. Two model families can look at an identical string and split it into a different count of pieces, and a gateway that counts once with a convenient tokenizer, then assumes it holds for whatever backend answers, is quietly building its budgets on sand.

One box, three jobs, no shared budget

Picture a gateway routing several local and hosted models: a private chat used a couple of times an hour, a coding loop firing all day, and a nightly extraction job running unattended. Same machine, three different tolerances. Before touching a config file, capture a baseline: one named model, one fixed prompt set, the exact server command, plus the model artifact and prompt template, easy to lose and easy to invalidate.

Count with that model's own tokenizer and keep margin for the chat wrapper and the output. Measure end to end: time to first token for loading and prompt work, steady decode rate for generation, completion time for what the user gets, plus peak memory, queue delay, and wall power when they matter. For the nightly job, count completed valid jobs per hour; for the chat, count the waits a person notices.

The record I keep is deliberately dull: artifact, runtime, launch flags, workload and input set, cold and warm timings at p50/p95, peak memory and wall energy, failed or abstained outputs, plus a decision, an owner, a retest date.

A benchmark with no decision attached is trivia; a setting with no owner attached is folklore.

Guessing at the ceiling

The mistake I see most is reaching for one convenient, model-agnostic token-count approximation and running it right against the number printed on the model card; it survives because the output still looks plausible. Local inference is full of failures that don't announce themselves: a model offloads partly onto the CPU, a cache misses without complaint, swap creeps up, a queue holds requests whose client already left, or a fallback moves the request across a privacy boundary you meant to keep closed. None of that throws an error. It just gets slower, or wronger, or both.

Catching it means watching runtime logs and OS counters while the test runs, changing one variable at a time unless comparing whole configurations, and repeating enough to rule out a lucky sample. Read the outputs; speed is not proof that quality held. If a change makes an important task worse, report that cost next to the gain.

What I'd actually keep running

Operational simplicity belongs in the same column as latency and cost, and it usually gets skipped. A clever fix needing manual repair after every driver or model update isn't free just because it was free to build. I wouldn't bother forcing every backend through one shared vocabulary; the upkeep costs more than the tidiness is worth. Prefer something boring enough to reproduce from a service file, a container definition, or a short script, and keep the hashes and commands without the raw artifacts, so a rerun means the same thing.

Making token accounting specific to the model you're running, not a shared approximation, is a smaller decision than buying a bigger card. What it buys is a stack whose limits are visible instead of invisible: a visible one gets routed around, scheduled, or priced into the job; an invisible one turns into an unexplained stall and an emergency upgrade.

Stop tuning once the workload hits its quality and latency target with margin left over. That spare capacity isn't sitting idle, it's headroom for a longer document, an extra user, a warm afternoon, or the next runtime release. Most of the skill is refusing to spend capacity where it doesn't change the outcome.

If I kept exactly one rule, it's the one I opened with: count with the tokenizer of the model you're actually running, and make the system fail loudly before it starts truncating evidence quietly.