Grammar-constrained decoding beats repeated JSON pleading
If you're still writing paragraphs of prompt instructions begging a local model for valid JSON, stop: llama.cpp's grammar support enforces the shape at the token level, no pleading required. That's the blunt version. The qualified version: enforcement only buys you syntax, and mixing that up with correctness is where this quietly stops paying off.
The question worth asking isn't which approach wins on paper. It's when the choice starts changing how the system behaves during an ordinary week of traffic. The rule underneath: a schema earns its keep only once the decoder enforces its syntax, not the prompt.
what's actually queued behind the endpoint
Most local-inference writeups open with a model name or a peak tokens-per-second figure, which tells you almost nothing about whether a grammar constraint is worth the trouble. What matters is the work waiting on the other side of the call. A private chat someone pokes at twice an hour, a coding loop firing requests every few seconds, and a nightly batch extraction job can share the same box and still want completely different things from it. Before touching a grammar file, pin a baseline: one named model artifact, one fixed prompt set, the exact server command, written down together. Lose the prompt template and next week's comparison means nothing.
The record I keep per run stays deliberately plain: artifact, runtime, launch flags, the workload and its fixed input set, cold-start and warm-start times next to p50 and p95, peak memory and wall energy, and whatever quality failures or outright abstentions showed up, closed out with a decision, an owner, and a retest date. A benchmark without a decision attached is trivia. A setting without an owner turns into folklore nobody can explain later.
schemas that fight the decoder instead of helping it
The mistake I keep seeing is an oversized schema that slows decoding down and still can't express the business rule that matters, and it survives review because the output still looks plausible on a screen. Local inference fails softly in ways that never throw an error: a model drops into partial CPU offload, a cache quietly stops hitting, swap creeps up, a queue keeps holding requests whose caller already hung up, or a fallback path moves the job somewhere your privacy assumptions didn't cover.
Watch the runtime logs and the OS counters while the test runs, not after. Change one variable per run unless you're deliberately comparing whole configurations. Run it more than once, because a single fast sample isn't evidence, it's luck with good timing, and read what the model actually produced instead of trusting a faster number as proof nothing broke. If a change makes one important case worse, that cost goes in the same paragraph as the win.
the boring configuration usually wins
Operational cost counts too. An optimization that needs manual repair every time a driver or model file gets bumped isn't free, whatever the benchmark claims. Favor whatever reproduces cleanly from a service file, a container definition, or a short script you'd trust at 2am. Keep the heavy artifacts out of the report itself, just the hashes and commands, so a rerun later means the same thing it meant today.
Constrain syntax in the runtime, leave meaning-checking to ordinary code that can reason about your domain. It's a less exciting call than swapping in a bigger model or a new accelerator, but it leaves a stack whose limits you can see: visible limits get scheduled around or priced in, invisible ones turn into an unexplained stall and a rushed hardware order. Stop tuning once the workload clears its quality and latency target with real headroom left, because that margin absorbs a longer document or a busy afternoon without a page at 2am.
None of this saves you from a grammar file so loose it still lets the model emit garbage that happens to parse.