Forcing local models to speak JSON

In late May I shipped a ticket triage pipeline for a logistics client: support emails go in, Qwen3 8B reads them on the 3090 through the same Ollama setup from my field notes, and a JSON verdict (category, severity, language, one-line summary) lands in Postgres. The model part worked on day one. The JSON part didn't. Seventeen percent of responses failed json.loads, and every failed parse became a ticket a human had to touch, which was the exact thing the pipeline existed to remove.

The failures were never exotic. A chatty preamble before the first brace. A trailing comma. Markdown fences hugging otherwise perfect JSON. One memorable answer was flawless JSON followed by the words 'I hope this helps'.

Two weeks of prompt begging

My first fix was everyone's first fix: prompt harder. RESPOND ONLY WITH VALID JSON, in caps. Three few-shot examples. A stern closing line about how no other text was permitted. Then a repair layer on top: strip fences, cut everything before the first brace, retry on failure. Two weeks of this moved me from 83% parseable to about 94%, where it plateaued and wobbled with every model swap and every ticket that arrived in Czech instead of English.

Prompt begging has a ceiling, and the ceiling is below 100.

Grammar beats begging

Ollama's format parameter takes either the literal string json or a full JSON schema, and it doesn't work by asking the model nicely. The schema gets compiled into a grammar that constrains decoding itself: at every step, any token that would break the declared structure is masked out and cannot be sampled. The preamble is illegal. The trailing comma is illegal. 'I hope this helps' is illegal, because no rule in the grammar produces it.

{
  "model": "qwen3:8b",
  "messages": [{"role": "user", "content": "ticket text here"}],
  "format": {"type": "object", "required": ["category", "severity"], "properties": {"category": {"type": "string"}, "severity": {"enum": ["low", "medium", "high", "critical"]}}}
}

I flipped that on June 3rd. Since then: 3,847 tickets, zero parse failures. I deleted the repair layer a week later, and deleting it was more satisfying than writing it ever was.

The enum that made it dumber

Then answer quality sagged, and it took me days to notice because everything parsed. Before the schema, the model wrote severity as free text and hedged when hedging was right: 'probably low, though the customer mentions a deadline'. The enum deleted the hedge. Forced to commit to one of four values on ambiguous tickets, the 8B committed. Full confidence, mediocre judgment. My Friday spot-check, 50 tickets against my own labels, usually agreed on 43 or 44. First full week under the schema: 38.

A schema is a fence, not a brain. The output always parses now, and it can still be confidently wrong.

Two changes clawed most of it back. I added unclear to the enum, which returned the model's escape hatch, and those tickets now route to a person, which is what should have happened anyway. I also moved a free-text evidence field above severity in the schema, because field order is generation order: the model writes its reasoning first, and the label conditions on it. The spot-check sits at 45 or 46 now, a notch above the pre-schema baseline.

What I'd tell past me

Constrained output relocates failure instead of removing it: out of syntax, where failures are loud and cheap, into semantics, where they're quiet and expensive. Still a good trade, because you spend review time on meaning instead of chasing commas. But keep validating downstream. A parseable lie is still a lie, and my rules for that layer are in the guardrails post. If you're on LangChain, with_structured_output rides this same machinery when it points at a local model, and I compared the two approaches in a separate piece.

Keep schemas shallow while you're at it. Every required field is a promise the model must keep even when it has nothing true to say, and deep nesting is how you get beautifully structured filler. One level, few required fields, an escape-hatch value anywhere you'd otherwise force a guess.

Beg less, constrain more, and then measure what the constraint quietly costs — on my pipeline it wasn't free, just worth it.