Put a retry budget on structured output

A single inference worker has exactly one queue, and a retry gets in line behind everything else waiting on that GPU, cold cache included, same as a brand-new request would. That's the same mechanism already bottlenecking every other request hitting that model. Retries pay for themselves only when the failure is stochastic and the next attempt can genuinely differ from the last. Resample a structurally wrong answer and you've spent the queue slot twice on the same mistake.

I ran this against a local extraction pipeline that returns validated JSON, nothing fancier than that. No lab conditions, no claim that one machine's numbers travel to your card or your batch size. The point is to turn "should this retry" into something you can check instead of something you feel. Name the job first: interactive chat, code completion, document extraction, overnight batch, each with its own tolerance for a slow retry and its own idea of "good enough."

Sort the wrong answers before you resend them

Not every malformed response deserves the same fix. A response that fails to parse is a different animal from one that parses but violates the schema, and different again from one that's valid and on-schema but wrong about the facts. Lump them together and you end up resending prompts for problems a text fix could have caught locally. Classify first, repair what you can without touching the model, and spend a retry only on what's left over, bounded to the specific thing that broke.

• parse failure (not valid JSON): repair locally first, closing brackets, stripping trailing commas, before burning a retry
• schema failure (valid JSON, wrong shape): correct the specific field that failed, not the whole prompt again
• semantic failure (valid, on-schema, wrong content): this is the one retry can plausibly fix, since a different sample might genuinely help

None of this needs custom tooling: schema validation is well-trodden ground, worth borrowing from existing work on validating structured output instead of writing your own repair regex at midnight. Run the sequence inside a graph and that retry step is just a bounded cycle, not a while loop with no exit.

What the retry log actually needs to hold

Use the same inputs for every run, fixed and versioned, and save the launch command next to the result, or you'll be reverse-engineering your own test in three weeks. One warm sample tells you almost nothing. Include a cold start if a human will ever hit one, and run long enough to expose heat or queueing under load. A version that answers faster but differently isn't a faster version of the same system, it's a different system wearing the old name.

workload:      one named, repeatable task
inputs:        fixed and versioned
retry policy:  classify -> repair -> bounded retry -> give up
latency:       cold / warm / tail
resources:     memory, power if it matters
outcome:       pass / fail / retried-N-times

Record only the numbers that could change your decision: time to first token, prompt-processing speed, generation speed, peak memory, wall power where relevant, and whether the task succeeded. Medians describe the normal case; a slow percentile catches the pause that makes someone close the tab. Note the setup friction too, next to the numbers, because that's what decides whether anyone keeps using this thing.

Three tries and a shrug isn't evidence

The trap I see most often: send the same prompt three times, get lucky once, file that under reliable. It isn't. Local inference is full of configurations that technically run without technically working. "It loaded" is not a performance result. "The answer looked fine" is not an evaluation, it's a vibe. Check the runtime logs and the OS metrics instead of trusting whatever flag you think you passed, because runtimes quietly fall back to a slower path and never say so. Change exactly one variable at a time and be able to say why you expect it to matter, or the exercise is just anecdotes with a timestamp.

The five-percent win that doesn't survive an upgrade

There's a maintenance cost here that people underprice. A fragile five-percent improvement evaporates the next time the model or the runtime underneath it changes, and something always changes. Keep a small script and a handful of prompts that represent your workload. Keep the results in a plain-text file you can diff after every upgrade, no observability platform required for one workstation. I wouldn't bother with anything heavier until the workload runs on more than one machine.

Where the budget runs out, on purpose

My rule: make failure visible, then cap how much inference you spend hiding it from yourself. Test the smallest plausible change first. Keep headroom in reserve rather than tuning it away, since that's what absorbs a longer prompt today and whatever the next model needs later. Stop the moment the workflow clears its latency and quality bar; chasing the number past that point is a hobby, not engineering.

Next time a batch job trips a validation error, don't just watch it retry and move on. Pull the runtime log for that run and check whether the second attempt got a different, bounded prompt, or whether it just got lucky on an identical resend.