Version prompts like small programs

A local model serving a support inbox and a nightly extraction batch has exactly one GPU to share between the two, whether you planned for that or not.

That's the constraint everything below sits on. Instructions, examples, model checkpoint, and decoding settings are the program, not decoration around the call. Change one and you've shipped a new build, even if the diff looks like a typo fix.

I test this against a small team running extraction and support workflows on their own hardware, nothing exotic, no claim it generalizes. One rule I don't bend: name the job before touching a setting. Chat, completion, extraction, overnight batch: each defines good differently, and it needs to be on paper first.

Treat the prompt file like a build artifact

The method that's held up for me is boring on purpose. Prompts live in files, in version control, next to fixtures that pin the inputs, noting decoding settings. Every run reuses the same inputs and saves its launch command. One warm sample tells you nothing: run a cold start if a human hits one, then repeat long enough for heat or queueing to show. Quality goes in the record beside the timing, always.

What belongs in that record:

• time to first token
• prompt and generation speed, kept separate
• peak memory, wall power when relevant
• task success against the fixture
• the median plus a slow percentile, because averages erase annoying pauses

run:      candidate prompt vs current, same fixtures
cold/warm: first request vs steady-state
quality:  pass / fail / abstain
verdict:  keep, revert, retest

Nobody notices a dashboard edit until it's in production

The failure I see most: someone edits a live prompt in a dashboard, no before-and-after test attached. Finishing without an error isn't a performance result, and glancing at one output isn't an evaluation, it's a vibe check. Trust runtime logs and OS metrics over whatever flag you think you passed: local setups that start don't always do what got asked for. Change one variable per test, say why you expect the number to move, or the exercise turns into anecdotes with timestamps.

There's a second cost too: a fragile five-percent win doesn't survive the next driver update, runtime bump, or model swap. I don't build a monitoring stack for this: something like agent observability tooling is overkill for two Python scripts and a cron job. A short script, a few representative prompts, and a plain-text result file you diff after upgrades is the whole budget.

The evidence bar, and where mine still cracks

My rule: a prompt change ships on the same evidence a code change needs. Test the smallest change that could matter, keep headroom under your latency and memory ceiling, and stop once you've hit what the workload needs. That spare capacity absorbs the longer prompt someone writes next quarter, or the background job fighting for the same GPU.

What I haven't solved: how you version the model itself when the weights behind an unchanged tag shift under you, the Ollama model churn problem in miniature. Fixtures catch a prompt regression fine. They don't catch a provider quietly retraining the thing your prompt was tuned against, and I don't have a clean answer for how often to rerun the whole suite.