Make review severity operational

Turn on an LLM reviewer for pull requests and the first week almost always goes the same way. The tool produces a wall of comments, most phrased like the reviewer just caught something serious, and a rename suggestion sits at the same severity as a null check that's missing. Developers stop reading past the first few lines. Somewhere in that pile is a real defect, and it gets the same shrug as everything else, because nothing says which is which.

Most of the time the model isn't wrong about the code. What's missing is a fixed meaning for severity, so it fills that gap with confidence instead of evidence. A finding worth reading names the impact, points at the exact line, and sketches how the bug bites someone. One that just sounds worried is not that, however polished the prose.

confident wording is not evidence

This shows up hardest in local setups, where a model reviewing application changes for a small team can sit on the same box as a private chat used a few times an hour and a nightly extraction job. Those three workloads want different trade-offs, and none of it says how good the reviewer's judgment is. The recurring failure is ranking a stylistic preference as high severity because the sentence sounds confident, and it survives because the tool still produces plausible text.

Underneath, local inference has failure modes that never trip an alarm. A few worth watching:

• the model silently offloads part of itself to CPU and slows down without complaint
• a cache quietly stops hitting and every request pays the cold cost
• swap grows until the machine is thrashing under something else entirely
• a queue keeps holding requests whose caller already disconnected
• a fallback path quietly changes where the data goes, which matters for privacy

None of these throw an error. The system just looks fine on a dashboard and behaves worse every day, which is what severity is supposed to catch.

give the reviewer a baseline before you trust its opinion

Before changing a prompt, a model, or a server flag, write down what you're starting from: one named model, one fixed prompt set, the exact launch command. Put the model artifact and the prompt template in the same record, since both drift quietly and either one can invalidate a comparison you'll rely on. Then decide what counts as blocking. A finding that can't point at a specific line and a specific consequence doesn't get to be blocking, no matter how it reads.

From there, track which findings developers accept versus argue with or ignore. That acceptance rate is the real signal, and most AI code review setups never capture it, because it lives in PR comments and closed threads instead of anywhere the tool can see. Add one field asking whether a finding was acted on, and start counting.

measure the whole request, not just the model

Once the baseline exists, let measurements follow the request through the entire system instead of stopping at the model. Time to first token covers loading and prompt processing, steady token rate describes decoding, and completion time is what the person waiting experiences, sometimes diverging from the first two. Add peak memory, queue delay, and wall power when they change the decision, and for background work count completed valid jobs per hour instead of any per-token figure, since batch work doesn't care how fast one token arrived.

I keep the notebook boring on purpose:

model artifact + runtime + launch flags
workload and the fixed input set used to test it
cold start, warm start, p50, p95
peak memory and wall energy
quality failures and abstentions
decision, owner, retest date

Change one variable at a time unless you're deliberately comparing two complete configurations. Run it more than once, since a single good result is a sample, not a trend. Read the outputs instead of trusting speed as a stand-in for correctness, and if a change makes an important task worse, write that cost down next to whatever it gained.

grade the grader against real bugs

The decision rule that holds up is calibrating the reviewer against defects your team resolved, not against a leaderboard or a bigger model. Duller than picking the newest checkpoint, but it produces a stack whose limits you can see. Visible limits get routed around, scheduled for, or priced into the process. Invisible ones turn into unexplained waiting and an emergency upgrade nobody budgeted for, and closing that gap is most of what evals and LLM-as-judge work is for.

Operational simplicity earns its own column too. A clever setup that needs manual repair after every driver or model update isn't free just because one test ran fast. Favor something boring enough to reproduce from a service file or a short script, and keep raw run artifacts out of the report, but hold onto the hashes and commands so a rerun matches.

Stop tuning once the workload hits its quality and latency target with headroom to spare. That margin isn't waste. It's what absorbs a longer diff, a second reviewer joining the team, or the next point release of whatever you're running.

What I'd check next: pull a handful of real bugs your team fixed, run them back through the reviewer with severity forced into your blocking categories, and count how many it would have caught versus buried under style notes. That number says more about the tool than any latency chart will.