Evals and LLM-as-judge: how to know your AI feature actually works
LLM output is nondeterministic and often subjective, which makes "it seems better now" the most dangerous sentence in AI engineering. Without evals you're flying blind: you can't tell an improvement from a regression, and you find out which one you shipped when users complain. Evals are how you replace vibes with evidence. Here's how to build them.
Evals vs observability: both, not either
A quick distinction, because they're confused: observability tells you what happened in production (traces, cost, errors). Evals are a controlled test set you run before shipping, on every change, to answer did this change help? One is a dashboard; the other is a test suite. You need both, and this post is about the second.
Building a golden set
The foundation is a set of tasks with known-good outcomes:
- Start small: 20 to 50 real tasks. Real beats synthetic; pull from actual usage. You don't need thousands to catch most regressions.
- Cover what matters plus what broke. Include the common cases and a growing regression set: every bug you fix becomes an eval case, so it can't silently come back.
- The hard part is "known-good." Curating the correct outcomes is the real work. Do it carefully, because the whole thing rests on it.
Three ways to grade
1. Deterministic: use it wherever you can. Exact match, JSON-schema validation, tests pass, a property holds. Cheap and objective, no arguing with the result. If your task has a checkable answer, this is the grader. (It's the same verification gate that powers a good router.)
2. LLM-as-judge: for the subjective stuff. When "correct" is a matter of quality, helpfulness, tone, or prose accuracy, have a strong model score the output against a rubric. Powerful and scalable, and full of traps (below).
3. Human: the calibration standard. The gold reference, too expensive to run on every change. Use it to calibrate your LLM judge, not as the everyday grader.
LLM-as-judge, done right
The judge is itself a fallible model, so engineer it like one:
- Give it a concrete rubric (not "is this good?") with specific, independently-checkable criteria. Vague rubrics produce noisy scores.
- Prefer pairwise comparison ("is A or B better?") over absolute scoring when you can, since models are more reliable at comparing than at assigning a number.
- Watch the known biases. Judges exhibit position bias (favoring the first or second option), length bias (longer = better, falsely), and self-preference (favoring output from the same model family). Randomize positions, control for length, and consider a different model as judge than the one being judged.
- Validate the judge against humans. Spot-check its scores against human labels; if they diverge, fix the rubric before you trust the judge at scale. An uncalibrated judge is just a confident number.
An LLM judge is a measurement instrument. An uncalibrated instrument that returns precise numbers is worse than no instrument — it's wrong with authority. Calibrate it against humans before you believe it.
Eval-driven development
Wire the evals into your workflow like tests:
- Run them on every change: prompt, model, tool, retrieval. In CI if you can.
- Track the score over time and block regressions the way you'd block a failing test.
- Change one variable at a time. Swapping the model and the prompt and the retrieval at once means you're measuring noise. Hold the rest fixed (the test-the-loop-not-the-model discipline).
The lazy version
You do not need an eval platform on day one:
- 20 real tasks, graded deterministically where possible and with a rubric-driven LLM judge for the rest.
- Run it on every change; eyeball the diff in scores.
- Grow the set every time something breaks.
That's real eval coverage. It's the difference between improving your AI feature on purpose and changing it at random. Everything fancier, automated judge pipelines, large labeled datasets, a dashboard, earns its place only once this minimal loop proves it isn't enough (ponytail, applied to your tooling).