Never trust the model's output: the validation layer

You ask the model for JSON, it returns JSON, your code parses it and acts on it, and this works for weeks. Then one day it returns almost-JSON with a trailing comma, or perfectly valid JSON containing a field you never defined, or a string that happens to carry an instruction aimed at whatever consumes the output downstream. Your code, which trusted the output because it always had, does something it shouldn't. That's the day you learn the rule that should have been there from the start: the model's output is untrusted until you've checked it.

This is the mirror image of the input-side security principle. You already know not to trust user input. The model is the same kind of boundary on the way out. It's a probabilistic text generator, and probabilistic means it will, eventually, return something malformed, something wrong, something outside policy, or something an injection turned into a weapon. The validation layer is the part of your architecture that stands between the model and the rest of your system and refuses to pass anything through until it's been checked.

There are three things worth checking, in roughly increasing difficulty.

Structure is the easy one. It's also the one people half-do. If you expect a schema, validate against it. Constrained generation and structured-output features make valid output likely, and you should use them, but likely is not guaranteed, so you still validate after, because the cost of a malformed response slipping into code that assumed it was well-formed is a crash or worse. The instinct to call json.loads on the response and trust the result is the instinct to design out.

Content and policy is the next layer, and it comes down to whether the output is something you're willing to emit. This is where you catch leaked secrets, a model that's about to repeat its own system prompt back, personally identifiable information that shouldn't go out, or content that's simply off-topic for the surface it's appearing on. These are filters on the way out, and they're the part most people mean when they say "guardrails."

Groundedness is the hard one, and it's specific to RAG and anything that's supposed to answer from sources. Does the answer actually follow from the retrieved material, or did the model confabulate something plausible that the sources don't support? You can't check this with a regex; it usually takes a second model evaluating the answer against the cited context. It's more expensive, and for high-stakes answers it's worth it, because a confident fabrication that passed structural validation is still wrong.

What the validation layer enables is the discipline of validate-then-act: no model output triggers a consequential action until it's passed validation. The sharp edge of this is that you never execute, render, or evaluate model output, and never pass it unescaped into a shell, a query, or a page, because an injection in the output becomes command execution or cross-site scripting the moment you trust it. Output that fails validation gets retried, ideally with the validation error fed back so the model can correct itself, and if it keeps failing you fail closed, you do not proceed with garbage because the retries ran out. Bound those retries so a stubbornly-invalid response doesn't loop forever.

Now the part that matters and gets missed: a guardrail is a layer, not a cure, and confusing the two is dangerous. A classifier that screens outputs for bad patterns raises the bar and catches the obvious cases. It's genuinely useful. It's also bypassable, because it's a model checking a model, and a determined injection or an unusual input can slip past it. So a guardrail is a filter, not a security boundary. If the thing you're protecting against is serious, least privilege and egress control are the boundary, and the guardrail is a helpful extra layer on top, never the thing you're relying on. Teams that treat an output classifier as their security model have a false sense of safety, which is worse than none.

Architecturally, the natural home for all of this is between the model and everything downstream, often inside the gateway your calls already route through, so that output validation lives in one place the way input validation does, rather than being reinvented or forgotten at each call site. The model is the one component in your system you cannot trust to return what you asked for. Wrap it accordingly, fail closed when it's wrong, and never mistake the filter for the wall.