The conversation about safe AI agents has matured past "add a content filter." Anyone who has shipped an agent that talks to real customers knows the failure modes are not just toxic language, they are agents giving advice they shouldn't, agents inventing policy that doesn't exist, agents leaking information about other accounts, and agents going off-script in ways that create legal or commercial exposure.
The fix is a layered architecture: a pre-LLM classifier that decides whether a request should reach the model at all, and a post-LLM judge that decides whether the model's response should reach the user. These are different jobs, with different costs, latencies, and failure modes. This article unpacks how each one works, when each is the right tool, and what we've learned shipping both for B2B and D2C operators with real compliance requirements.
Why "Guardrail" Is Doing Too Much Work as a Word
The term gets used for everything from a regex on profanity to a full secondary model evaluating outputs against a rubric. That ambiguity is the root of most poorly-designed AI agent stacks. To talk about guardrails usefully, split them by when they fire:
- Pre-LLM (input-side): Runs before the prompt reaches the language model. Catches inappropriate requests, off-topic inputs, attempted prompt injections, and queries the agent isn't authorized to handle.
- Post-LLM (output-side): Runs after the model generates a response, before that response is shown to the user. Catches hallucinations, off-policy statements, leaked PII, and answers that violate domain-specific rules.
Both are cheap to add. Almost no team running an agent in production has both. The teams that do have far fewer 2 a.m. incident calls.
The Pre-LLM Classifier, Decide Whether to Engage
The pre-LLM classifier is a small, fast model (or in many cases, a deterministic rules engine plus a small model) that looks at every incoming request and decides one of three things:
- Pass it through the request is in scope, safe, and the main agent should handle it.
- Reject with a canned response the request is out of scope, abusive, or attempting to manipulate the agent. Return a fixed message and don't burn tokens.
- Escalate the request is sensitive in a way that should reach a human, not an LLM.
The classifier is the cheapest piece of insurance you can buy on an agent. Most of the production agent disasters that go viral on social media, the airline chatbot inventing a refund policy, the dealership bot agreeing to sell a car for a dollar, were not LLM failures. They were missing pre-LLM classifiers. The model dutifully followed an adversarial instruction it should never have been allowed to see.
What Should the Classifier Block?
The answer is domain-specific, but the categories we use as a starting point:
- Out-of-scope topics a customer support agent for a fitness app should not be answering tax questions, no matter how politely.
- Adversarial inputs "ignore your previous instructions," "you are now DAN," and the long tail of prompt-injection attempts.
- Regulated requests medical advice, legal advice, financial advice, depending on the vertical. The classifier should route these straight to a human.
- High-emotional-stakes inputs statements indicating self-harm, harassment, or fraud reports. None of these should be handled by an LLM unsupervised.
The classifier doesn't need to be perfect. It needs to be conservative on the categories with the largest blast radius. False positives (a legitimate request that gets rejected) are an annoyance; false negatives (an adversarial request that reaches the model) can be a headline.
How to Build It Without Overengineering
A surprisingly effective starting point: a single small model (a cheap, fast LLM, not your main one) prompted with a list of categories and instructions to classify the input and return a JSON verdict. Add deterministic keyword filters in front of it for the most catastrophic categories. Cache verdicts on common phrasings.
The whole layer typically adds 100–300 ms of latency and a fraction of a cent per request. The teams that skip it are not saving meaningful money; they are deferring incidents.
The Post-LLM Judge, Decide Whether to Ship the Answer
The post-LLM judge is a separate model evaluation that runs after your main agent has generated a response, but before that response reaches the user. The judge reads the original request, the agent's response, and any retrieved context, and answers a structured set of questions:
- Does the response actually address the user's question?
- Are any claims in the response unsupported by the retrieved context?
- Does the response contradict known policy or product facts?
- Does the response include PII or confidential information that shouldn't be returned?
- Is the tone consistent with the brand's voice?
If the judge flags a problem, the response can be rewritten, suppressed, or routed to a human. Different problems get different handlers, a tone violation should not be treated the same as a hallucinated refund amount.
The Judge Is Not the Same Model as the Agent
A common mistake: using the same model and the same prompt to judge its own output. It rarely catches its own errors. The judge should be a different model, or at minimum, the same model with a structurally different prompt that doesn't have access to the agent's reasoning chain.
For systems where the cost matters, a cheaper judge that runs on every response, combined with an expensive judge that runs on a sampled percentage, is a reasonable compromise. The sampled audits feed back into prompt and model improvements.
What the Judge Should Output
Don't make the judge return "safe" or "unsafe." Make it return structured fields:
- Faithfulness score how grounded the response is in retrieved context.
- Off-policy flags specific named violations (refund policy, pricing, sales claims).
- PII detected categories of PII present in the output.
- Suggested action pass, rewrite, suppress, escalate.
- Confidence how sure the judge is.
Structured judge outputs let you build aggregated dashboards over time: faithfulness scores trending down means your retrieval is degrading; off-policy flags spiking on a specific topic means your system prompt needs an update or your retrieval is missing a critical doc.
The System Prompt Is Where Most Teams Should Start
Before you build a classifier or a judge, write a system prompt that includes examples of correct and incorrect responses. Most agent failure modes in production can be addressed by showing the model what good and bad look like, not by telling it abstractly.
Good examples we include in agent system prompts:
- A correct response when the user asks something in scope.
- A correct refusal when the user asks something out of scope.
- A correct escalation when the user mentions a regulated topic.
- An incorrect response, with an explanation of why it's incorrect, for each of the failure modes you care about.
System prompts are not a substitute for classifiers and judges, but they are the prerequisite. A model with no examples of correct behavior will continue to fail in ways no guardrail can catch.
Decision Trees Are a Tool, Not a Shame
Not every interaction needs an LLM. We routinely build customer-facing agents where the first layer is a decision tree, deterministic, predictable, easy to audit, and the LLM only takes over when the decision tree's branches run out.
For agents in regulated industries, this matters more than guardrails. A decision tree handling the first three intents (status check, password reset, change of address) plus an LLM for everything else can be safer, faster, and cheaper than a fully LLM-driven agent.
The combination, tree for known intents, LLM for everything else, classifier and judge wrapping the LLM, is what production looks like for most B2B and D2C agents we ship.
How to Test Guardrails Before You Trust Them
Three test suites every agent stack should have:
- Adversarial input suite. A maintained list of prompt-injection attempts, jailbreak patterns, and off-topic provocations. Run on every model or prompt change. If the pass rate drops, you have a regression.
- Policy regression suite. Real questions with known-correct policy-grounded answers. The judge should not be flagging any of these.
- Hallucination probe suite. Questions where the correct answer is "I don't know" or "let me escalate." Agents that confidently fabricate answers to these are unsafe.
All three should run in CI before any prompt or model change ships. Without them, every change is a guess.
Frequently Asked Questions
What's the difference between an LLM classifier and an LLM judge?
A classifier runs on the input, before the main model sees it, and decides whether the request should reach the model. A judge runs on the output, after the main model has answered, and decides whether the response is safe and accurate enough to show the user. They solve different failure modes and should be built separately.
Can I just use OpenAI's moderation API as my guardrail?
The standard moderation APIs catch a narrow band of harmful content. They are not aware of your domain, your policies, your retrieval context, or your brand voice. They are a useful component of a pre-LLM classifier, not a replacement for one.
How much latency do guardrails add?
A well-built pre-LLM classifier adds 100–300 ms. A post-LLM judge adds another 300–800 ms depending on whether you wait for it before responding (you usually should) or stream the response and retract if needed (riskier, but lower perceived latency).
Do we need guardrails if we're using an off-the-shelf agent platform?
Most off-the-shelf platforms ship with very thin defaults. If you're in any regulated industry, or your agent makes decisions with real commercial impact (refunds, pricing, eligibility), you need your own classifier and judge layer regardless of the platform.
How do you evaluate whether the judge itself is correct?
The same way you evaluate the agent: with a curated set of inputs and known-correct verdicts. The judge gets its own regression suite, and its own audit cadence. The teams that don't audit the judge end up with an agent gated by an arbiter no one trusts.
Guardrails are unglamorous infrastructure, but they are the difference between an agent you can ship and one that ships you. If you're building an AI agent that touches customers, see how we ship production AI agents for B2B and D2C operators or book a strategy call to pressure-test your architecture.