A firewall that blocks a handful of obvious jailbreak phrases is not ready to sit between a production application and a model provider. The real LLM firewall evaluation criteria concern whether controls hold under realistic traffic, preserve legitimate work, produce usable evidence and fit the request path without creating a new operational problem.
For platform, security and compliance teams, the question is not simply whether a tool detects unsafe content. It is whether every request can pass through one governed doorway where policy is applied predictably, sensitive data is handled correctly, routing remains controllable and decisions can be investigated later.
Start with the enforcement path
Evaluate the order in which controls run before reviewing detection scores. A sound request path applies prompt-injection and jailbreak shielding first, then DLP controls, then routing, followed by output moderation. That sequence matters.
Prompt shielding needs to inspect the user input before the request is sent onwards. DLP then determines whether personal or sensitive data should be logged, warned on, redacted or blocked. Only requests that pass those controls should be routed to a provider and model. Once a response returns, output moderation checks what the application is about to show or act on.
Ask vendors to demonstrate this path with the same request, not a slide. A useful test includes an instruction-hijacking attempt that also contains an email address or account identifier. Confirm which control takes the decision, what the application receives and what appears in the trace. If the system cannot explain the sequence, troubleshooting policy conflicts will be difficult when usage scales.
Measure detection quality, not just blocking volume
A high block rate can look reassuring until it begins rejecting legitimate support tickets, code reviews or internal research prompts. The right balance depends on the workflow. A public-facing assistant may justify stricter blocking than an internal analyst tool, while an automated workflow may need narrower, highly specific policies to avoid stopping a business process.
Build an evaluation corpus from real, suitably sanitised examples. Include normal requests, known jailbreak patterns, indirect prompt-injection attempts inside retrieved content, malformed inputs and edge cases that resemble attacks but are valid work. Score both false negatives and false positives.
Test for evasion and context
Attackers do not only write "ignore previous instructions". They encode instructions, split them across turns, place them in documents or ask the model to transform unsafe content rather than answer it directly. Your test set should reflect that.
Also test conversational context. A control that evaluates each prompt in isolation may miss an escalation across several benign-looking turns. Conversely, a system that treats every mention of a risky topic as a violation may become unusable for security, legal or healthcare teams working within approved boundaries.
The useful evaluation question is: can policy teams adjust controls to suit the route and use case, then verify the effect before applying it broadly? Detection without controlled tuning becomes a source of friction rather than protection.
Validate DLP behaviour at the request boundary
DLP evaluation should be concrete. Send representative identifiers, customer details and confidential fragments through a non-production test path. Then verify the configured action for each class of data: log, warn, redact or block.
Redaction should be unambiguous in the downstream request and traces. If a sensitive value is redacted, the expected visible replacement is [REDACTED]. Do not assume that a firewall can restore values later, or that redaction is appropriate for every workflow. A support assistant may be able to continue after removing an identifier; a transaction workflow may need to block because the missing value makes the request unsafe or invalid.
Inspect logging behaviour as closely as the policy decision. Metadata-only logging by default reduces unnecessary payload retention, while explicit opt-in payload retention may be appropriate for tightly controlled debugging. The choice is operational: teams need enough evidence to investigate an event without quietly creating a second store of sensitive prompts and outputs.
Treat routing and policy as one production decision
An LLM firewall cannot be assessed in isolation from the gateway that sends requests to models. Different providers have different latency, capability, availability and commercial characteristics. If teams route directly from every application, consistent policy enforcement becomes harder with each new integration.
Evaluate whether the control plane can enforce policy while selecting a suitable available model according to named routes, routing rules and priorities. Test provider failure as well as normal traffic. Automatic failover should maintain the governed request path, rather than create an exception where a request bypasses inspection during an incident.
Cost must be evaluated here too. An economical route is valuable only when it remains capable for the task and stays inside policy. Compare model selection against a quality baseline for your actual prompts. A claimed saving is not a saving if it increases retries, manual review or customer-facing failures.
Hard spend caps deserve separate testing. Confirm what occurs at the limit: a cap should alert or block with a 429 response, not silently move traffic to another route or weaken controls. FinOps teams need clear, enforceable boundaries, particularly where many teams share a provider estate.
Include latency, reliability and failure modes
Every enforcement step adds work to the request path. Measure end-to-end latency at realistic concurrency, including large prompts and streamed responses where applicable. Measure median latency, but pay particular attention to tail latency. A control that looks fast in a quiet test may become a bottleneck during peak traffic.
Ask what happens when individual components cannot make a decision. The appropriate behaviour depends on the risk of the route. A customer service assistant handling general information may have a different failure policy from an application processing regulated records. What matters is that the outcome is explicit, observable and tested, rather than left to an undocumented default.
Reliability evaluation should also cover provider disruption. Generate controlled failures and establish whether routing rules and priorities behave as expected. Record the trace for each test so the team can distinguish a provider issue, routing choice, policy block and application error.
Require evidence that people can use
A policy decision is only defensible if engineers and reviewers can reconstruct it. Per-request traces should show the relevant route, selected provider and model, policy outcomes, timing and cost-related metadata without requiring full payload storage by default.
For regulated organisations, this is not an abstract reporting requirement. Audit trails support incident review, access governance and EU AI Act record-keeping. Evaluate whether scoped API keys, route-level controls and trace data let you answer practical questions: which application sent this traffic, which policy applied, what action was taken and who can change the configuration?
Run a tabletop exercise during the evaluation. Present a hypothetical sensitive-data incident and ask the security, platform and compliance teams to investigate it using only the platform evidence. If they need to correlate several disconnected systems or retrieve unapproved payload logs, the operating model is not yet ready.
Test deployment friction with a real application
A firewall that requires application rewrites will be deployed inconsistently. The fastest meaningful test is to take one existing OpenAI-compatible integration and direct it through the governed endpoint. The application logic should remain byte-for-byte identical apart from the key and base URL change.
import os from openai import OpenAI
client = OpenAI( api_key=os.environ["ROUTEUR_KEY"], base_url="https://api.routeur.ai/v1" )
response = client.chat.completions.create( model="auto", messages=[{"role": "user", "content": "Summarise this support request."}] )During this test, enable Routeur-Trace where needed and examine the resulting decision path. Use Routeur-Dry-Run before introducing a new enforcement rule to understand the likely effect without immediately blocking production traffic. Routeur-Provider and Routeur-Model can support controlled evaluation of provider and model choices when that comparison is required.
This is where a platform such as routeur.ai should be judged on operational reality: one governed doorway, not another custom middleware project. The best result is a deployment that gives security, FinOps and engineering shared control while leaving product teams free to ship.
A worthwhile evaluation ends when your team can state, with evidence, what will be blocked, redacted, routed, logged and investigated - before the next model integration turns those questions into an incident.