A production LLM request can expose customer data, trigger a policy breach, select an unnecessarily expensive model or produce an unsafe response in a fraction of a second. An AI governance framework must therefore operate where those decisions occur: on every request, before data reaches a model provider and before output reaches a user or downstream system.
For engineering and compliance teams, governance is not a document describing acceptable AI use. It is the set of technical controls that make acceptable use enforceable, observable and repeatable across applications, teams and providers. The test is practical: can you show what happened to a specific request, which policy applied, where it was processed and why it was allowed, changed or blocked?
What an AI governance framework must control
A useful framework connects organisational accountability to request-level enforcement. It should set the rules for approved use cases and data, but it must also make those rules executable in the production path. Otherwise, governance depends on individual application teams implementing the same controls correctly, provider by provider.
Four areas need to work together:
- Identity and ownership: Know which team, application and scoped API key made a request, and who owns the relevant route and policy decisions.
- Data and security: Inspect for prompt injection, jailbreak attempts and sensitive data before sending a request to a provider. Apply a defined action when a match occurs: log, warn, redact or block.
- Model and output control: Define which providers and models can serve which workloads, then moderate generated output before it is returned.
- Cost and evidence: Apply hard spend caps and budgets, and retain per-request traces that explain routing and enforcement decisions without collecting more payload data than necessary.
These are interdependent controls. A spend cap without attribution cannot resolve which product caused the increase. A provider policy without routing rules can be bypassed by an application using a direct provider key. Logging every prompt for audit purposes can itself create a sensitive-data retention problem.
The architectural answer is one governed doorway between applications and model providers. Applications send requests through a central control plane, rather than each service independently deciding which provider to call and what to record. This separates business logic from operational policy: teams can continue building product features while platform, security and FinOps teams manage controls in one place.
Start with decisions, not a policy catalogue
Many governance programmes begin by listing principles such as fairness, transparency and human oversight. Those principles matter, particularly for high-impact use cases, but they are too broad to configure an API gateway. Start instead with the decisions your organisation must make repeatedly.
For each LLM workload, establish whether it is customer-facing or internal, what data classification it handles, which model capabilities it needs, what failure behaviour is acceptable and who can change its configuration. A support summarisation flow may permit a lower-cost model after PII redaction. A legal drafting workflow may require a named route, stricter provider constraints and mandatory human review. The framework should make those differences explicit rather than applying one generic standard to every prompt.
This is where trade-offs become visible. Blocking all detected personal data can protect against leakage but can also break legitimate workflows that need names or account references. Redaction reduces exposure but may reduce response quality when the missing value is material to the task. Logging warnings can support early rollout and tuning, while blocking is appropriate when the risk is unacceptable. There is no universal action; there should be an accountable decision, a reason and a consistent enforcement point.
Put controls in the request path
Governance becomes credible when it is applied in a fixed, inspectable sequence. Before routing a request, first run prompt shielding to identify prompt-injection and jailbreak patterns. Next, apply DLP controls to detect sensitive information and perform the configured action. Only then route the permitted request to an eligible provider and model. Finally, moderate the output before it reaches the calling application.
That order matters. Routing before data controls could send information to a provider before a redaction or block decision is made. Moderating only output ignores a major source of risk: malicious or sensitive instructions entering the system. A clear sequence also gives security teams an operational model for testing controls and investigating incidents.
DLP policies need predictable behaviour. Where redaction is selected, the relevant content becomes [REDACTED]; it is not converted into a recoverable substitute. Where blocking is selected, the request does not proceed. Teams should test these outcomes against real prompt patterns and business workflows before enforcing stricter actions globally.
The same principle applies to model selection. A framework should distinguish between capability requirements and provider preference. Some requests need strong reasoning, others need speed or lower cost, and some need a particular data-residency posture. Named routes, routing rules and priorities make those decisions visible and manageable. Automatic failover can improve continuity, but it must remain within the allowed provider and model boundaries for that workload.
Keep the application change small
A framework that demands every product team rebuild its integration will remain incomplete for years. A governed gateway should be deployable with a controlled configuration change, preserving existing application behaviour while moving provider access behind central policy enforcement.
For OpenAI-compatible applications, the integration can be limited to the base URL and API key. The application can continue to request auto while routing policy determines the most cost-effective capable approved model.
import os from openai import OpenAI
client = OpenAI( api_key=os.environ["ROUTEUR_KEY"], base_url="https://api.routeur.ai/v1", )
response = client.chat.completions.create( model="auto", messages=[ {"role": "user", "content": "Summarise this support case."} ], )That small change has a significant governance effect. Applications no longer need embedded provider keys, separate moderation integrations or duplicated cost logic to participate in central controls. Their business logic stays byte-for-byte identical, while policy can evolve without application code deploys.
routeur.ai applies this model as a governed control plane across OpenAI, Google Gemini, DeepSeek, Anthropic Claude, Mistral and Cohere. The value is not merely another endpoint. It is the ability to apply routing, DLP redaction, prompt shielding, output moderation, budgets and traceability to the same request path.
Make evidence proportionate to the risk
Auditability does not mean retaining every prompt and response indefinitely. For many workloads, metadata-only logging provides the evidence needed to understand request volume, route selection, provider usage, latency, policy outcomes and spend, while reducing the sensitivity of retained records. Payload retention should be explicit opt-in, with a defined purpose, access model and retention period.
Per-request traces are the operational record that turns policy into evidence. A trace should allow an authorised reviewer to understand how a request moved through shielding, DLP, routing and output moderation, alongside the resulting decision. This supports incident investigation, internal assurance and the record-keeping expectations that arise under the EU AI Act without treating compliance as a once-a-year exercise.
Trace data also improves engineering decisions. If a route regularly falls back, the issue may be provider availability, a route priority that needs adjustment or a capability mismatch. If DLP warnings cluster around one application, that team may need a different input design rather than a looser policy. Governance telemetry should be used to improve systems, not only to produce reports after a problem.
Treat cost control as a policy control
Uncontrolled LLM spend is often framed as a procurement problem. In production, it is a runtime control problem. A model choice made for one request can be appropriate; the same choice for millions of routine requests may not be. Intelligent routing can select a cost-effective capable model, while hard spend caps and budgets prevent an account or project from exceeding its approved limit.
A hard cap has to mean something operationally. When the limit is reached, the system should alert or block further requests with a 429 response rather than silently switching traffic in a way that changes the established policy. Teams can then decide whether to increase the budget, adjust routing rules, reduce unnecessary usage or redesign the workflow.
Savings should not become the only metric. A lower-cost route that degrades answer quality, increases latency or pushes a sensitive workload outside approved boundaries is not a governance success. Evaluate cost alongside task success, provider reliability, policy outcomes and user impact. The right model is workload-specific, and the right route can change as model capability and pricing change.
Assign owners and rehearse exceptions
A framework works when responsibilities are concrete. Platform teams should own gateway operation, route configuration and service reliability. Security teams should own control requirements and incident pathways. Privacy and compliance teams should define data handling and evidence expectations. Product owners should remain accountable for whether a use case is appropriate and whether users receive the right disclosures or review points.
Exceptions need the same discipline as normal operation. Define who can approve a new provider, allow payload retention, alter a DLP action or raise a hard cap. Set a review date and record the business reason. Temporary exceptions that are never revisited become the informal architecture of an AI estate.
The practical goal is not to slow down model adoption. It is to give every new use case a known path into production: a scoped key, an approved route, request-level controls, measurable cost and evidence that the system behaved as intended. When a team can add that control through one governed doorway, governance becomes part of delivery rather than a gate at the end of it.