An LLM feature can move from a controlled pilot to an unaccountable production dependency in a single release. One team selects a premium model for reliability, another sends customer text directly to a different provider, and a third adds logging that captures more payload than security approved. AI governance software addresses that operational gap by placing controls at the request boundary, where model choice, sensitive data, cost and policy can be handled consistently.
For engineering leaders, the question is not whether a model has passed an initial assessment. It is whether every request made next week, by every application and team, will still follow the rules. Governance that depends on developers remembering provider-specific settings, copied middleware or manual cost reviews will eventually drift.
What AI governance software must control
Useful governance software is not a document repository and it is not a dashboard that reports a problem after the response has reached a user. It is a control plane that sits between applications and model providers: one governed doorway through which production LLM traffic passes.
That position matters because it allows policies to be enforced before data leaves the application boundary and before a model response is returned. A practical system should govern four connected decisions: whether a prompt is safe to process, whether it contains protected data, which available model should handle it, and whether the output is suitable to return.
The order is significant. Prompt shielding comes first, followed by DLP inspection, then routing, then output moderation. Treating those functions as separate tools can create gaps. A request may be routed cheaply but still expose personal information, or a safe input may produce unsafe output. Request-level enforcement keeps the sequence explicit and inspectable.
This is also where AI governance becomes an engineering concern rather than a quarterly compliance exercise. Security leaders need policies that hold under real traffic. FinOps teams need spend controls that act before a bill becomes a surprise. Platform teams need an implementation pattern that does not require every product squad to build and maintain its own provider layer.
Governance should not require an application rewrite
The adoption test is straightforward: can a team introduce control without changing its business logic? If the answer involves a new SDK, provider-specific branching, or weeks of application changes, governance will arrive unevenly and late.
A governed API gateway is designed to reduce that friction. Applications continue making OpenAI-compatible requests, while the platform team changes the API key and base URL. The application behaviour and business logic remain byte-for-byte identical. Model selection can use auto where routing rules determine the most cost-effective capable option, or a plain logical route name where a workload needs a defined path.
That does not mean every workload should be treated identically. A customer-facing legal workflow may justify a higher-performing model and a more restrictive data policy. Internal content classification may be suited to a lower-cost option. The point is to make those choices named, deliberate and centrally managed, rather than embedded across repositories.
Scoped API keys add another useful boundary. A key can represent a team, application or environment, making access and usage attributable without relying on informal conventions. Named routes and routing priorities then let platform teams express intended service behaviour while retaining provider flexibility and automatic failover.
routeur.ai applies this model as a production control plane across OpenAI, Google Gemini, DeepSeek, Anthropic Claude, Mistral and Cohere. The aim is not to make model choice invisible. It is to make the decision governable, measurable and reversible.
Put data protection in the request path
Sensitive data handling is where abstract AI policy most often meets production reality. A support assistant may receive an email address in a troubleshooting message. A finance workflow may include account information in a document extract. A developer may paste credentials into a prompt while investigating an incident.
DLP-based PII controls should give teams a defined response for these cases: log, warn, redact or block. The right action depends on the workload and risk appetite. Logging may suit a monitored internal trial. A warning can surface behaviour that needs correction. Redaction replaces detected content with [REDACTED], while blocking prevents the request from continuing where sending the information is unacceptable.
Redaction has a trade-off. Removing identifying content can reduce the quality of a task that genuinely depends on it. That is a reason to classify the workflow and select a proportionate policy, not a reason to leave the decision to each application. Teams should decide which fields are necessary for the model to perform useful work, then apply the minimum exposure needed.
Prompt-injection and jailbreak shielding belongs in the same request path. A model may be capable, but it cannot independently enforce an organisation's boundary between trusted instructions and untrusted content. Shielding before routing helps prevent hostile or manipulative prompt content from reaching the provider as an ordinary request.
Output moderation completes the control loop. The input may meet policy, yet the generated response may still be unsuitable for a customer, employee or automated downstream process. Moderation provides a final enforcement point before the response returns to the application.
Make evidence useful without retaining everything
Auditability is often misunderstood as collecting every prompt and response forever. For many teams, that approach creates another data protection problem. It increases the amount of sensitive content held in logs and expands the scope of an incident if access controls fail.
A more disciplined default is metadata-only logging. Teams can retain the operational evidence needed to understand a request - such as route, provider, model, latency, policy outcome, cost and trace information - without automatically retaining payloads. Payload retention should be an explicit opt-in decision for workflows that genuinely need it.
Per-request traces make that evidence actionable. During an incident or quality investigation, an engineer should be able to establish what happened to a request: which policies were evaluated, which route was selected, whether a provider failed over, and how the request contributed to spend. Trace controls also support controlled diagnostics when a team needs greater visibility for a specific call.
For UK and EU organisations, this operating record supports the practical record-keeping expected around higher-risk AI use. It does not remove the need for governance processes, impact assessments or accountable owners. It gives those processes technical evidence instead of relying on spreadsheets and recollection. EU data residency and GDPR processor terms may be decisive requirements, but they should be assessed alongside the actual data flow, retention settings and organisational obligations.
Control spend where requests are made
LLM cost management is frequently treated as a reporting problem. Monthly provider invoices are useful, but they cannot stop a loop, a faulty deployment or a newly popular feature from consuming budget during the month.
Hard spend caps and budgets operate closer to the point of use. When a cap is reached, the system can alert or block further requests with a 429 response. That is intentionally different from silently routing traffic elsewhere. A budget control is a financial boundary, not a quality-of-service policy, and teams should know when it has been reached.
Routing is where cost optimisation belongs. A route can select the most cost-effective capable model for the request, with priorities and automatic failover designed around availability and intended performance. This avoids a common pattern in which every task is sent to the most expensive model because it was the first integration completed.
Savings depend on workload shape, prompt size, output length, model mix and quality requirements. About 20% average savings is a useful benchmark, not a promise that every application will achieve the same result. Measure quality and task completion alongside cost. A cheaper response that causes more manual review may cost more overall.
A sensible path to production governance
Start with the applications that already have meaningful traffic, sensitive inputs or unpredictable spend. Route them through one control point and establish a small set of policies that are easy to explain: what must be blocked, what may be redacted, which routes are approved, and who owns each budget.
Use dry-run behaviour to understand the effect of a proposed policy before enforcing it broadly. Review per-request traces with the teams operating the workload, especially where a DLP rule, shield or moderation outcome changes expected behaviour. This is how policy becomes an engineering feedback loop rather than a remote compliance instruction.
Then standardise the pattern for new services. Teams should not need to negotiate security, routing and logging design from first principles whenever they add an LLM feature. They should inherit a governed doorway and retain enough flexibility to choose the route appropriate to their use case.
The practical goal is not to constrain useful AI work. It is to make each production request explainable, protected and economically intentional - before it becomes somebody else's incident.