A production copilot can pass every pre-release test and still create a material incident on Monday morning. A provider may change behaviour, a user may insert a hidden instruction, a new team may send personal data through an unapproved model, or a seemingly small feature may double monthly inference spend. Model risk management is the operating discipline that makes those events visible, controllable and defensible.
For organisations using large language models, the issue is not whether a model is capable. It is whether every request is being handled within the right security, data-protection, cost and accountability boundaries. That requires controls where requests actually flow, not a policy document stored separately from the application.
Model risk management starts at runtime
Traditional model risk management grew up around relatively stable statistical models. Teams could validate a model, document its inputs and assumptions, approve its release, then review performance on a schedule. That remains useful for credit, fraud and forecasting systems, but LLM applications introduce a different operating reality.
The model itself may be selected dynamically. Providers can experience availability changes. Prompts are partly supplied by users, and their content may attempt to override system instructions. Inputs can contain personal or commercially sensitive information. Outputs can be unsafe, inaccurate or unsuitable for the business process they enter. Cost also varies by model, context length and usage pattern.
A sensible control framework therefore has two layers. The first is design-time governance: define permitted use cases, acceptable providers, data classifications, evaluation criteria, owners and approval routes. The second is runtime governance: inspect and control each request before it reaches a provider, then assess the returned output before the application uses it.
Neither layer replaces the other. A well-written risk assessment cannot stop an unexpected sensitive-data upload. Equally, a gateway rule cannot establish whether an AI use case should exist in the first place. Production control comes from connecting governance intent to technical enforcement.
The risk is in the system, not only the model
Treating a provider model as the entire risk surface is a common mistake. The relevant unit is the AI system: the application, prompt construction, retrieval sources, users, model configuration, provider, output handling and downstream workflow.
Consider an internal assistant that drafts customer responses. Its model may be approved, but risk still changes if a product team adds CRM context, allows staff to paste account details, or automatically sends generated text without human review. A model card or annual vendor review will not reveal those changes quickly enough.
This is why effective ownership is shared. Product teams own the intended use and customer outcome. Engineering owns application behaviour and integration. Security owns protective controls. Compliance and legal define the applicable requirements. FinOps or platform teams need visibility of consumption and limits. The practical goal is not to create a committee for every prompt. It is to give each group evidence and controls appropriate to its responsibility.
Build controls around the request path
A governed control plane gives teams one governed doorway between applications and model providers. It centralises the controls that otherwise become inconsistent middleware, duplicated policy logic and incomplete logging across products.
The sequence matters. A request should first pass through a prompt shield to identify prompt-injection and jailbreak attempts. DLP checks then assess sensitive information and apply the chosen action: log, warn, redact using [REDACTED], or block. Only then should routing select a capable provider and model. Returned content should pass through output moderation before the application receives it.
That order protects both the business and the integrity of later decision-making. Routing first could send content to a provider before the organisation has applied its data rules. Moderating only the final output overlooks attacks embedded in user input. Centralising this path also means a policy change applies consistently, rather than waiting for multiple application releases.
For many teams, deployment should not require a new application architecture. An OpenAI-compatible gateway can preserve existing client behaviour while moving the base URL and API key to the control plane. For example:
import os from openai import OpenAI
client = OpenAI( api_key=os.environ["ROUTEUR_KEY"], base_url="https://api.routeur.ai/v1" )
response = client.chat.completions.create( model="auto", messages=[ {"role": "user", "content": "Summarise these support notes."} ] )The application’s business logic stays byte-for-byte identical. The operational policy sits at the governed doorway instead: a named route may constrain a workload to approved options, while routing rules and priorities can choose the most cost-effective capable model and fail over when necessary. A team can also use a plain logical route name where a workload needs a defined model policy rather than automatic selection.
This approach is particularly useful when an organisation has grown through experimentation. One product may call one provider directly, another may have its own proxy, and an internal tool may be using a developer key with no clear owner. Consolidation does not make every risk disappear, but it establishes a reliable enforcement point.
Evidence must be useful without becoming a new exposure
Auditability is often confused with retaining everything. For LLM workloads, indiscriminate prompt and output retention can create a second sensitive-data store that is difficult to justify and protect. The better default is metadata-only logging, with payload retention enabled only when there is a clear diagnostic or governance need.
Per-request traces should show what happened operationally: the route used, provider and model selection, policy outcomes, latency, consumption and relevant request metadata. That gives engineering teams a way to investigate failures, gives FinOps a view of spend drivers, and gives compliance teams evidence that controls operated.
The level of detail depends on the use case. A customer-facing healthcare workflow may require stricter payload handling than an internal code-assistance service. A regulated decision-support process may need stronger review and record-keeping than a low-risk drafting tool. The principle is consistent: retain enough evidence to explain control operation, but no more sensitive content than necessary.
Scoped API keys are part of this discipline. They make it possible to limit access by team or workload and to trace usage to an accountable owner. Combined with named routes, they reduce the chance that a temporary experiment becomes an uncontrolled production dependency.
Spend is a model risk control, not just a finance metric
Unbounded AI spend can become an operational incident. A prompt loop, a sudden increase in context size or an enthusiastic product launch can consume budget before a monthly report surfaces the change. Cost controls belong in model risk management because they define the conditions under which an AI service may continue operating.
Hard spend caps provide a clear boundary. When a cap is reached, requests can be blocked with a 429 rather than silently continuing to accumulate cost. That may be the right outcome for a non-critical feature; for a customer-critical workflow, teams should set limits and route design with the service impact in mind.
Cost optimisation also needs judgement. The cheapest model is not automatically suitable, and the most capable model is not automatically necessary. Evaluate workloads against the quality, latency, tool-use and safety requirements that matter to the user. Then use routing to match work to capable models, while monitoring the result. routeur.ai brings that routing, policy enforcement, data controls and spend governance into the same request path, rather than asking teams to operate separate layers.
Make control operation testable
Governance is credible only when teams can show that a control works under realistic conditions. Test prompt-injection defences with adversarial inputs. Verify that DLP redaction produces [REDACTED] where expected and that block rules prevent transmission. Confirm that an unapproved provider cannot be reached through a named route. Exercise a spend cap before a production launch, not after an incident.
Request tracing supports this work. During testing, the Routeur-Trace header can help teams inspect request-level behaviour. The Routeur-Dry-Run header is useful when assessing what a routing or policy decision would do without treating the result as a normal production change. Where diagnosis requires it, Routeur-Provider and Routeur-Model can make provider or model selection explicit within the documented request controls.
Control testing should be repeated when the system changes: a new provider, a revised prompt template, an altered retrieval source, a new geography, or an automated downstream action. The cadence depends on risk, but change events matter more than calendar dates alone.
For UK and EU organisations, the evidence should also map to real obligations. EU data residency, GDPR processor terms and audit trails suitable for EU AI Act record-keeping address different parts of the assurance picture. They do not remove the need for use-case ownership, human oversight where appropriate, or clear internal decision records.
The most useful outcome is not a larger risk register. It is a production environment where every AI request has an owner, a permitted path, enforceable boundaries and evidence of what occurred. That is the point at which AI delivery can move faster without asking security, compliance and finance to accept blind spots.