A production LLM integration can begin as one provider key in an environment file and quickly become a control problem: inconsistent model choice, rising spend, sensitive data in prompts and no useful evidence of what happened to a request. To configure an OpenAI compatible API gateway is to place one governed doorway between your application and the providers behind it, without rewriting the application behaviour that already works.
The implementation change should be deliberately small. Your application continues to use its OpenAI client and request format. The gateway takes responsibility for applying request-level policy, selecting an eligible model and recording the operational trace.
What an OpenAI-compatible gateway changes
An OpenAI-compatible gateway accepts the API shape your application already sends. Instead of pointing the client directly at a single model provider, you point it at a gateway base URL and authenticate with the gateway key. The client code, message structure and completion-handling logic remain byte-for-byte identical apart from those connection settings.
That compatibility matters most when LLM calls sit inside products, internal copilots or automated workflows. Replacing SDKs or embedding provider-specific routing logic across services turns a platform decision into an application migration. A gateway keeps the decision at the infrastructure layer, where platform, security and FinOps teams can operate it consistently.
This is not simply an abstraction layer. A useful production gateway should make a decision for every request: whether the input is acceptable, whether sensitive data requires action, which route can serve the workload, whether the output is safe to return and how the request affects spend and audit evidence.
Configure the OpenAI compatible API gateway
Start by storing the gateway credential outside source control. In a deployment environment, set ROUTEUR_KEY through your existing secret-management process. Then change the client base URL and API key.
import os from openai import OpenAI
client = OpenAI( api_key=os.environ["ROUTEUR_KEY"], base_url="https://api.routeur.ai/v1", )
response = client.chat.completions.create( model="auto", messages=[ {"role": "user", "content": "Summarise the incident notes for the operations team."} ], )
print(response.choices[0].message.content)Using model="auto" delegates model selection to the gateway's routing rules and priorities. That is the right default when the objective is to use the most cost-effective capable model within the controls your organisation has set. It also avoids hard-coding provider choices into business logic that will be expensive to unwind later.
Where a workload has a defined operational purpose, use a plain logical route name instead. Named routes let teams separate workloads such as a customer-facing assistant, document classification or an internal drafting tool. The application expresses intent through the route; the platform team controls the eligible providers, models and priorities behind it.
The result is a clean division of responsibilities. Product teams own prompts and user experience. Platform teams own routing policy. Security and compliance teams own the controls that apply before data leaves the application boundary.
Keep provider selection out of application code
A common first attempt at multi-provider resilience is an application-side chain: try one provider, catch an error, then try another. This can help availability, but it duplicates logic across services and gives little central visibility into why a fallback occurred or which data rules were applied.
Central routing is more maintainable. Define priorities once, allow automatic failover within the approved route, and inspect request-level traces when behaviour needs explaining. If a team needs to test a particular provider or model during investigation, the supported request headers are Routeur-Provider and Routeur-Model. Those are operational controls, not a substitute for governed routing rules.
Put policy before routing
The order of controls determines whether a gateway actually reduces risk. routeur.ai processes a request through prompt shielding first, then DLP, then routing, followed by output moderation. That sequence is practical: hostile instruction patterns are assessed before the request is sent onwards; sensitive information is handled before a model provider receives the prompt; and the generated response is checked before it reaches the caller.
Prompt-injection and jailbreak shielding is particularly relevant for applications that retrieve documents, process uploaded content or ingest untrusted text. A malicious instruction can arrive indirectly through a source document rather than a user message. Treating all inbound content as equally trustworthy is a design choice that becomes harder to defend once workflows operate at scale.
DLP policies should reflect the use case rather than apply one blanket response to every field. Depending on the policy, a detected data category can be logged, warned on, redacted or blocked. Redaction replaces matched content with [REDACTED]; it is not tokenisation, and the original value is not restored later. For a low-risk internal workflow, warning may provide the evidence needed to improve prompts. For a regulated customer journey, blocking may be the appropriate response.
Output moderation completes the boundary. Even a well-controlled prompt can produce unsuitable text, especially where users can steer an assistant in unexpected directions. Applying moderation centrally means the same standard is not dependent on every individual team remembering to implement it.
Make cost controls enforceable
A gateway is most valuable when cost is a governed operational signal, not a monthly surprise. Start with hard spend caps and budgets for the teams, environments or routes that matter. A hard cap should alert or block further requests with a 429 response once the limit is reached. It should not quietly reroute requests in an attempt to bypass the budget.
This distinction matters. If a service exceeds its allocation, the owning team needs a visible decision: increase the budget, reduce usage or change the route's model policy. That is more accountable than allowing spend to continue in a less obvious form.
Routing rules can still reduce cost before a cap is reached. A well-designed route considers capability, price and provider availability, then uses priorities to select an appropriate model. The right configuration depends on the workload. A high-stakes analysis task may justify a narrower set of stronger models. Classification, extraction and routine summarisation often allow more flexibility. The goal is not to select the cheapest model at all times; it is to prevent teams from paying for excess capability by default.
Test configuration without changing production behaviour
Before enabling a new route or DLP action, validate it against representative requests. Use Routeur-Dry-Run where you need to inspect the decision path without relying on a live routing outcome. This is useful for checking whether a route would select the intended provider or whether a DLP policy would redact or block a realistic payload.
For live diagnostics, use Routeur-Trace and inspect the resulting per-request trace. Traces give engineering teams a way to answer practical questions: which route handled this call, what policy action occurred, did failover happen, and where did latency arise?
Logging policy deserves the same care as routing policy. Metadata-only logging by default limits unnecessary payload retention while preserving operational evidence. Payload retention should be an explicit opt-in decision with a defined purpose, access model and retention period. For UK and EU organisations, this approach supports a more defensible posture alongside EU data residency, GDPR processor obligations and audit trails that can support EU AI Act record-keeping.
Operate the gateway as shared infrastructure
Configuration is not finished when the base URL is deployed. Treat routes, policies and budgets as managed infrastructure with clear owners. Review traces after releases, investigate blocked requests for false positives, and revise model priorities when workload quality or provider economics change.
The benefit is cumulative. A single team can begin with the base URL and API-key change, while the organisation gains a controlled path to add providers, apply security policy and manage spend without asking every application team to rebuild its integration. Keep the doorway central, keep the application code simple, and let each request leave behind the evidence needed to improve the next one.