An LLM integration often starts with a small wrapper around one provider SDK. Then a second team needs a different model, Finance asks where the spend went, Security finds prompts containing personal data, and an outage makes the single-provider dependency visible. At that point, the API gateway versus custom middleware decision is no longer an implementation preference. It determines where control lives and how reliably it can be applied.
For production AI systems, the practical question is not whether your team can build middleware. Most capable platform teams can. It is whether owning that middleware remains the best use of engineering time once it must route requests, enforce policy, preserve audit evidence, manage provider changes and give multiple teams a consistent operating model.
What each approach actually owns
Custom middleware is code your organisation writes and operates between an application and one or more model-provider APIs. It may begin as a thin adapter that normalises provider requests. Over time, it commonly grows to include retries, model selection, provider failover, logging, usage metering, content controls and tenant-specific rules.
An API gateway puts this layer behind a shared endpoint. Applications make their normal model calls to one governed doorway, while the gateway applies centrally managed routing and policy before forwarding a request to an appropriate provider. The application retains its product logic. The control plane owns the cross-cutting concerns.
That distinction matters because LLM controls are rarely isolated. A routing decision can affect cost, availability, data residency and output quality. A logging decision affects debugging, privacy and retention obligations. When these decisions are implemented independently in several services, consistent enforcement becomes harder with every new application and provider.
When custom middleware is the right answer
Custom middleware remains a sensible choice in a limited set of circumstances. A team running one low-volume internal use case, with one provider and no material policy or audit requirements, may reasonably prefer a small wrapper. It can be shaped precisely around a narrow workflow and may introduce no new operational dependency.
It can also be appropriate where model calls are inseparable from specialised domain logic. For example, an application may need to construct a highly specific retrieval process, perform deterministic validation or co-ordinate a bespoke multi-step workflow. Those decisions belong in the application or service layer, regardless of whether a gateway is used.
The boundary is important. Domain logic should remain close to the product. Shared infrastructure controls should not need to be rebuilt in every product. Problems start when a local wrapper becomes the place where teams copy provider credentials, reimplement safety checks and maintain separate rules for spend or sensitive data.
A custom solution also creates a long-term ownership commitment. Someone must track provider API changes, test failure modes, measure routing outcomes, patch vulnerabilities, maintain dashboards and answer audit questions. The initial code may be modest; the operational surface area is not.
API gateway versus custom middleware for production control
The strongest case for an API gateway appears when more than one of these pressures exists: several LLM applications, multiple providers, regulated data, meaningful token expenditure or a need for central security oversight.
A gateway gives every application the same enforcement point. Rather than hoping each team has adopted the current version of a shared package, platform and security teams can apply controls at request level. Scoped API keys, named routes, routing rules and priorities can be managed without requiring each application team to alter business logic.
The result is not that applications become less flexible. It is that flexibility is governed. A product team can use a logical route appropriate to its workload while the platform team determines the eligible providers, fallback order and policy boundaries behind it. Changes to those controls do not have to become coordinated application deployments.
For teams that already use an OpenAI-compatible client, adoption can be deliberately small: update the API key and base URL, then use model="auto" or a plain logical route name. The request shape and product logic remain byte-for-byte identical. This is particularly useful when governance work has been delayed because a broad application rewrite is difficult to justify.
The hidden cost is consistency, not only code
A build-versus-buy comparison can become misleading if it focuses only on engineering days needed to create a proxy. Routing requests is straightforward. Maintaining reliable, explainable behaviour across production traffic is the real work.
Consider a provider incident. A local middleware implementation needs clear retry behaviour, carefully tested failover logic and observability that shows what happened for each request. It must avoid obscuring the original error while protecting the application from unnecessary disruption. The same work repeats as providers add models, change limits or exhibit different performance characteristics.
Consider sensitive data. A team may add a regular expression to one service, while another logs full prompts for debugging. Neither decision is necessarily malicious or careless, but the result is uneven protection. A central gateway can perform DLP-based checks with explicit actions: log, warn, redact using [REDACTED], or block. Metadata-only logging by default, with payload retention as an explicit opt-in, creates a clearer baseline for privacy-sensitive workloads.
The same pattern applies to spending. An individual service can record token use, but organisation-wide controls need a consistent source of truth. Hard spend caps and budgets work best when enforcement is central. A cap should alert or block with a 429 response when reached, rather than silently changing model behaviour and leaving the caller uncertain about what happened.
Policy order should be visible and predictable
Policy enforcement is not a collection of unrelated filters. Order changes outcomes. A production control plane should make that sequence explicit: prompt shield first, then DLP, then routing, then output moderation.
Prompt-injection and jailbreak shielding runs before the request is sent onwards. DLP then evaluates sensitive content under the configured action. Only eligible requests proceed to routing, where the gateway can select the most cost-effective capable model according to defined rules and priorities. Output moderation is applied to the resulting response.
This sequence avoids a common middleware failure: treating provider choice as the primary decision, then attempting to attach security controls around it later. For organisations handling customer data or regulated workflows, the policy decision must come before the provider decision.
Per-request traces make this operationally useful rather than merely architectural. Teams need to see which route was used, which provider served the request, what policy actions occurred and how the request performed. That evidence supports incident analysis, FinOps review and the audit trails expected for EU AI Act record-keeping without making prompt payload retention the default.
A practical decision framework
Choose custom middleware when the scope is genuinely narrow, the traffic is low-risk, and your team is willing to own the entire lifecycle. Keep it deliberately small. Do not let it become an undocumented platform through gradual additions.
Choose an API gateway when AI usage is becoming shared infrastructure. The tipping point is often earlier than expected: a second provider, a second production team, a customer-data use case or a need to explain spend can be enough. At that point, centralisation reduces duplicated effort and makes policy enforceable rather than advisory.
For many organisations, the best architecture is not one or the other in absolute terms. Keep workflow-specific logic in your services, but move provider selection, security controls, data handling rules and financial guardrails into the governed doorway. This gives product teams room to build while allowing platform teams to set non-negotiable boundaries.
routeur.ai follows this model as a governed control plane for LLM traffic, combining intelligent multi-provider routing, automatic failover, DLP, output moderation, hard spend caps and per-request traces behind an OpenAI-compatible gateway. For UK and EU teams, EU data residency and GDPR processor terms can also remove friction from the governance review that often delays wider adoption.
The useful test is simple: if a control must work the same way for every application, it should not depend on every application team remembering to implement it. Put that control at the doorway, keep product logic where it belongs, and let each new AI use case start from a safer operational baseline.