A provider outage is not the only moment that tests an LLM application. Elevated latency, exhausted capacity, regional disruption and a model-specific error can all turn a single-provider dependency into a customer-facing incident. Provider failover policies determine what happens next: which alternative is permitted, when traffic may move, and which controls must remain in force while it does.
For production teams, the objective is not simply to get any answer from any model. A useful response delivered through an unsuitable provider, outside an approved data boundary or at an unplanned cost can create a different operational problem. Failover must therefore be treated as a governed routing decision, not an emergency shortcut.
What provider failover policies should decide
A failover policy is the set of rules that preserves service intent when the preferred provider or model cannot serve a request. It should distinguish between a temporary provider failure and a request that needs a different capability. Those are related routing problems, but they should not be collapsed into one vague rule such as “use the next available model”.
The policy starts with a route. A named route can represent a production workload with a clear purpose: a customer support assistant, document extraction process or internal coding copilot. Routing rules and priorities then identify the preferred model and the acceptable alternatives for that workload.
Those alternatives need to be deliberately compatible. A fallback for a short, general text classification task may be broad. A fallback for a workflow that depends on structured output quality, multilingual performance, a particular context requirement or strict latency expectations will be narrower. The right question is not whether another model exists. It is whether it can carry this request without materially changing the product behaviour that users and downstream systems rely on.
A sound policy normally addresses four decisions:
- the conditions that justify failover, such as provider unavailability or unacceptable response latency;
- the ordered providers or models that may receive the request;
- the capability, residency, security and cost constraints that every fallback must satisfy; and
- the action when no approved fallback remains, including a controlled error rather than an ungoverned provider switch.
The final decision is often overlooked. Failing visibly can be preferable to silently changing the quality, location or handling of a sensitive workload. Availability matters, but predictable failure is easier to operate, explain and remediate than unrecorded policy drift.
Failover must preserve the request control path
Failover should not bypass the controls applied to the original request. The request path matters because the conditions of a provider incident are exactly when teams are most tempted to introduce exceptions.
At routeur.ai, one governed doorway keeps the sequence consistent: prompt shield first, then DLP, then routing, then output moderation. A request that contains a prompt-injection attempt should not be rerouted in an attempt to obtain a response. Sensitive data must receive its configured DLP action before a provider is selected. If a policy calls for redaction, the value is replaced with [REDACTED]; if it calls for blocking, the request does not become eligible for a fallback simply because the preferred provider is unavailable.
This order has practical consequences. A fallback provider is not a separate security posture. It receives only requests that have passed the same request-level controls, under the same route-level intent. Output moderation also remains relevant after the chosen provider responds. Otherwise, failover can improve apparent uptime while creating a blind spot in the content returned to users.
The same discipline applies to logging. Metadata-only logging by default limits unnecessary payload retention, while per-request traces provide the operational evidence needed to investigate a routing decision. Where payload retention is required for a defined purpose, it should be an explicit opt-in rather than an accidental consequence of incident handling.
Build policies around workload classes, not providers
Many teams begin with a spreadsheet of provider accounts and model names. That is useful inventory, but it is a poor policy boundary. Build failover around workload classes instead.
A low-risk internal summarisation route can usually tolerate more variation in model choice than a regulated customer-facing workflow. An extraction route may value consistency above low cost. A real-time assistant may favour latency and availability, while an overnight evaluation job can wait for its preferred model or use a lower-cost alternative. Each workload should state which trade-offs are acceptable before an incident forces a decision.
This approach also avoids a common error: treating price as the primary failover rule. The least expensive alternative might be capable enough, but cost optimisation is only valid within the constraints of the route. A fallback that produces more retries, more manual review or weaker task performance may cost more in practice.
Hard spend caps should remain independent guardrails. A cap can alert or block requests with a 429 when the limit is reached; it should not quietly push traffic to another provider as if budget exhaustion were a technical outage. Keeping availability policy separate from financial controls makes both easier to audit.
Define acceptable degradation explicitly
Not every fallback has to be identical to the primary model, but the differences must be intentional. Define whether the route may accept slower responses, shorter context, a different quality band or reduced throughput. For user-facing products, decide how that change will be presented in the product experience. For automated processes, decide whether the task can proceed, should enter a review queue or should stop.
The point is not to eliminate degradation. It is to make degradation bounded, testable and visible. A policy that says “best available” offers neither engineers nor compliance teams enough evidence to judge what occurred.
Test the failure path before it is needed
Failover policies are configuration, but they need the same engineering discipline as application code. Test them against representative prompts, sensitive-data cases and expected output checks. Confirm that a fallback receives only an eligible request, that blocked requests stay blocked, and that traces identify the selected route, provider and model outcome.
Routeur-Trace can support request-level investigation when diagnosing a particular transaction. Routeur-Dry-Run is useful when teams need to inspect how a routing decision would be evaluated without treating the exercise as normal production traffic. These controls are particularly valuable before changing priorities for a high-volume route.
Tests should include more than a simulated provider outage. Exercise partial failures: elevated latency, intermittent errors, capacity pressure and a primary provider returning responses that do not meet the route’s operational expectations. Also test recovery. When the preferred provider becomes healthy again, an immediate return may create instability if health is fluctuating. The right recovery behaviour depends on the workload and should be reviewed as part of the policy.
Make ownership and evidence operational
A policy without an owner becomes stale quickly. Engineering may own route design and service behaviour; security may define DLP and output controls; FinOps may own budgets; compliance may need evidence of provider choices and data handling. The operating model should identify who can change each part, who approves exceptions and how quickly routes are reviewed after an incident.
Scoped API keys help contain access to the routes and environments a team is authorised to use. That matters during an outage, when broad credentials and improvised changes create avoidable risk. The safer pattern is to prepare approved routes and priorities in advance, then make controlled changes through established ownership.
Per-request traces supply a factual record: what route was used, how the request was handled and where it was sent. Combined with defined policies, this turns an availability event into an explainable operational record rather than a guess reconstructed from scattered application logs. For organisations maintaining EU AI Act record-keeping, that distinction is material.
Treat failover as a product decision
The best provider failover policies are not invisible plumbing. They express what the business is willing to trade when a dependency degrades: speed, cost, model preference or, when necessary, temporary availability. Put those choices into named routes, enforce them at one governed doorway and test them under realistic pressure. Then the next provider incident becomes a controlled service decision, not a hurried rewrite of your application’s rules.