A production incident should not force a choice between operational visibility and exposing customer data. Yet many LLM teams still log complete prompts and responses because conventional application logging treats the request body as the easiest debugging artefact. To implement metadata-only logging is to make a different engineering decision: retain the evidence needed to operate the system, while keeping prompt and output payloads out of routine logs.
For regulated teams, that distinction affects more than storage costs. Prompts can contain customer records, commercial terms, source code, health information or credentials pasted in error. Outputs can reproduce that material, introduce new sensitive content, or reveal how an internal workflow behaves. If those payloads enter observability tools by default, every retention rule, access review and incident response process becomes harder.
What metadata-only logging keeps - and excludes
Metadata-only logging records the context of an LLM request without recording its content. The useful record normally includes a timestamp, request identifier, selected provider and model, route, latency, status, token usage, estimated cost and policy outcomes. It supports questions such as: Which route is failing? Which team is consuming the budget? Did a provider fail over? Are requests being blocked more often after a policy change?
The payload is deliberately absent. That means no prompt text, no retrieved document excerpts, no tool arguments and no generated answer in the standard request trace. This is not the same as hiding fields in a dashboard. The objective is to avoid retaining the content in the logging path in the first place.
The boundary matters. A request ID may be operationally necessary, but it can still become personal data if another system can use it to identify an individual. A hashed email address may remove the plain-text value but remain linkable across requests. Treat metadata design as data-protection design, not merely as a smaller schema.
Design the logging contract before changing the gateway
Start by writing down the operational questions your team must answer without payload access. Engineering may need latency by model, error rate by provider and traces for failed requests. FinOps may need token consumption and cost allocation by route. Security may need policy-event counts and evidence that controls ran. Compliance may need retention periods, access boundaries and an audit trail of configuration changes.
This exercise prevents a common failure mode: collecting every available field because it might be useful later. Each metadata field should have an owner, a purpose, a retention period and an access rule. If a field does not support an identified operational decision, do not collect it.
Separate traceability from content retention
Per-request traces are valuable even when content is unavailable. A trace can establish that a request reached the gateway, the route selected a model, a policy action occurred and the request completed within a given duration. It can also show where the process failed without exposing the material being processed.
Content retention is a separate control. There are legitimate exceptions: a tightly scoped quality investigation, a customer-approved test case, or a short-lived reproduction of a critical defect. These cases should be explicit opt-ins with narrow access, a defined purpose and a deletion point. They should not quietly become the default because a troubleshooting process is inconvenient.
Keep identifiers proportionate
Avoid placing email addresses, account names, case numbers or free-text labels into route names and log annotations. Prefer internal identifiers that are meaningful only within a controlled system, and review whether those identifiers can be joined with other records. The best metadata model is not the most detailed one. It is the least detailed model that still allows the team to operate safely.
Implement metadata-only logging at the governed doorway
An API gateway is the practical enforcement point because every model request crosses the same boundary. routeur.ai operates as one governed doorway between an application and its LLM providers, so policy enforcement, routing and tracing do not need to be rebuilt separately for each provider integration.
For an OpenAI-compatible application, the application change can remain limited to the API key and base URL. Business logic stays byte-for-byte identical:
import os from openai import OpenAI
client = OpenAI( api_key=os.environ["ROUTEUR_KEY"], base_url="https://api.routeur.ai/v1" )
response = client.chat.completions.create( model="auto", messages=[ {"role": "user", "content": "Summarise this support request."} ] )The logging decision must then be enforced in the control plane, rather than relying on every application team to remember not to log request bodies. Metadata-only logging by default makes payload retention an explicit operational decision instead of an accidental side effect of enabling traces.
The request path should also be understood clearly. Prompt shielding runs first, followed by DLP controls, then routing, then output moderation. If DLP is configured to redact, sensitive values are replaced with [REDACTED]; other actions can log, warn or block according to policy. This order matters because routing should not become a way around input controls, and outputs should still be assessed before they return to the application.
Retain enough evidence to run the service
Metadata-only logging is not an argument for sparse observability. It is an argument for observability that is fit for purpose. A useful operational view can show request volume, latency percentiles, provider availability, model selection, token use, cost trends, policy events and blocked-request rates. Teams can compare routes, identify unexpected spend and detect provider degradation without reading a single prompt.
This is particularly useful where multiple teams share model access. A central view can expose whether one workflow is driving a sudden increase in spend, while hard spend caps provide a separate boundary. A cap should alert or block with a 429 when reached; it should not be treated as a mechanism that silently reroutes traffic to another model.
Audit evidence also improves when it is designed around events rather than content. Record the fact that a DLP policy acted, that output moderation blocked a response, or that a routing rule selected a provider. Pair that with scoped access, named routes and clear retention rules. The result is evidence that can support internal review without creating a second, uncontrolled repository of sensitive payloads.
Plan for the trade-offs
The trade-off is real: without stored prompts and outputs, engineers cannot replay every historical request from a log record. Quality investigations require a more deliberate process. A customer may need to supply a safe reproduction, a team may create a sanitised test fixture, or an authorised investigator may enable temporary payload retention for a defined case.
That friction is useful when it forces a decision about necessity. The alternative is permanent broad access to content that most people do not need and should not see. For a low-risk internal prototype, a short payload-retention window may be proportionate. For a customer-facing assistant handling sensitive data, metadata-only logging should usually be the baseline.
Do not overlook the systems around the gateway. Application frameworks, error trackers, reverse proxies, browser telemetry and support tooling can all capture request bodies independently. A metadata-only setting in one component does not protect a payload copied elsewhere. Review the complete path from client to provider, including failed requests and debug modes.
Test the policy, not just the happy path
Before production rollout, send controlled test prompts containing representative sensitive values and prompt-injection attempts. Confirm that traces retain the expected metadata, that payload fields are absent, and that DLP behaviour matches the configured action. If redaction is enabled, verify that logs and downstream handling show [REDACTED], not the original value.
Test provider failures, policy blocks and spend-cap responses as well. These are the cases most likely to trigger verbose emergency logging. The implementation is only complete when the system remains observable under failure without switching to content capture.
A good metadata-only design gives teams a disciplined answer when someone asks for full prompts “just in case”. Keep the trace, identify the operational question, and collect content only when a named investigation genuinely requires it.