Observability Exporters

Observability exporters send usage events from the gateway to destinations used for tracing, logging, storage, or accounting. This guide configures an OTLP/HTTP exporter through the control-plane path for AISIX Enterprise and AISIX Cloud or the resources-file path for AISIX OSS. It then explains destination choices, content capture, and delivery behavior.

Prerequisites

Before starting, prepare the following:

• One of these configuration paths:
  • AISIX Enterprise or AISIX Cloud with an environment, an attached gateway, and a write-scoped admin token. For AISIX Enterprise, follow the AISIX Enterprise getting-started guide. To request AISIX Cloud access, contact API7.
  • An open-source AISIX gateway that loads a declarative resources.yaml file.
• A telemetry destination for the exporter you plan to use.
• A working model alias and caller API key for delivery verification.

Export the gateway and request values used for verification:

# AISIX_PROXY has no trailing slash or endpoint path
# The local quickstarts use http://127.0.0.1:3000
export AISIX_PROXY="YOUR_AISIX_GATEWAY_URL"
export AISIX_API_KEY="YOUR_CALLER_API_KEY"
export MODEL_ALIAS="YOUR_MODEL_ALIAS"

For the control-plane examples, export the Admin API base URL, admin token, and environment ID:

# AISIX_CP includes /api and has no trailing slash
# The local AISIX Enterprise quickstart uses http://localhost:8080/api
export AISIX_CP="YOUR_AISIX_CONTROL_PLANE_API_URL"
export AISIX_TOKEN="YOUR_ADMIN_TOKEN"
export ENV_ID="YOUR_ENVIRONMENT_ID"

In other deployments, use the reachable AISIX Cloud Admin API URL.

Choose an Exporter Kind

All exporter kinds are available through both configuration paths. AISIX Cloud Admin API requests and resource-file entries select a kind with the kind value, and the dashboard shows a label for the same underlying kind:

kind Value	Dashboard Label	Use When
otlp_http	OTLP/HTTP	You already collect traces through an OTLP/HTTP collector or vendor endpoint.
object_store	Object storage	You want batched NDJSON request events in Amazon S3, S3-compatible storage, Google Cloud Storage, or Azure Blob.
datadog	Datadog	You use Datadog Logs HTTP intake.
aliyun_sls	Alibaba Cloud SLS	You use Alibaba Cloud Simple Log Service as the log destination.

The gateway sends telemetry directly to the selected destination.

Configure an OTLP Exporter

Set the collector endpoint and authorization header used by the example:

# Replace with your values
export OTLP_ENDPOINT="https://collector.example.com/v1/traces"
export OTLP_AUTH_HEADER="Bearer YOUR_COLLECTOR_TOKEN"

The endpoint must be reachable from the gateway process. It can be an https:// or http:// URL with any host, such as an in-cluster collector at http://otel-collector-dev:4318/v1/traces. A receiver running beside a gateway process on the same host can use http://localhost:4318/v1/traces; a containerized gateway needs the receiver's container-network or host address instead. Remove the headers block from the exporter when the receiver does not require authentication.

With http://, the static headers and the exported telemetry cross the network in plaintext. The headers can carry an authorization token and the telemetry can include prompt and response content, so use http:// on networks you trust.

Gateways 1.7.0 and earlier accept an operator-configured http:// endpoint only when its host is localhost, 127.0.0.1, or otel-collector. While such a gateway is registered in the environment, the AISIX Cloud Admin API refuses to save an http:// endpoint on any other host with 422 DP_INCOMPATIBLE. Upgrade the environment's gateways to save it.

AISIX Enterprise or AISIX Cloud

Create the exporter and capture its ID:

EXPORTER_ID=$(curl -sS -X POST "$AISIX_CP/environments/$ENV_ID/observability_exporters" \
  -H "Authorization: Bearer $AISIX_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "prod-otlp",
    "kind": "otlp_http",
    "endpoint": "'"${OTLP_ENDPOINT}"'",
    "headers": {
      "Authorization": "'"${OTLP_AUTH_HEADER}"'"
    },
    "sample_rate": 1
  }' | jq -r '.observability_exporter.id')

❶ Set static headers only when the OTLP destination requires them. Header values are encrypted at rest and never returned by read operations.

❷ sample_rate: 1 keeps the delivery check deterministic. It is equivalent to omitting the field, which exports every request trace. After verification, lower the value when you need to reduce span volume.

Retrieve the exporter to confirm the stored configuration:

curl -sS "$AISIX_CP/environments/$ENV_ID/observability_exporters/$EXPORTER_ID" \
  -H "Authorization: Bearer $AISIX_TOKEN"

You should see a response similar to the following. Reads expose the configured header names through header_keys and confirm stored values through headers_set; the header values themselves are never returned.

{
  "observability_exporter": {
    "id": "b46a9f5d-6a4d-4bb1-ae2c-4ab1b22f5e80",
    "env_id": "0f2f6a1e-9d33-4a8f-9a6e-2a7b6a9c1d2e",
    "name": "prod-otlp",
    "enabled": true,
    "kind": "otlp_http",
    "endpoint": "https://collector.example.com/v1/traces",
    "header_keys": ["Authorization"],
    "headers_set": true,
    "sample_rate": 1,
    "created_at": "2026-07-22T08:30:00Z",
    "updated_at": "2026-07-22T08:30:00Z"
  }
}

Keep $EXPORTER_ID for later updates or deletion. Exporters are enabled by default. Set enabled: false to save the resource without sending telemetry. Disabling or deleting an exporter stops its delivery pipeline on each gateway once the configuration change arrives: events already queued are delivered first, and the exporter drops out of the gateway's delivery health report. Re-enabling it starts a fresh pipeline with its counters reset.

AISIX OSS

Add this exporter to observability_exporters in the complete resources file. Environment interpolation keeps the collector token out of the file:

resources.yaml (OTLP exporter)

observability_exporters:
  - name: prod-otlp
    kind: otlp_http
    endpoint: ${OTLP_ENDPOINT}
    headers:
      Authorization: ${OTLP_AUTH_HEADER}
    sample_rate: 1

Validate the assembled complete file:

aisix validate --resources resources.yaml

Start or restart the gateway with OTLP_ENDPOINT and OTLP_AUTH_HEADER in its process environment. Reload a running gateway only if those variables are already available to the process. Exporters are enabled by default; add enabled: false to keep the entry without sending telemetry.

Verify OTLP Delivery

A saved exporter confirms only that AISIX accepted its configuration. Verify the OTLP exporter configured above by capturing the request ID that AISIX returns, finding that ID at the destination, and checking the gateway's delivery signal. If you are checking an existing OTLP exporter with sample_rate below 1, temporarily set the rate to 1 so this request cannot be sampled out.

Send one successful request through a working model alias and capture the ID AISIX returns. This remains accurate whether AISIX accepts caller-supplied IDs or generates its own:

export EXPORTER_NAME="prod-otlp"

if TEST_REQUEST_ID=$(
  set -o pipefail
  curl -fsS -D - -o /dev/null \
    -X POST "$AISIX_PROXY/v1/chat/completions" \
    -H "Authorization: Bearer $AISIX_API_KEY" \
    -H "Content-Type: application/json" \
    -d '{
      "model": "'"${MODEL_ALIAS}"'",
      "messages": [{"role": "user", "content": "Reply with: exporter check"}]
    }' \
    | awk 'tolower($1) == "x-aisix-request-id:" { id = $2; sub(/\r$/, "", id) } END { print id }'
) && test -n "$TEST_REQUEST_ID"; then
  export TEST_REQUEST_ID
  printf 'request ID: %s\n' "$TEST_REQUEST_ID"
else
  unset TEST_REQUEST_ID
  printf 'verification request failed or returned no request ID\n' >&2
  false
fi

Wait for the exporter to flush the batch, then search your OTLP backend for a span whose aisix.request_id attribute equals $TEST_REQUEST_ID. Finding the span confirms end-to-end delivery: the gateway loaded the exporter, authenticated to the destination when required, and the destination accepted the batch.

For AISIX Enterprise or AISIX Cloud, the exporter row in the environment's Observability view reports whether delivery is healthy and how many batches online gateways have shipped. You can inspect the same per-gateway heartbeat data through the Admin API:

curl -sS "$AISIX_CP/environments/$ENV_ID/dp_nodes" \
  -H "Authorization: Bearer $AISIX_TOKEN" \
  | jq --arg exporter "$EXPORTER_NAME" \
    '[.data[] | select(.status != "offline") | .exporter_health[]? | select(.name == $exporter)]'

After the next heartbeat, at least one online gateway should report delivered_batches greater than zero and last_error: null. The counters reset when a gateway restarts or the exporter configuration changes. A non-null last_error means delivery is currently failing. The historical fields failed_batches and last_failure_unix remain populated after a later success clears last_error.

For an open-source AISIX gateway, confirm success at the destination. If the record does not arrive, check gateway logs for sink delivery failed; retrying or sink delivery dropped.

Understand Delivery Behavior

Every exporter runs its own delivery pipeline: an in-memory queue in front of a worker that batches records and ships them to the destination. The behavior below is fixed and has no configuration.

Records buffer until the batch reaches 100 records or one second passes, whichever comes first.

A destination that refuses a batch transiently — a connection failure, a 429, or a 5xx — is retried on a time budget rather than a fixed number of attempts. The delay starts at 200 milliseconds and doubles up to a 30-second ceiling, and retries continue until 300 seconds after that batch's first attempt. When the budget is spent, the batch is dropped and counted as retries_exhausted in aisix_otlp_fanout_drops_total. A Retry-After header on a 429 or 503 replaces the computed delay, capped at the same 30 seconds, so a destination asking for a longer pause is honored rather than hammered. A budget rather than an attempt count is what lets an exporter survive a receiver outage of a given length: an outage shorter than the budget loses nothing.

While a batch is being retried, new records keep arriving. Each exporter queue holds 8,192 records; when it is full the newest record is refused and counted as queue_full, so the records already queued still have their chance at the destination.

caution

An exporter with full content capture enabled owns its records, prompt and response text included. With content_max_bytes at its default of 131,072, a full 8,192-record queue behind a stalled destination can hold hundreds of megabytes of resident memory. Size content_max_bytes against the gateway's memory limit when you enable full capture on a destination that can stall. See Configure Content Capture.

Interpret OTLP Traces

AISIX exports one connected trace hierarchy per sampled request. See OTLP Trace Reference for span parentage, W3C trace continuation, names, status, semantic-convention attributes, captured content, and resource fields.

Configure Other Exporters

Use another exporter kind when telemetry should go to object storage or a logging service instead of an OTLP trace backend. With AISIX Enterprise or AISIX Cloud, send one of the following objects as the body of POST $AISIX_CP/environments/$ENV_ID/observability_exporters.

Object Storage

For object storage, choose the storage provider, bucket, and key prefix. The default authentication mode uses a credential reference resolved by the gateway:

{
  "name": "request-events-s3",
  "kind": "object_store",
  "provider": "s3",
  "bucket": "acme-aisix-events",
  "prefix": "ai-gateway",
  "region": "us-east-1",
  "credential_ref": "acme_s3"
}

Object storage supports Amazon S3, Google Cloud Storage, Azure Blob, and S3-compatible targets. Cloud identity is supported only for S3 or GCS when the gateway has an attached identity that can write to the bucket. Use credential_ref for Azure Blob and for S3-compatible targets that require static credentials.

For an S3-compatible target such as MinIO, Cloudflare R2, or Alibaba Cloud OSS, set the target endpoint explicitly. Without an endpoint, the exporter uses the native AWS S3 endpoint.

endpoint overrides the storage provider's native endpoint, and its meaning follows provider:

provider	What endpoint sets
s3	The S3-compatible host to upload to.
gcs	The Cloud Storage XML API base URL, such as a private endpoint. Uploads use the XML API, so the endpoint must serve it; an emulator that accepts only JSON API uploads, such as the stock fake-gcs-server image, cannot receive objects. It takes precedence over a gcs_base_url in the service-account JSON.
azure_blob	The Blob service endpoint.

The value can be an https:// or http:// URL with any host, such as an in-cluster MinIO at http://minio.storage.svc:9000. With http://, the uploaded request events cross the network in plaintext, so use it on networks you trust. Gateways 1.7.0 and earlier accept http:// only for localhost, 127.0.0.1, and a few emulator host names. While such a gateway is registered in the environment, the AISIX Cloud Admin API refuses to save an http:// endpoint on any other host with 422 DP_INCOMPATIBLE. Upgrade the environment's gateways to save it.

endpoint cannot be combined with cloud identity: the AISIX Cloud Admin API rejects the combination, and an open-source AISIX gateway that loads it never uploads and reports a permanent delivery error. The dashboard offers the field for S3-compatible targets, Cloud Storage, and Azure Blob, and hides it when cloud identity is selected.

A gcs endpoint needs gateways at 1.5.0 or later, because earlier gateways ignore it and upload to Google's public Cloud Storage endpoint instead. Every gateway release that AISIX Enterprise and AISIX Cloud support, 1.6.0 or later, honors it. A connected gateway older than 1.6.0 is not checked when you save; upgrade it before relying on the endpoint. See Mixed-Version Behavior.

Alibaba Cloud SLS

Configure the endpoint host, project, log store, and credential reference:

{
  "name": "request-events-sls",
  "kind": "aliyun_sls",
  "endpoint": "ap-southeast-3.log.aliyuncs.com",
  "project": "acme-observability",
  "logstore": "ai-gateway",
  "credential_ref": "acme_sls"
}

Datadog

Configure the Datadog site, service name, tags, and credential reference:

{
  "name": "request-events-datadog",
  "kind": "datadog",
  "site": "datadoghq.com",
  "service": "ai-gateway",
  "tags": ["team:platform", "tier:prod"],
  "credential_ref": "acme_datadog"
}

For the resources-file path, add these entries to observability_exporters:

resources.yaml (request exporters)

observability_exporters:
  - name: request-events-s3
    kind: object_store
    provider: s3
    bucket: acme-aisix-events
    prefix: ai-gateway
    region: us-east-1
    credential_ref: acme_s3

  - name: request-events-sls
    kind: aliyun_sls
    endpoint: ap-southeast-3.log.aliyuncs.com
    project: acme-observability
    logstore: ai-gateway
    credential_ref: acme_sls

  - name: request-events-datadog
    kind: datadog
    site: datadoghq.com
    service: ai-gateway
    tags: ["team:platform", "tier:prod"]
    credential_ref: acme_datadog

Keep only the destinations you use, then validate the resources file and start or reload the gateway as described for the OTLP example. Verify these exporters with the same request-ID approach used for OTLP: search object-storage and SLS records by request_id, or Datadog logs by aisix.request_id. The Snowflake guide shows how to inspect object-storage output directly.

SLS, Datadog, and object-storage exporters keep destination credentials out of the exporter resource by using credential references or cloud identity. The gateway resolves those credentials locally when it sends telemetry.

Configure Content Capture

Exporters include request status, token counts, model and provider identifiers, request IDs, finish reason, and timing by default. Prompt and response bodies remain excluded unless full content capture is enabled on an OTLP/HTTP, SLS, or Datadog exporter.

Failure metadata needs separate handling. Every failed event carries an error_message of up to 2,048 characters, which can include a rejected request value or error detail returned by an upstream. content_mode: metadata_only does not remove that field. Approve the destination to receive failure details even when you do not enable full content capture.

Enable Full Content Capture

Add content_mode and content_max_bytes when creating an exporter. In AISIX Enterprise and AISIX Cloud, patch an existing exporter with the same fields:

curl -sS -X PATCH "$AISIX_CP/environments/$ENV_ID/observability_exporters/$EXPORTER_ID" \
  -H "Authorization: Bearer $AISIX_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "content_mode": "full",
    "content_max_bytes": 131072
  }'

In the dashboard, set Content mode to Full, then adjust Max content bytes in the exporter form.

For the resources-file path, add the fields to the exporter entry, then validate and reload the file:

observability_exporters:
  - name: prod-otlp
    kind: otlp_http
    endpoint: ${OTLP_ENDPOINT}
    headers:
      Authorization: ${OTLP_AUTH_HEADER}
    content_mode: full
    content_max_bytes: 131072

Use full content capture only when the destination is approved to receive end-user prompt and response text. AISIX applies the configured byte cap to each captured content field independently.

Understand Truncated Records

When valid JSON exceeds content_max_bytes, AISIX first tries to reduce the value structurally so the exported field remains valid JSON. If the reduced value still cannot fit the configured cap, AISIX falls back to a UTF-8-safe byte cut. The captured prompt is the serialized request body and follows the same behavior, after AISIX replaces its base64 media.

Content	Truncation Behavior
Long string	Keeps a prefix and adds the inline marker ...[aisix: truncated, N bytes total].
Base64 data URI	Replaces the encoded data with a size placeholder.
Long array	Keeps head and tail samples around an {"_aisix_truncated": true, "omitted_items": N} element that accounts for every omitted item.
Non-JSON content, or JSON that cannot fit after structural reduction	Cuts the value at a UTF-8 character boundary.

When truncation occurs, OTLP records include aisix.content_truncated: true. Datadog and SLS records include content_truncated: true.

Base64 Media in Captured Content

With content_mode: full, AISIX replaces base64 media in a captured request with the placeholder [aisix: base64 data uri omitted, N bytes], whatever its size and before applying content_max_bytes. The same rule covers every captured request type, including embeddings, rerank, completions, decisions, and MCP tools/call arguments, and every content-bearing destination: OTLP, Alibaba Cloud SLS, and Datadog. The object_store kind carries no content. Only the exported copy changes. The upstream request and the client's response are not affected.

AISIX recognizes base64 media by its form or by the part that declares it:

• A string that is a base64 data URI, in any field. This covers image, audio, and PDF parts. The URI keeps its head, as in data:image/png;base64,[aisix: base64 data uri omitted, N bytes], and N is the length of the whole URI.
• source.data of a base64 image or document block, including one inside a tool result.
• input_audio.data, file_data on an input_file part or a file object, and the result of a replayed image_generation_call.
• inlineData.data of Gemini bodies and source.bytes of Bedrock Converse bodies, which appear on passthrough routes.
• data of an MCP image or audio block beside its mimeType, and resource.blob of an embedded resource block.

For a bare field like these, N is the length of the payload.

The structured fields (gen_ai.input.messages on OTLP and Datadog, input_messages on SLS) keep the media part and any media type available, so a record still shows that the request carried media:

{"type":"blob","modality":"image","mime_type":"image/png","content":"[aisix: base64 data uri omitted, 1368 bytes]"}

An http or https URL stays as sent. AISIX does not replace a data URI embedded in other text, a JSON string of tool arguments, or bare base64 in a field not listed above. Most content that does not parse as JSON, such as A2A message text or a stream frame cut partway through a payload, has no fields to read, so AISIX cuts it at the first media marker it shows, such as ;base64,, and writes the placeholder there.

On the response side, the replacement covers three cases. On /v1/images/generations and /v1/images/edits, b64_json and any data URI become the placeholder, and url, revised_prompt, and other fields stay. On passthrough routes, the same recognition runs over the upstream's own body and raw stream frames, where it also covers partial_image_b64 and bytesBase64Encoded, while a model's extracted answer text is never rewritten. On MCP tools/call, the captured result gets the placeholder, as described below. Other captured responses keep their base64 unless truncation replaces it, including embeddings.

In an MCP tools/call result, including one that reports isError, AISIX replaces data of image and audio blocks, resource.blob, and any string that starts as a base64 data URI, wherever it sits, including in structuredContent. The block, its mimeType, and the resource uri stay. AISIX does not replace base64 embedded in a text block or in resource.text, or bare base64 under a structuredContent key it does not recognize. A common case is a tool that follows the MCP specification and also returns its structuredContent serialized into a text block: the payload in that text block is still exported.

A gateway at 1.8.0 or earlier can export base64 media as sent, and captures the image endpoints' b64_json.

On passthrough routes, a response that is not valid UTF-8, nests deeper than 128 levels, or holds a number the parser cannot read, such as 1e999, can reach the export with its base64 intact, or cut at the first media marker, depending on the route's protocol. A shape with no marker, such as an MCP blob, is exported as is in that case.

Captured Content in Datadog and SLS Records

With content_mode: full, the Datadog and Alibaba Cloud SLS exporters report a message exchange with the same structured content as the OTLP exporter's Captured Content attributes: the same messages, the same text, reasoning, and tool_call output parts, and the same finish_reason values. Each sink uses its own field names:

Content	Alibaba Cloud SLS	Datadog
Request messages	input_messages	gen_ai.input.messages
Anthropic system or Responses instructions	system_instructions	gen_ai.system_instructions
Declared tools	tool_definitions	gen_ai.tool.definitions
Assistant output message	output_messages	gen_ai.output.messages
Prompt side that is not structured	prompt	gen_ai.prompt
Response side that is not structured	response	gen_ai.completion

SLS writes each structured value as a compact JSON string. Datadog writes it as nested JSON. A structured field is written only when the record has that content, so system_instructions appears only for an Anthropic Messages system field or a Responses instructions field. A chat completions system message stays in input_messages.

The prompt and response sides are judged separately, by the same rule as OTLP. A side stays in the flat field when capture truncated it, when it does not convert, or when a structured value would exceed content_max_bytes. Content that does not convert includes payloads that are not a message exchange, such as embeddings input, images, audio, jobs, and agent text. The output message falls back the same way as on OTLP: when the captured output parts outgrow the cap, it is built from the response text, and the response stays flat only when that text was truncated too. One record can therefore carry structured input messages beside a flat response, or the other way round. MCP tool calls keep their arguments in the flat prompt field and their result in the flat response field. content_truncated: true marks a record whose content was truncated, as before.

A side with neither structured content nor captured text is still written as an empty flat field. For example, a request abandoned before any response records its messages in input_messages and an empty response. OTLP omits such a side.

content_max_bytes caps each field separately, so one record can carry up to four capped content fields: the three input fields and the output message.

Changes from 1.6.0 and earlier

A gateway at 1.6.0 or earlier writes every captured prompt to SLS prompt and Datadog gen_ai.prompt, and every captured response to SLS response and Datadog gen_ai.completion. After upgrading, those fields hold only the sides that are not structured.

• Move SLS indexes, saved queries, dashboards, and alerts that read prompt or response to the structured fields. Keep reading prompt and response for truncated or unconvertible sides.
• Create SLS field indexes for input_messages, system_instructions, tool_definitions, and output_messages. Without a field index, these fields can be reached only by full-text search. Use a JSON-type index to query keys inside a value, or enable analytics on the field and read it with json_extract in SQL.
• Move Datadog facets and monitors on gen_ai.prompt or gen_ai.completion to the gen_ai.input.messages and gen_ai.output.messages attributes.

Capture Passthrough Content

For successfully relayed passthrough route traffic, full content capture records the request body as a string, subject to the exporter's content cap and JSON-aware structural truncation. Buffered responses record extracted text when the response matches a supported extraction shape and otherwise record the body as text. Streamed responses record accumulated extracted text; opaque data payloads are retained as text. Base64 media is replaced in each, as described in Base64 Media in Captured Content.

Capture Failed Requests

When any OTLP/HTTP, SLS, or Datadog exporter runs at content_mode: full, every failure row carries the request body and the response body for that row, on every endpoint and status. A failure row is one with a status of 400 or above or a non-empty error_class. That includes 401 and 403 from an upstream, a gateway refusal such as a forbidden model, and each failed attempt of a retry or fallback chain. The same request exported under metadata_only has neither prompt nor response.

The request goes where the success path puts it. On /v1/chat/completions, /v1/messages, and /v1/responses it becomes structured input messages when it converts (gen_ai.input.messages on OTLP and Datadog, input_messages on SLS), and otherwise stays in the flat prompt field. Multipart requests report their text fields plus a <name>_sha256 checksum for each file, and a /v1/files upload reports its text fields plus file_sha256. File bytes are never stored.

The response goes in the flat response field: aisix.content.response on OTLP, response on SLS, and gen_ai.completion on Datadog. content_max_bytes caps it, as it does the request. AISIX chooses the response in this order:

1. What a stream delivered before it failed part-way or the caller left.
2. The body the upstream sent for this attempt, up to the first 64 KiB.
3. On the request's terminal row, the error body the gateway sent the caller. That is the Anthropic shape on /v1/messages and /v1/messages/count_tokens, and JSON-RPC on MCP and A2A.
4. Otherwise, a body AISIX builds: {"error":{"message":<error_message>,"type":<error_class>}}.

So response is not always what the upstream said. When an upstream answers with no body, the terminal row's response is the error the gateway sent the caller, such as {"error":{"message":"upstream returned 404 Not Found","type":"upstream_error"}}. Read error_message in that case: with no error detail in the response says the upstream gave none. See Read an Upstream Error Message.

Some failures differ:

• A 499 row carries the request and no invented response, because the caller left. Anything a stream had delivered still appears.
• A response-side guardrail refusal records the refusal returned to the caller, not the blocked upstream output.
• A passthrough route does not decode an upstream's compressed error body. Its error_message reads upstream returned HTTP <status> with a compressed error body, and response holds the body AISIX builds from that message rather than compressed bytes.
• An A2A agent's error body is read only for a full-content exporter, and the gateway reads it before answering. A slow agent therefore delays the caller's answer, up to the agent's timeout.

A failure row has no request body when the route has none. That covers job routes without a body, such as get, list, delete, and cancel, passthrough requests with an empty body, and every Realtime error event. A multipart row that fails before its fields are read has none either, and neither does a passthrough rejection before the body is read, including authentication and authorization.

Some failures produce no usage event, and so no row, only an access-log line: a request body the gateway cannot parse, such as malformed JSON, and a caller authentication failure. See What the Usage Log Records for these and for the passthrough exception. Content reaches exporters only: the console does not display request bodies.

A gateway at 1.8.0 or earlier does not apply this complete request-and-response capture consistently across failure paths.

Protect Failure Content

Treat a destination with content_mode: full as a content-bearing data source, and restrict access to it accordingly. Failure rows are not always masked:

• A row that failed before guardrails ran carries the request body as sent, unmasked. Examples are a 404 for an unknown model, a 403, and an IP refusal.
• A 499 from a caller that left while a guardrail was still running carries the body unmasked on images, embeddings, rerank, completions, speech, and count_tokens.
• A row that a guardrail blocked has mask-action rules applied, but the text that triggered the block stays in the row. A mask that needs a remote guardrail service, such as AWS Bedrock, is not applied, because the request is already refused.
• Passthrough routes, job routes, and A2A cannot rewrite a request, so their rows carry the body as sent, including on a guardrail block.

Every failed attempt of a fallback chain repeats the request body, so full-content volume grows on failing traffic.

Review Captured Content by Endpoint

Endpoint Type	Captured Content
Text generation	Response text. OTLP, Datadog, and Alibaba Cloud SLS export the output as structured parts, including reasoning and tool calls, as described in Captured Content and Captured Content in Datadog and SLS Records.
MCP tools/call	params.arguments after traffic-control and input-guardrail checks, with any data masking applied; the JSON-RPC result when present, with base64 media replaced by a placeholder as described in Base64 Media in Captured Content. An output-guardrail block records the arguments and the JSON-RPC refusal, not the blocked result. Other MCP methods are not captured.
A2A	Text from request and reply message parts on message/send and message/stream. File parts, data parts, and JSON-RPC envelopes are not captured, except that a failure row for a call with no message text, such as a task read or a cancel, reports its JSON-RPC body.
Embeddings and rerank	Full response JSON.
Image generation and editing	Response JSON, with b64_json replaced by a placeholder as described in Base64 Media in Captured Content. url, revised_prompt, and other fields stay. Only the export changes: the client still receives the image.
Audio transcription	Returned transcript. Uploaded audio is not captured; its SHA-256 checksum is recorded alongside the request's text fields.
Text to speech	Binary speech response is not captured.

Model Fields in Exported Telemetry

Usage telemetry records both the model alias requested by the caller and the model that served an attempt when those values differ. This distinction matters for routed and ensemble traffic, where one caller-facing alias can resolve to several target calls.

Destinations render these values in their own telemetry format. OTLP traces use the following fields:

• aisix.requested_model contains the caller-requested alias or Model Group name.
• gen_ai.request.model contains the model name sent to the upstream for that attempt.
• gen_ai.response.model contains the concrete response model version when the provider reports one.
• aisix.model_id identifies the model resource that calls the upstream: the requested direct model, or the target a routing model or semantic router selected for that attempt. It is empty on a record written before a routing target is selected. See Which Model a Record Names.
• aisix.requested_model_id identifies the model resource the caller requested. See Correlate by the Requested Model ID.

A gateway at 1.6.0 or earlier puts the caller-requested alias in gen_ai.request.model.

Correlate by the Requested Model ID

aisix.requested_model is a name, so renaming a model splits its telemetry into two series. aisix.requested_model_id carries the ID of the model the caller requested, which a rename does not change. Group or filter by it to follow a routing model, semantic router, or ensemble model across a rename.

Its value depends on the kind of model the caller requested:

Requested model	aisix.requested_model_id	aisix.model_id
Direct or embedding model	The model's ID.	The same ID.
Wildcard model	The wildcard model's ID.	The same ID.
Routing model or semantic router	Its own ID, on every record of the request, including the failed attempts, the one that answered, and the records written before a target is selected.	The target model that attempt was dispatched to. Empty on a record written before a target is selected.
Ensemble model	The ensemble model's ID, on the record of every panel member and of the judge.	The panel member or judge model of that record. The ensemble's aggregate record and failures after the panel carry the ensemble model's ID. Empty when the request is refused before the panel runs.

When the requested name resolved to a model but the gateway selected no target, such as a request the caller cancelled before the first attempt, a request a guardrail blocked, a request a rate limit rejected with 429, or a response-cache hit, aisix.requested_model_id still carries the requested model's ID. For a routing model or semantic router, aisix.model_id is empty on those records, so a filter on the group's ID in aisix.model_id misses them; filter on aisix.requested_model_id instead.

When the name does not resolve to a configured model, the field is empty and the exporters omit it. That covers an unknown model name, the body model of a passthrough route, which the gateway does not look up as a model, and MCP and A2A calls, which name no model.

A gateway at 1.6.0 or earlier does not export aisix.requested_model_id.

Model Fields in Other Destinations

Datadog logs use aisix.requested_model for the caller-requested alias, aisix.requested_model_id for the ID of the requested model resource, gen_ai.response.model for the concrete response model version, and aisix.model_id for the model resource that calls the upstream, with the same values as the OTLP attribute.

Object storage and Alibaba Cloud SLS retain usage-event field names such as requested_model, requested_model_id, model_id, and provider_model_version. Their model_id takes the same values as aisix.model_id. Alibaba Cloud SLS omits an empty requested_model_id, and object storage writes it as an empty string.

Request Kind in Exported Telemetry

Every record also carries the kind of work the request asked for — a chat completion, an image generation, a video submission, a tool call. See Tell Request Kinds Apart for the values and what they mean.

Destinations name it differently:

• Object storage and Alibaba Cloud SLS keep the usage-event name, operation.
• Datadog maps it to aisix.operation.
• OTLP carries it as the aisix.operation span attribute, on every span of the request's hierarchy so a trace can be filtered by kind at its root.

OTLP spans also carry the OpenTelemetry gen_ai.operation.name attribute, and the two answer different questions. That one uses OpenTelemetry's own vocabulary where it has a value, which distinguishes model inference from agent and tool calls, and the aisix.operation value otherwise. It reports chat for chat completions, messages, and responses alike, so it cannot separate those three. Filter on aisix.operation when the endpoint matters.

Request and Response Headers in Exported Telemetry

Each exporter can record the values of selected inbound request headers and of selected upstream response headers. Every exporter kind accepts two optional lists of header names:

• request_headers: inbound request headers recorded on every usage event of the request, so callers that share one API key can be attributed by headers they send.
• upstream_response_headers: upstream response headers recorded per attempt, each read off that attempt's own upstream response, such as a provider's request ID or region.

Each exporter records only the headers in its own lists, so two exporters can record different headers. Recording does not depend on content_mode, and the values are never sent to the control plane. See Attribute Events with Request Headers for the configuration, the name rules, and the value rules.

Destinations name the values differently:

Exporter	Request Headers	Upstream Response Headers
Object storage	A nested request_headers object in each record.	A nested upstream_response_headers object in each record.
Alibaba Cloud SLS	One request_headers field holding the object as a compact JSON string.	One upstream_response_headers field holding the object as a compact JSON string.
Datadog	A nested aisix.request_headers attribute, so each header is aisix.request_headers.<name>.	A nested aisix.upstream_response_headers attribute, so each header is aisix.upstream_response_headers.<name>.
OTLP	One http.request.header.<name> attribute per header on the SERVER span. A gateway at 1.6.0 or earlier puts it on the operation span instead.	One http.response.header.<name> attribute per header on the GenAI operation span, which is the attempt span when the request has attempts.

Both OTLP attributes are string arrays with one element, following the OpenTelemetry HTTP semantic conventions.

Token Counts in Exported Telemetry

Exporters receive the token counts the usage event recorded, which keep the accounting the upstream used rather than the usage block the caller read. See Recorded Token Counts.

Destinations report them as follows:

• Object storage and Alibaba Cloud SLS keep the usage-event fields as recorded, including reasoning_tokens, total_tokens, and the audio and image counts prompt_audio_tokens, prompt_image_tokens, completion_audio_tokens, and completion_image_tokens. A zero reasoning_tokens and an unreported total_tokens or audio or image count are omitted.
• Datadog logs carry gen_ai.usage.input_tokens and gen_ai.usage.output_tokens, plus the recorded aisix.cache_creation_tokens, aisix.cache_creation_5m_tokens, aisix.cache_creation_1h_tokens, aisix.cache_read_tokens, aisix.reasoning_tokens, aisix.total_tokens, aisix.prompt_audio_tokens, aisix.prompt_image_tokens, aisix.completion_audio_tokens, and aisix.completion_image_tokens when present.
• OTLP spans carry gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, gen_ai.usage.cache_read.input_tokens, gen_ai.usage.cache_creation.input_tokens, gen_ai.usage.reasoning.output_tokens, and the gen_ai.usage.{audio,image}.{input,output}_tokens counts. See Token Usage Attributes.

In the OpenTelemetry generative-AI convention, input tokens include every cached token. OTLP and Datadog therefore report gen_ai.usage.input_tokens as prompt_tokens + cache_creation_tokens + cache_read_tokens. The two cache fields are non-zero only for usage reported in the Anthropic shape, such as Anthropic and Amazon Bedrock, which counts cache reads and writes beside the input tokens. An OpenAI-shape upstream already counts its cached tokens inside prompt_tokens, so its exported input count is unchanged. For example, an Anthropic reply with 19 input tokens, 1,200 cache read tokens, and 300 cache creation tokens exports gen_ai.usage.input_tokens as 1,519. A gateway at 1.6.0 or earlier exports 19 for the same reply.

In the OpenTelemetry generative-AI convention, output tokens include reasoning. OTLP and Datadog therefore report gen_ai.usage.output_tokens as completion_tokens + reasoning_tokens when the event counts reasoning beside the completion, and as completion_tokens otherwise. The test is the one the control plane prices by: total_tokens and reasoning_tokens are both greater than zero, and total_tokens equals prompt + completion + reasoning tokens but not prompt + completion tokens. See How Reasoning Tokens Are Billed.

The test applies per event, not per provider. A Gemini thinking model on the Vertex AI adapter normally passes it, and so does an OpenAI-compatible upstream that reports reasoning outside completion_tokens with a total that includes it. For Gemini, the exported output count is the same as before; the OpenAI-compatible case exports completion + reasoning where a gateway at 1.4.0 or earlier exported the completion alone.

AISIX Enterprise or AISIX Cloud

The dashboard manages the same exporter resources from the target environment's Observability view. The exporter form collects the destination fields, content mode, and any kind-specific options.

Whether an exporter is saved through the API or the dashboard, the control plane projects the configuration to the AISIX gateways attached to that environment.

The following behavior applies to exporters projected to AISIX gateways:

• The AISIX gateway sends request telemetry directly to your destination. The control plane does not proxy exported telemetry.
• Prompt and response content stays on the gateway unless you enable full content capture on an exporter.
• Credential references are resolved by the AISIX gateway. When a destination needs runtime credentials, the dashboard shows the environment variables to configure on the gateway.
• Delivery health comes from gateway heartbeat data and shows whether batches are being shipped or whether the gateway is reporting a delivery error.

For OTLP/HTTP exporters, the dashboard provides presets for Langfuse, Honeycomb, and Grafana Cloud Tempo, and accepts custom OTLP endpoints.

The trace UI URL template is optional. Use it when the Request Logs view should link a request record to an external trace UI. The template must include {request_id} so the control plane can replace it with the request ID from the log record.

Next Steps

Use OTLP Trace Reference to interpret trace hierarchy, parentage, status, and attributes. See Export Request Traces to ClickHouse to connect AISIX to ClickStack or deploy a Collector for an existing ClickHouse database.

Continue with Load Request Telemetry into Snowflake to query object-storage telemetry in Snowflake. Use Metrics and Usage Events to compare exported records with gateway metrics, and Access Logs and Request Correlation to trace individual requests.