# Observability Exporters

Observability exporters send usage events from the gateway to destinations used for tracing, logging, storage, or accounting. This guide configures an OTLP/HTTP exporter through the control-plane path for AISIX Enterprise and AISIX Cloud or the resources-file path for AISIX OSS. It then explains destination choices, content capture, and delivery behavior.

## Prerequisites[​](#prerequisites "Direct link to Prerequisites")

Before starting, prepare the following:

* One of these configuration paths:

  <!-- -->

  * AISIX Enterprise or AISIX Cloud with an environment, an attached gateway, and a write-scoped admin token. For AISIX Enterprise, follow the [AISIX Enterprise getting-started guide](https://docs.api7.ai/ai-gateway/getting-started/aisix-cloud-quickstart.md). To request AISIX Cloud access, [contact API7](https://api7.ai/contact).
  * An open-source AISIX gateway that loads a declarative [`resources.yaml`](https://docs.api7.ai/ai-gateway/reference/resources-file.md) file.

* A telemetry destination for the exporter you plan to use.

* A working model alias and caller API key for delivery verification.

Export the gateway and request values used for verification:

```
# AISIX_PROXY has no trailing slash or endpoint path

# The local quickstarts use http://127.0.0.1:3000

export AISIX_PROXY="YOUR_AISIX_GATEWAY_URL"

export AISIX_API_KEY="YOUR_CALLER_API_KEY"

export MODEL_ALIAS="YOUR_MODEL_ALIAS"
```

For the control-plane examples, export the Admin API base URL, admin token, and environment ID:

```
# AISIX_CP includes /api and has no trailing slash

# The local AISIX Enterprise quickstart uses http://localhost:8080/api

export AISIX_CP="YOUR_AISIX_CONTROL_PLANE_API_URL"

export AISIX_TOKEN="YOUR_ADMIN_TOKEN"

export ENV_ID="YOUR_ENVIRONMENT_ID"
```

In other deployments, use the reachable AISIX Cloud Admin API URL.

## Choose an Exporter Kind[​](#choose-an-exporter-kind "Direct link to Choose an Exporter Kind")

All exporter kinds are available through both configuration paths. AISIX Cloud Admin API requests and resource-file entries select a kind with the `kind` value, and the dashboard shows a label for the same underlying kind:

| `kind` Value   | Dashboard Label   | Use When                                                                                                         |
| -------------- | ----------------- | ---------------------------------------------------------------------------------------------------------------- |
| `otlp_http`    | OTLP/HTTP         | You already collect traces through an OTLP/HTTP collector or vendor endpoint.                                    |
| `object_store` | Object storage    | You want batched NDJSON request events in Amazon S3, S3-compatible storage, Google Cloud Storage, or Azure Blob. |
| `datadog`      | Datadog           | You use Datadog Logs HTTP intake.                                                                                |
| `aliyun_sls`   | Alibaba Cloud SLS | You use Alibaba Cloud Simple Log Service as the log destination.                                                 |

The gateway sends telemetry directly to the selected destination.

## Configure an OTLP Exporter[​](#configure-an-otlp-exporter "Direct link to Configure an OTLP Exporter")

Set the collector endpoint and authorization header used by the example:

```
# Replace with your values

export OTLP_ENDPOINT="https://collector.example.com/v1/traces"

export OTLP_AUTH_HEADER="Bearer YOUR_COLLECTOR_TOKEN"
```

The endpoint must be reachable from the gateway process. It can be an `https://` or `http://` URL with any host, such as an in-cluster collector at `http://otel-collector-dev:4318/v1/traces`. A receiver running beside a gateway process on the same host can use `http://localhost:4318/v1/traces`; a containerized gateway needs the receiver's container-network or host address instead. Remove the `headers` block from the exporter when the receiver does not require authentication.

With `http://`, the static headers and the exported telemetry cross the network in plaintext. The headers can carry an authorization token and the telemetry can include prompt and response content, so use `http://` on networks you trust.

### AISIX Enterprise or AISIX Cloud[​](#aisix-enterprise-or-aisix-cloud "Direct link to AISIX Enterprise or AISIX Cloud")

Create the exporter and capture its ID:

```
EXPORTER_ID=$(curl -sS -X POST "$AISIX_CP/environments/$ENV_ID/observability_exporters" \

  -H "Authorization: Bearer $AISIX_TOKEN" \

  -H "Content-Type: application/json" \

  -d '{

    "name": "prod-otlp",

    "kind": "otlp_http",

    "endpoint": "'"${OTLP_ENDPOINT}"'",

    "headers": {

      "Authorization": "'"${OTLP_AUTH_HEADER}"'"

    },

    "sample_rate": 1

  }' | jq -r '.observability_exporter.id')
```

❶ Set static headers only when the OTLP destination requires them. Header values are encrypted at rest and never returned by read operations.

❷ `sample_rate: 1` keeps the delivery check deterministic. It is equivalent to omitting the field, which exports every request trace. After verification, lower the value when you need to reduce span volume.

Retrieve the exporter to confirm the stored configuration:

```
curl -sS "$AISIX_CP/environments/$ENV_ID/observability_exporters/$EXPORTER_ID" \

  -H "Authorization: Bearer $AISIX_TOKEN"
```

You should see a response similar to the following. Reads expose the configured header names through `header_keys` and confirm stored values through `headers_set`; the header values themselves are never returned.

```
{

  "observability_exporter": {

    "id": "b46a9f5d-6a4d-4bb1-ae2c-4ab1b22f5e80",

    "env_id": "0f2f6a1e-9d33-4a8f-9a6e-2a7b6a9c1d2e",

    "name": "prod-otlp",

    "enabled": true,

    "kind": "otlp_http",

    "endpoint": "https://collector.example.com/v1/traces",

    "header_keys": ["Authorization"],

    "headers_set": true,

    "sample_rate": 1,

    "created_at": "2026-07-22T08:30:00Z",

    "updated_at": "2026-07-22T08:30:00Z"

  }

}
```

Keep `$EXPORTER_ID` for later updates or deletion. Exporters are enabled by default. Set `enabled: false` to save the resource without sending telemetry. Disabling or deleting an exporter stops its delivery pipeline on each gateway once the configuration change arrives: events already queued are delivered first, and the exporter drops out of the gateway's delivery health report. Re-enabling it starts a fresh pipeline with its counters reset.

### AISIX OSS[​](#aisix-oss "Direct link to AISIX OSS")

Add this exporter to `observability_exporters` in the complete resources file. Environment interpolation keeps the collector token out of the file:

resources.yaml (OTLP exporter)

```
observability_exporters:

  - id: b9e50432-01ee-42bb-882c-3fbd2ac29a18

    name: prod-otlp

    kind: otlp_http

    endpoint: ${OTLP_ENDPOINT}

    headers:

      Authorization: ${OTLP_AUTH_HEADER}

    sample_rate: 1
```

Validate the assembled complete file:

```
aisix validate --resources resources.yaml
```

Start or restart the gateway with `OTLP_ENDPOINT` and `OTLP_AUTH_HEADER` in its process environment. Reload a running gateway only if those variables are already available to the process. Exporters are enabled by default; add `enabled: false` to keep the entry without sending telemetry.

## Verify OTLP Delivery[​](#verify-otlp-delivery "Direct link to Verify OTLP Delivery")

A saved exporter confirms only that AISIX accepted its configuration. Verify the OTLP exporter configured above by capturing the request ID that AISIX returns, finding that ID at the destination, and checking the gateway's delivery signal. If you are checking an existing OTLP exporter with `sample_rate` below `1`, temporarily set the rate to `1` so this request cannot be sampled out.

Send one successful request through a working model alias and capture the ID AISIX returns. This remains accurate whether AISIX accepts caller-supplied IDs or generates its own:

```
export EXPORTER_NAME="prod-otlp"



if TEST_REQUEST_ID=$(

  set -o pipefail

  curl -fsS -D - -o /dev/null \

    -X POST "$AISIX_PROXY/v1/chat/completions" \

    -H "Authorization: Bearer $AISIX_API_KEY" \

    -H "Content-Type: application/json" \

    -d '{

      "model": "'"${MODEL_ALIAS}"'",

      "messages": [{"role": "user", "content": "Reply with: exporter check"}]

    }' \

    | awk 'tolower($1) == "x-aisix-request-id:" { id = $2; sub(/\r$/, "", id) } END { print id }'

) && test -n "$TEST_REQUEST_ID"; then

  export TEST_REQUEST_ID

  printf 'request ID: %s\n' "$TEST_REQUEST_ID"

else

  unset TEST_REQUEST_ID

  printf 'verification request failed or returned no request ID\n' >&2

  false

fi
```

Wait for the exporter to flush the batch, then search your OTLP backend for a span whose `aisix.request_id` attribute equals `$TEST_REQUEST_ID`. Finding the span confirms end-to-end delivery: the gateway loaded the exporter, authenticated to the destination when required, and the destination accepted the batch.

For AISIX Enterprise or AISIX Cloud, the exporter row in the environment's Observability view reports whether delivery is healthy and how many batches online gateways have shipped. You can inspect the same per-gateway heartbeat data through the Admin API:

```
curl -sS "$AISIX_CP/environments/$ENV_ID/dp_nodes" \

  -H "Authorization: Bearer $AISIX_TOKEN" \

  | jq --arg exporter "$EXPORTER_NAME" \

    '[.data[] | select(.status != "offline") | .exporter_health[]? | select(.name == $exporter)]'
```

After the next heartbeat, at least one online gateway should report `delivered_batches` greater than zero and `last_error: null`. The counters reset when a gateway restarts or the exporter configuration changes. A non-null `last_error` means delivery is currently failing. The historical fields `failed_batches` and `last_failure_unix` remain populated after a later success clears `last_error`.

For an open-source AISIX gateway, confirm success at the destination. If the record does not arrive, check gateway logs for `sink delivery failed; retrying` or `sink delivery dropped`.

## Understand Delivery Behavior[​](#understand-delivery-behavior "Direct link to Understand Delivery Behavior")

Every exporter runs its own delivery pipeline: an in-memory queue in front of a worker that batches records and ships them to the destination. The behavior below is fixed and has no configuration.

Records buffer until the batch reaches 100 records or one second passes, whichever comes first.

A destination that refuses a batch transiently — a connection failure, a 429, or a 5xx — is retried on a time budget rather than a fixed number of attempts. The delay starts at 200 milliseconds and doubles up to a 30-second ceiling, and retries continue until **300 seconds after that batch's first attempt**. When the budget is spent, the batch is dropped and counted as `retries_exhausted` in `aisix_otlp_fanout_drops_total`. A `Retry-After` header on a 429 or 503 replaces the computed delay, capped at the same 30 seconds, so a destination asking for a longer pause is honored rather than hammered. A budget rather than an attempt count is what lets an exporter survive a receiver outage of a given length: an outage shorter than the budget loses nothing.

While a batch is being retried, new records keep arriving. Each exporter queue holds 8,192 records; when it is full the **newest** record is refused and counted as `queue_full`, so the records already queued still have their chance at the destination.

caution

An exporter with full content capture enabled owns its records, prompt and response text included. With `content_max_bytes` at its default of 131,072, a full 8,192-record queue behind a stalled destination can hold hundreds of megabytes of resident memory. Size `content_max_bytes` against the gateway's memory limit when you enable full capture on a destination that can stall. See [Configure Content Capture](#configure-content-capture).

## Interpret OTLP Traces[​](#interpret-otlp-traces "Direct link to Interpret OTLP Traces")

AISIX exports one connected trace hierarchy per sampled request. See [OTLP Trace Reference](https://docs.api7.ai/ai-gateway/reference/otlp-traces.md) for span parentage, W3C trace continuation, names, status, semantic-convention attributes, captured content, and resource fields.

## Configure Other Exporters[​](#configure-other-exporters "Direct link to Configure Other Exporters")

Use another exporter kind when telemetry should go to object storage or a logging service instead of an OTLP trace backend. With AISIX Enterprise or AISIX Cloud, send one of the following objects as the body of `POST $AISIX_CP/environments/$ENV_ID/observability_exporters`.

### Object Storage[​](#object-storage "Direct link to Object Storage")

For object storage, choose the storage provider, bucket, and key prefix. The default authentication mode uses a credential reference resolved by the gateway:

```
{

  "name": "request-events-s3",

  "kind": "object_store",

  "provider": "s3",

  "bucket": "acme-aisix-events",

  "prefix": "ai-gateway",

  "region": "us-east-1",

  "credential_ref": "acme_s3"

}
```

Object storage supports Amazon S3, Google Cloud Storage, Azure Blob, and S3-compatible targets. Cloud identity is supported only for S3 or GCS when the gateway has an attached identity that can write to the bucket. Use `credential_ref` for Azure Blob and for S3-compatible targets that require static credentials.

For an S3-compatible target such as MinIO, Cloudflare R2, or Alibaba Cloud OSS, set the target endpoint explicitly. Without an endpoint, the exporter uses the native AWS S3 endpoint.

`endpoint` overrides the storage provider's native endpoint, and its meaning follows `provider`:

| `provider`   | What `endpoint` sets                                                                                                                                                                                                                                                                                              |
| ------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `s3`         | The S3-compatible host to upload to.                                                                                                                                                                                                                                                                              |
| `gcs`        | The Cloud Storage XML API base URL, such as a private endpoint. Uploads use the XML API, so the endpoint must serve it; an emulator that accepts only JSON API uploads, such as the stock `fake-gcs-server` image, cannot receive objects. It takes precedence over a `gcs_base_url` in the service-account JSON. |
| `azure_blob` | The Blob service endpoint.                                                                                                                                                                                                                                                                                        |

The value can be an `https://` or `http://` URL with any host, such as an in-cluster MinIO at `http://minio.storage.svc:9000`. With `http://`, the uploaded request events cross the network in plaintext, so use it on networks you trust.

`endpoint` cannot be combined with cloud identity: the AISIX Cloud Admin API rejects the combination, and an open-source AISIX gateway that loads it never uploads and reports a permanent delivery error. The dashboard offers the field for S3-compatible targets, Cloud Storage, and Azure Blob, and hides it when cloud identity is selected.

### Alibaba Cloud SLS[​](#alibaba-cloud-sls "Direct link to Alibaba Cloud SLS")

Configure the endpoint host, project, log store, and credential reference:

```
{

  "name": "request-events-sls",

  "kind": "aliyun_sls",

  "endpoint": "ap-southeast-3.log.aliyuncs.com",

  "project": "acme-observability",

  "logstore": "ai-gateway",

  "credential_ref": "acme_sls"

}
```

### Datadog[​](#datadog "Direct link to Datadog")

Configure the Datadog site, service name, tags, and credential reference:

```
{

  "name": "request-events-datadog",

  "kind": "datadog",

  "site": "datadoghq.com",

  "service": "ai-gateway",

  "tags": ["team:platform", "tier:prod"],

  "credential_ref": "acme_datadog"

}
```

For the resources-file path, add these entries to `observability_exporters`:

resources.yaml (request exporters)

```
observability_exporters:

  - id: f49606bd-fca7-4636-8ef5-2bbcc0afa00f

    name: request-events-s3

    kind: object_store

    provider: s3

    bucket: acme-aisix-events

    prefix: ai-gateway

    region: us-east-1

    credential_ref: acme_s3



  - id: 54ca2bec-f4a8-44d0-b3ba-d74e9966e9b0

    name: request-events-sls

    kind: aliyun_sls

    endpoint: ap-southeast-3.log.aliyuncs.com

    project: acme-observability

    logstore: ai-gateway

    credential_ref: acme_sls



  - id: 6d589715-66bd-4382-b501-60ce5f184869

    name: request-events-datadog

    kind: datadog

    site: datadoghq.com

    service: ai-gateway

    tags: ["team:platform", "tier:prod"]

    credential_ref: acme_datadog
```

Keep only the destinations you use, then validate the resources file and start or reload the gateway as described for the OTLP example. Verify these exporters with the same request-ID approach used for OTLP: search object-storage and SLS records by `request_id`, or Datadog logs by `aisix.request_id`. The [Snowflake guide](https://docs.api7.ai/ai-gateway/observability/load-logs-into-snowflake.md) shows how to inspect object-storage output directly.

SLS, Datadog, and object-storage exporters keep destination credentials out of the exporter resource by using credential references or cloud identity. The gateway resolves those credentials locally when it sends telemetry.

## Configure Content Capture[​](#configure-content-capture "Direct link to Configure Content Capture")

Exporters include request status, token counts, model and provider identifiers, request IDs, finish reason, and timing by default. They carry no cost: the control plane prices each request, so read cost from its [usage data](https://docs.api7.ai/ai-gateway/cloud/model-pricing.md#how-aisix-resolves-prices). Gateways at 1.9.0 or earlier also export a `cost_usd` field, which is `0` except on Realtime sessions and Batch jobs. Prompt and response bodies remain excluded unless full content capture is enabled on an OTLP/HTTP, SLS, or Datadog exporter.

Failure metadata needs separate handling. Every failed event carries an `error_message` of up to 2,048 characters, which can include a rejected request value or error detail returned by an upstream. `content_mode: metadata_only` does not remove that field. Approve the destination to receive failure details even when you do not enable full content capture.

### Enable Full Content Capture[​](#enable-full-content-capture "Direct link to Enable Full Content Capture")

Add `content_mode` and `content_max_bytes` when creating an exporter. In AISIX Enterprise and AISIX Cloud, patch an existing exporter with the same fields:

```
curl -sS -X PATCH "$AISIX_CP/environments/$ENV_ID/observability_exporters/$EXPORTER_ID" \

  -H "Authorization: Bearer $AISIX_TOKEN" \

  -H "Content-Type: application/json" \

  -d '{

    "content_mode": "full",

    "content_max_bytes": 131072

  }'
```

In the dashboard, set **Content mode** to **Full**, then adjust **Max content bytes** in the exporter form.

For the resources-file path, add the fields to the exporter entry, then validate and reload the file:

```
observability_exporters:

  - id: b9e50432-01ee-42bb-882c-3fbd2ac29a18

    name: prod-otlp

    kind: otlp_http

    endpoint: ${OTLP_ENDPOINT}

    headers:

      Authorization: ${OTLP_AUTH_HEADER}

    content_mode: full

    content_max_bytes: 131072
```

Use full content capture only when the destination is approved to receive end-user prompt and response text. AISIX applies the configured byte cap to each captured content field independently.

### Understand Truncated Records[​](#understand-truncated-records "Direct link to Understand Truncated Records")

When valid JSON exceeds `content_max_bytes`, AISIX first tries to reduce the value structurally so the exported field remains valid JSON. If the reduced value still cannot fit the configured cap, AISIX falls back to a UTF-8-safe byte cut. The captured prompt is the serialized request body and follows the same behavior, after AISIX [replaces its base64 media](#base64-media-in-captured-content).

| Content                                                              | Truncation Behavior                                                                                                                  |
| -------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------ |
| Long string                                                          | Keeps a prefix and adds the inline marker `...[aisix: truncated, N bytes total]`.                                                    |
| Base64 data URI                                                      | Replaces the encoded data with a size placeholder.                                                                                   |
| Long array                                                           | Keeps head and tail samples around an `{"_aisix_truncated": true, "omitted_items": N}` element that accounts for every omitted item. |
| Non-JSON content, or JSON that cannot fit after structural reduction | Cuts the value at a UTF-8 character boundary.                                                                                        |

When truncation occurs, OTLP records include `aisix.content_truncated: true`. Datadog and SLS records include `content_truncated: true`.

### Base64 Media in Captured Content[​](#base64-media-in-captured-content "Direct link to Base64 Media in Captured Content")

With `content_mode: full`, AISIX replaces base64 media in a captured request with the placeholder `[aisix: base64 data uri omitted, N bytes]`, whatever its size and before applying `content_max_bytes`. The same rule covers every captured request type, including embeddings, rerank, completions, decisions, and MCP `tools/call` arguments, and every content-bearing destination: OTLP, Alibaba Cloud SLS, and Datadog. The `object_store` kind carries no content. Only the exported copy changes. The upstream request and the client's response are not affected.

AISIX recognizes base64 media by its form or by the part that declares it:

* A string that is a base64 data URI, in any field. This covers image, audio, and PDF parts. The URI keeps its head, as in `data:image/png;base64,[aisix: base64 data uri omitted, N bytes]`, and `N` is the length of the whole URI.
* `source.data` of a base64 `image` or `document` block, including one inside a tool result.
* `input_audio.data`, `file_data` on an `input_file` part or a `file` object, and the `result` of a replayed `image_generation_call`.
* `inlineData.data` of Gemini bodies and `source.bytes` of Bedrock Converse bodies, which appear on [passthrough routes](https://docs.api7.ai/ai-gateway/endpoints/provider-passthrough.md).
* `data` of an MCP `image` or `audio` block beside its `mimeType`, and `resource.blob` of an embedded `resource` block.

For a bare field like these, `N` is the length of the payload.

The structured fields (`gen_ai.input.messages` on OTLP and Datadog, `input_messages` on SLS) keep the media part and any media type available, so a record still shows that the request carried media:

```
{"type":"blob","modality":"image","mime_type":"image/png","content":"[aisix: base64 data uri omitted, 1368 bytes]"}
```

An `http` or `https` URL stays as sent. AISIX does not replace a data URI embedded in other text, a JSON string of tool arguments, or bare base64 in a field not listed above. Most content that does not parse as JSON, such as A2A message text or a stream frame cut partway through a payload, has no fields to read, so AISIX cuts it at the first media marker it shows, such as `;base64,`, and writes the placeholder there.

On the response side, the replacement covers three cases. On `/v1/images/generations` and `/v1/images/edits`, `b64_json` and any data URI become the placeholder, and `url`, `revised_prompt`, and other fields stay. On passthrough routes, the same recognition runs over the upstream's own body and raw stream frames, where it also covers `partial_image_b64` and `bytesBase64Encoded`, while a model's extracted answer text is never rewritten. On MCP `tools/call`, the captured `result` gets the placeholder, as described below. Other captured responses keep their base64 unless truncation replaces it, including embeddings.

In an MCP `tools/call` result, including one that reports `isError`, AISIX replaces `data` of `image` and `audio` blocks, `resource.blob`, and any string that starts as a base64 data URI, wherever it sits, including in `structuredContent`. The block, its `mimeType`, and the resource `uri` stay. AISIX does not replace base64 embedded in a `text` block or in `resource.text`, or bare base64 under a `structuredContent` key it does not recognize. A common case is a tool that follows the MCP specification and also returns its `structuredContent` serialized into a `text` block: the payload in that text block is still exported.

A gateway at 1.8.0 or earlier can export base64 media as sent, and captures the image endpoints' `b64_json`.

On passthrough routes, a response that is not valid UTF-8, nests deeper than 128 levels, or holds a number the parser cannot read, such as `1e999`, can reach the export with its base64 intact, or cut at the first media marker, depending on the route's protocol. A shape with no marker, such as an MCP `blob`, is exported as is in that case.

### Captured Content in Datadog and SLS Records[​](#captured-content-in-datadog-and-sls-records "Direct link to Captured Content in Datadog and SLS Records")

With `content_mode: full`, the Datadog and Alibaba Cloud SLS exporters report a message exchange with the same structured content as the OTLP exporter's [Captured Content](https://docs.api7.ai/ai-gateway/reference/otlp-traces.md#captured-content) attributes: the same messages, the same `text`, `reasoning`, and `tool_call` output parts, and the same `finish_reason` values. Each sink uses its own field names:

| Content                                        | Alibaba Cloud SLS     | Datadog                      |
| ---------------------------------------------- | --------------------- | ---------------------------- |
| Request messages                               | `input_messages`      | `gen_ai.input.messages`      |
| Anthropic `system` or Responses `instructions` | `system_instructions` | `gen_ai.system_instructions` |
| Declared tools                                 | `tool_definitions`    | `gen_ai.tool.definitions`    |
| Assistant output message                       | `output_messages`     | `gen_ai.output.messages`     |
| Prompt side that is not structured             | `prompt`              | `gen_ai.prompt`              |
| Response side that is not structured           | `response`            | `gen_ai.completion`          |

SLS writes each structured value as a compact JSON string. Datadog writes it as nested JSON. A structured field is written only when the record has that content, so `system_instructions` appears only for an Anthropic Messages `system` field or a Responses `instructions` field. A chat completions system message stays in `input_messages`. For a Responses request, the declared tools also include the tools of its `additional_tools` input items, after any top-level tools and with repeated declarations merged, and those items do not appear among the request messages. The declared tools also include the tools a client-executed tool search loaded, as described in [Tool Parameters](https://docs.api7.ai/ai-gateway/endpoints/responses-api.md#tool-parameters). The search's `tool_search_call` and `tool_search_output` items appear among the request messages as a tool call and a tool response.

The prompt and response sides are judged separately, by the same rule as OTLP. A side stays in the flat field when capture truncated it, when it does not convert, or when a structured value would exceed `content_max_bytes`. Content that does not convert includes payloads that are not a message exchange, such as embeddings input, images, audio, jobs, and agent text. The output message falls back the same way as on OTLP: when the captured output parts outgrow the cap, it is built from the response text, and the response stays flat only when that text was truncated too. One record can therefore carry structured input messages beside a flat `response`, or the other way round. MCP tool calls keep their arguments in the flat prompt field and their result in the flat response field. `content_truncated: true` marks a record whose content was truncated, as before.

A side with neither structured content nor captured text is still written as an empty flat field. For example, a request abandoned before any response records its messages in `input_messages` and an empty `response`. OTLP omits such a side.

`content_max_bytes` caps each field separately, so one record can carry up to four capped content fields: the three input fields and the output message.

### Capture Passthrough Content[​](#capture-passthrough-content "Direct link to Capture Passthrough Content")

For successfully relayed [passthrough route](https://docs.api7.ai/ai-gateway/endpoints/provider-passthrough.md) traffic, full content capture records the request body as a string, subject to the exporter's content cap and JSON-aware structural truncation. Buffered responses record extracted text when the response matches a supported extraction shape and otherwise record the body as text. Streamed responses record accumulated extracted text; opaque data payloads are retained as text. Base64 media is replaced in each, as described in [Base64 Media in Captured Content](#base64-media-in-captured-content).

### Capture Failed Requests[​](#capture-failed-requests "Direct link to Capture Failed Requests")

When any OTLP/HTTP, SLS, or Datadog exporter runs at `content_mode: full`, every failure row carries the request body and the response body for that row, on every endpoint and status. A failure row is one with a status of `400` or above or a non-empty `error_class`. That includes `401` and `403` from an upstream, a gateway refusal such as a forbidden model, and each failed attempt of a retry or fallback chain. The same request exported under `metadata_only` has neither `prompt` nor `response`.

The request goes where the success path puts it. On `/v1/chat/completions`, `/v1/messages`, and `/v1/responses` it becomes structured input messages when it converts (`gen_ai.input.messages` on OTLP and Datadog, `input_messages` on SLS), and otherwise stays in the flat prompt field. Multipart requests report their text fields plus a `<name>_sha256` checksum for each file, and a `/v1/files` upload reports its text fields plus `file_sha256`. File bytes are never stored.

The response goes in the flat response field: `aisix.content.response` on OTLP, `response` on SLS, and `gen_ai.completion` on Datadog. `content_max_bytes` caps it, as it does the request. AISIX chooses the response in this order:

1. What a stream delivered before it failed part-way or the caller left.
2. The body the upstream sent for this attempt, up to the first 64 KiB.
3. On the request's terminal row, the error body the gateway sent the caller. That is the Anthropic shape on `/v1/messages` and `/v1/messages/count_tokens`, and JSON-RPC on MCP and A2A.
4. Otherwise, a body AISIX builds: `{"error":{"message":<error_message>,"type":<error_class>}}`.

So `response` is not always what the upstream said. When an upstream answers with no body, the terminal row's `response` is the error the gateway sent the caller, such as `{"error":{"message":"upstream returned 404 Not Found","type":"upstream_error"}}`. Read `error_message` in that case: `with no error detail in the response` says the upstream gave none. See [Read an Upstream Error Message](https://docs.api7.ai/ai-gateway/deployment/troubleshooting.md#read-an-upstream-error-message).

Some failures differ:

* A `499` row carries the request and no invented response, because the caller left. Anything a stream had delivered still appears.
* A response-side guardrail refusal records the refusal returned to the caller, not the blocked upstream output.
* A passthrough route does not decode an upstream's compressed error body. Its `error_message` reads `upstream returned HTTP <status> with a compressed error body`, and `response` holds the body AISIX builds from that message rather than compressed bytes.
* A Realtime upstream that refuses the handshake with an empty or compressed body has no error body to record, because the caller gets a WebSocket frame instead. `response` holds the body AISIX builds, such as `{"error":{"message":"upstream returned HTTP 500 with no error detail in the response","type":"upstream_status"}}`.
* An A2A agent's error body is read only for a full-content exporter, and the gateway reads it before answering. A slow agent therefore delays the caller's answer, up to the agent's timeout.

A failure row has no request body when the route has none. That covers job routes without a body, such as get, list, delete, and cancel, passthrough requests with an empty body, and every Realtime error event. A multipart row that fails before its fields are read has none either, and neither does a passthrough rejection before the body is read, including authentication and authorization.

Some failures produce no usage event, and so no row, only an access-log line: a request body the gateway cannot parse, such as malformed JSON, and a caller authentication failure. See [What the Usage Log Records](https://docs.api7.ai/ai-gateway/observability/metrics-and-usage-events.md#what-the-usage-log-records) for these and for the passthrough exception. Content reaches exporters only: the console does not display request bodies.

A gateway at 1.8.0 or earlier does not apply this complete request-and-response capture consistently across failure paths.

### Protect Failure Content[​](#protect-failure-content "Direct link to Protect Failure Content")

Treat a destination with `content_mode: full` as a content-bearing data source, and restrict access to it accordingly. Failure rows are not always masked:

* A row that failed before guardrails ran carries the request body as sent, unmasked. Examples are a `404` for an unknown model, a `403`, and an IP refusal.
* A `499` from a caller that left while a guardrail was still running carries the body unmasked on images, embeddings, rerank, completions, speech, and `count_tokens`.
* A row that a guardrail blocked has mask-action rules applied, but the text that triggered the block stays in the row. A mask that needs a remote guardrail service, such as AWS Bedrock, is not applied, because the request is already refused.
* Passthrough routes, job routes, and A2A cannot rewrite a request, so their rows carry the body as sent, including on a guardrail block.

Every failed attempt of a fallback chain repeats the request body, so full-content volume grows on failing traffic.

### Review Captured Content by Endpoint[​](#review-captured-content-by-endpoint "Direct link to Review Captured Content by Endpoint")

| Endpoint Type                | Captured Content                                                                                                                                                                                                                                                                                                                                                                                                 |
| ---------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Text generation              | Response text. OTLP, Datadog, and Alibaba Cloud SLS export the output as structured parts, including reasoning and tool calls, as described in [Captured Content](https://docs.api7.ai/ai-gateway/reference/otlp-traces.md#captured-content) and [Captured Content in Datadog and SLS Records](#captured-content-in-datadog-and-sls-records).                                                                    |
| MCP `tools/call`             | `params.arguments` after traffic-control and input-guardrail checks, with any data masking applied; the JSON-RPC `result` when present, with base64 media replaced by a placeholder as described in [Base64 Media in Captured Content](#base64-media-in-captured-content). An output-guardrail block records the arguments and the JSON-RPC refusal, not the blocked result. Other MCP methods are not captured. |
| A2A                          | Text from request and reply message parts on `message/send` and `message/stream`. File parts, data parts, and JSON-RPC envelopes are not captured, except that a failure row for a call with no message text, such as a task read or a cancel, reports its JSON-RPC body.                                                                                                                                        |
| Embeddings and rerank        | Full response JSON.                                                                                                                                                                                                                                                                                                                                                                                              |
| Image generation and editing | Response JSON, with `b64_json` replaced by a placeholder as described in [Base64 Media in Captured Content](#base64-media-in-captured-content). `url`, `revised_prompt`, and other fields stay. Only the export changes: the client still receives the image.                                                                                                                                                    |
| Audio transcription          | Returned transcript. Uploaded audio is not captured; its SHA-256 checksum is recorded alongside the request's text fields.                                                                                                                                                                                                                                                                                       |
| Text to speech               | Binary speech response is not captured.                                                                                                                                                                                                                                                                                                                                                                          |

## Model Fields in Exported Telemetry[​](#model-fields-in-exported-telemetry "Direct link to Model Fields in Exported Telemetry")

Usage telemetry records both the model alias requested by the caller and the model that served an attempt when those values differ. This distinction matters for routed and ensemble traffic, where one caller-facing alias can resolve to several target calls.

Destinations render these values in their own telemetry format. OTLP traces use the following fields:

* `aisix.requested_model` contains the caller-requested alias or Model Group name.
* `gen_ai.request.model` contains the model name sent to the upstream for that attempt.
* `gen_ai.response.model` contains the concrete response model version when the provider reports one.
* `aisix.model_id` identifies the model resource that calls the upstream: the requested direct model, or the target a routing model or semantic router selected for that attempt. It is empty on a record written before a routing target is selected. See [Which Model a Record Names](https://docs.api7.ai/ai-gateway/observability/metrics-and-usage-events.md#which-model-a-record-names).
* `aisix.requested_model_id` identifies the model resource the caller requested. See [Correlate by the Requested Model ID](#correlate-by-the-requested-model-id).

A gateway at 1.6.0 or earlier puts the caller-requested alias in `gen_ai.request.model`.

### Correlate by the Requested Model ID[​](#correlate-by-the-requested-model-id "Direct link to Correlate by the Requested Model ID")

`aisix.requested_model` is a name, so renaming a model splits its telemetry into two series. `aisix.requested_model_id` carries the ID of the model the caller requested, which a rename does not change. Group or filter by it to follow a routing model, semantic router, or ensemble model across a rename.

Its value depends on the kind of model the caller requested:

| Requested model                  | `aisix.requested_model_id`                                                                                                                             | `aisix.model_id`                                                                                                                                                                                     |
| -------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Direct or embedding model        | The model's ID.                                                                                                                                        | The same ID.                                                                                                                                                                                         |
| Wildcard model                   | The wildcard model's ID.                                                                                                                               | The same ID.                                                                                                                                                                                         |
| Routing model or semantic router | Its own ID, on every record of the request, including the failed attempts, the one that answered, and the records written before a target is selected. | The target model that attempt was dispatched to. Empty on a record written before a target is selected.                                                                                              |
| Ensemble model                   | The ensemble model's ID, on the record of every panel member and of the judge.                                                                         | The panel member or judge model of that record. The ensemble's aggregate record and failures after the panel carry the ensemble model's ID. Empty when the request is refused before the panel runs. |

When the requested name resolved to a model but the gateway selected no target, such as a request the caller cancelled before the first attempt, a request a guardrail blocked, a request a rate limit rejected with `429`, or a response-cache hit, `aisix.requested_model_id` still carries the requested model's ID. For a routing model or semantic router, `aisix.model_id` is empty on those records, so a filter on the group's ID in `aisix.model_id` misses them; filter on `aisix.requested_model_id` instead.

When the name does not resolve to a configured model, the field is empty and the exporters omit it. That covers an unknown model name, the body `model` of a passthrough route, which the gateway does not look up as a model, and MCP and A2A calls, which name no model.

A gateway at 1.6.0 or earlier does not export `aisix.requested_model_id`.

### Model Fields in Other Destinations[​](#model-fields-in-other-destinations "Direct link to Model Fields in Other Destinations")

Datadog logs use `aisix.requested_model` for the caller-requested alias, `aisix.requested_model_id` for the ID of the requested model resource, `gen_ai.response.model` for the concrete response model version, and `aisix.model_id` for the model resource that calls the upstream, with the same values as the OTLP attribute.

Object storage and Alibaba Cloud SLS retain usage-event field names such as `requested_model`, `requested_model_id`, `model_id`, and `provider_model_version`. Their `model_id` takes the same values as `aisix.model_id`. Alibaba Cloud SLS omits an empty `requested_model_id`, and object storage writes it as an empty string.

## Request Kind in Exported Telemetry[​](#request-kind-in-exported-telemetry "Direct link to Request Kind in Exported Telemetry")

Every record also carries the kind of work the request asked for — a chat completion, an image generation, a video submission, a tool call. See [Tell Request Kinds Apart](https://docs.api7.ai/ai-gateway/observability/metrics-and-usage-events.md#tell-request-kinds-apart) for the values and what they mean.

Destinations name it differently:

* Object storage and Alibaba Cloud SLS keep the usage-event name, `operation`.
* Datadog maps it to `aisix.operation`.
* OTLP carries it as the `aisix.operation` span attribute, on every span of the request's hierarchy so a trace can be filtered by kind at its root.

OTLP spans also carry the OpenTelemetry `gen_ai.operation.name` attribute, and the two answer different questions. That one uses OpenTelemetry's own vocabulary where it has a value, which distinguishes model inference from agent and tool calls, and the `aisix.operation` value otherwise. It reports `chat` for chat completions, messages, and responses alike, so it cannot separate those three. Filter on `aisix.operation` when the endpoint matters.

## Request and Response Headers in Exported Telemetry[​](#request-and-response-headers-in-exported-telemetry "Direct link to Request and Response Headers in Exported Telemetry")

Each exporter can record the values of selected inbound request headers and of selected upstream response headers. Every exporter kind accepts two optional lists of header names:

* `request_headers`: inbound request headers recorded on every usage event of the request, so callers that share one API key can be attributed by headers they send.
* `upstream_response_headers`: upstream response headers recorded per attempt, each read off that attempt's own upstream response, such as a provider's request ID or region.

Each exporter records only the headers in its own lists, so two exporters can record different headers. Recording does not depend on `content_mode`, and the values are never sent to the control plane. See [Attribute Events with Request Headers](https://docs.api7.ai/ai-gateway/observability/metrics-and-usage-events.md#attribute-events-with-request-headers) for the configuration, the name rules, and the value rules.

Destinations name the values differently:

| Exporter          | Request Headers                                                                                                                                | Upstream Response Headers                                                                                                                    |
| ----------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------- |
| Object storage    | A nested `request_headers` object in each record.                                                                                              | A nested `upstream_response_headers` object in each record.                                                                                  |
| Alibaba Cloud SLS | One `request_headers` field holding the object as a compact JSON string.                                                                       | One `upstream_response_headers` field holding the object as a compact JSON string.                                                           |
| Datadog           | A nested `aisix.request_headers` attribute, so each header is `aisix.request_headers.<name>`.                                                  | A nested `aisix.upstream_response_headers` attribute, so each header is `aisix.upstream_response_headers.<name>`.                            |
| OTLP              | One `http.request.header.<name>` attribute per header on the SERVER span. A gateway at 1.6.0 or earlier puts it on the operation span instead. | One `http.response.header.<name>` attribute per header on the GenAI operation span, which is the attempt span when the request has attempts. |

Both OTLP attributes are string arrays with one element, following the OpenTelemetry HTTP semantic conventions.

## Token Counts in Exported Telemetry[​](#token-counts-in-exported-telemetry "Direct link to Token Counts in Exported Telemetry")

Exporters receive the token counts the usage event recorded, which keep the accounting the upstream used rather than the usage block the caller read. See [Recorded Token Counts](https://docs.api7.ai/ai-gateway/observability/metrics-and-usage-events.md#recorded-token-counts).

Destinations report them as follows:

* Object storage and Alibaba Cloud SLS keep the usage-event fields as recorded, including `reasoning_tokens`, `total_tokens`, and the audio and image counts `prompt_audio_tokens`, `prompt_image_tokens`, `completion_audio_tokens`, and `completion_image_tokens`. A zero `reasoning_tokens` and an unreported `total_tokens` or audio or image count are omitted.
* Datadog logs carry `gen_ai.usage.input_tokens` and `gen_ai.usage.output_tokens`, plus the recorded `aisix.cache_creation_tokens`, `aisix.cache_creation_5m_tokens`, `aisix.cache_creation_1h_tokens`, `aisix.cache_write_30m_tokens`, `aisix.cache_write_12h_tokens`, `aisix.cache_read_tokens`, `aisix.reasoning_tokens`, `aisix.total_tokens`, `aisix.prompt_audio_tokens`, `aisix.prompt_image_tokens`, `aisix.completion_audio_tokens`, and `aisix.completion_image_tokens` when present.
* OTLP spans carry `gen_ai.usage.input_tokens`, `gen_ai.usage.output_tokens`, `gen_ai.usage.cache_read.input_tokens`, `gen_ai.usage.cache_creation.input_tokens`, `gen_ai.usage.reasoning.output_tokens`, and the `gen_ai.usage.{audio,image}.{input,output}_tokens` counts. See [Token Usage Attributes](https://docs.api7.ai/ai-gateway/reference/otlp-traces.md#token-usage-attributes).

In the OpenTelemetry generative-AI convention, input tokens include every cached token. OTLP and Datadog therefore report `gen_ai.usage.input_tokens` as `prompt_tokens` + `cache_creation_tokens` + `cache_read_tokens`. The two cache fields are non-zero only for usage reported in the Anthropic shape, such as Anthropic and Amazon Bedrock, which counts cache reads and writes beside the input tokens. An OpenAI-shape upstream already counts its cached tokens inside `prompt_tokens`, so its exported input count is unchanged. For example, an Anthropic reply with 19 input tokens, 1,200 cache read tokens, and 300 cache creation tokens exports `gen_ai.usage.input_tokens` as 1,519. A gateway at 1.6.0 or earlier exports 19 for the same reply.

In the OpenTelemetry generative-AI convention, output tokens include reasoning. OTLP and Datadog therefore report `gen_ai.usage.output_tokens` as `completion_tokens` + `reasoning_tokens` when the event counts reasoning beside the completion, and as `completion_tokens` otherwise. The test is the one the control plane prices by: `total_tokens` and `reasoning_tokens` are both greater than zero, and `total_tokens` equals prompt + completion + reasoning tokens but not prompt + completion tokens. See [How Reasoning Tokens Are Billed](https://docs.api7.ai/ai-gateway/cloud/model-pricing.md#how-reasoning-tokens-are-billed).

The test applies per event, not per provider. A Gemini thinking model on the Vertex AI adapter normally passes it, and so does an OpenAI-compatible upstream that reports reasoning outside `completion_tokens` with a total that includes it.

## AISIX Enterprise or AISIX Cloud[​](#aisix-enterprise-or-aisix-cloud-1 "Direct link to AISIX Enterprise or AISIX Cloud")

The dashboard manages the same exporter resources from the target environment's Observability view. The exporter form collects the destination fields, content mode, and any kind-specific options.

Whether an exporter is saved through the API or the dashboard, the control plane projects the configuration to the AISIX gateways attached to that environment.

The following behavior applies to exporters projected to AISIX gateways:

* The AISIX gateway sends request telemetry directly to your destination. The control plane does not proxy exported telemetry.
* Prompt and response content stays on the gateway unless you enable full content capture on an exporter.
* Credential references are resolved by the AISIX gateway. When a destination needs runtime credentials, the dashboard shows the environment variables to configure on the gateway.
* Delivery health comes from gateway heartbeat data and shows whether batches are being shipped or whether the gateway is reporting a delivery error.

For OTLP/HTTP exporters, the dashboard provides presets for Langfuse, Honeycomb, and Grafana Cloud Tempo, and accepts custom OTLP endpoints.

The trace UI URL template is optional. Use it when the **Request Logs** view should link a request record to an external trace UI. The template must include `{request_id}` so the control plane can replace it with the request ID from the log record.

## Next Steps[​](#next-steps "Direct link to Next Steps")

Use [OTLP Trace Reference](https://docs.api7.ai/ai-gateway/reference/otlp-traces.md) to interpret trace hierarchy, parentage, status, and attributes. See [Export Request Traces to ClickHouse](https://docs.api7.ai/ai-gateway/observability/export-traces-to-clickhouse.md) to connect AISIX to ClickStack or deploy a Collector for an existing ClickHouse database.

Continue with [Load Request Telemetry into Snowflake](https://docs.api7.ai/ai-gateway/observability/load-logs-into-snowflake.md) to query object-storage telemetry in Snowflake. Use [Metrics and Usage Events](https://docs.api7.ai/ai-gateway/observability/metrics-and-usage-events.md) to compare exported records with gateway metrics, and [Access Logs and Request Correlation](https://docs.api7.ai/ai-gateway/observability/access-logs-and-request-correlation.md) to trace individual requests.
