Observability Exporters

Observability exporters send usage events from the gateway to destinations used for tracing, logging, storage, or accounting. This guide configures an OTLP/HTTP exporter using AISIX Cloud or the open-source AISIX gateway, then explains destination choices, content capture, and delivery behavior.

Prerequisites

Before starting, prepare the following:

• One of these configuration paths:
  • AISIX Cloud with an environment, an attached gateway, and a write-scoped admin token. For On-Premises, follow the AISIX Cloud Quickstart. To request Hybrid Cloud access, contact API7.
  • An open-source AISIX gateway that loads a declarative resources.yaml file.
• A telemetry destination for the exporter you plan to use.
• A working model alias and caller API key for delivery verification.

Export the gateway and request values used for verification:

# AISIX_PROXY has no trailing slash or endpoint path
# The local quickstarts use http://127.0.0.1:3000
export AISIX_PROXY="YOUR_AISIX_GATEWAY_URL"
export AISIX_API_KEY="YOUR_CALLER_API_KEY"
export MODEL_ALIAS="YOUR_MODEL_ALIAS"

For the AISIX Cloud examples, export the Admin API base URL, admin token, and environment ID:

# AISIX_CP includes /api and has no trailing slash
# The local On-Premises quickstart uses http://localhost:8080/api
export AISIX_CP="YOUR_AISIX_CLOUD_ADMIN_API_URL"
export AISIX_TOKEN="YOUR_ADMIN_TOKEN"
export ENV_ID="YOUR_ENVIRONMENT_ID"

In other deployments, use the reachable AISIX Cloud Admin API URL.

Choose an Exporter Kind

All exporter kinds are available through both configuration paths. AISIX Cloud API requests and resource-file entries select a kind with the kind value, and the dashboard shows a label for the same underlying kind:

kind Value	Dashboard Label	Use When
otlp_http	OTLP/HTTP	You already collect traces through an OTLP/HTTP collector or vendor endpoint.
object_store	Object storage	You want batched NDJSON request events in Amazon S3, S3-compatible storage, Google Cloud Storage, or Azure Blob.
datadog	Datadog	You use Datadog Logs HTTP intake.
aliyun_sls	Alibaba Cloud SLS	You use Alibaba Cloud Simple Log Service as the log destination.

The gateway sends telemetry directly to the selected destination.

Configure an OTLP Exporter

Set the collector endpoint and authorization header used by the example:

# Replace with your values
export OTLP_ENDPOINT="https://collector.example.com/v1/traces"
export OTLP_AUTH_HEADER="Bearer YOUR_COLLECTOR_TOKEN"

The endpoint must be reachable from the gateway process. A receiver running beside a gateway process on the same host can use http://localhost:4318/v1/traces; a containerized gateway needs the receiver's container-network or host address instead. Remove the headers block from the exporter when the receiver does not require authentication.

AISIX Cloud

Create the exporter and capture its ID:

EXPORTER_ID=$(curl -sS -X POST "$AISIX_CP/environments/$ENV_ID/observability_exporters" \
  -H "Authorization: Bearer $AISIX_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "prod-otlp",
    "kind": "otlp_http",
    "endpoint": "'"${OTLP_ENDPOINT}"'",
    "headers": {
      "Authorization": "'"${OTLP_AUTH_HEADER}"'"
    },
    "sample_rate": 1
  }' | jq -r '.observability_exporter.id')

❶ Set static headers only when the OTLP destination requires them. Header values are encrypted at rest and never returned by read operations.

❷ sample_rate: 1 keeps the delivery check deterministic. It is equivalent to omitting the field, which exports every request trace. After verification, lower the value when you need to reduce span volume.

Retrieve the exporter to confirm the stored configuration:

curl -sS "$AISIX_CP/environments/$ENV_ID/observability_exporters/$EXPORTER_ID" \
  -H "Authorization: Bearer $AISIX_TOKEN"

You should see a response similar to the following. Reads expose the configured header names through header_keys and confirm stored values through headers_set; the header values themselves are never returned.

{
  "observability_exporter": {
    "id": "b46a9f5d-6a4d-4bb1-ae2c-4ab1b22f5e80",
    "env_id": "0f2f6a1e-9d33-4a8f-9a6e-2a7b6a9c1d2e",
    "name": "prod-otlp",
    "enabled": true,
    "kind": "otlp_http",
    "endpoint": "https://collector.example.com/v1/traces",
    "header_keys": ["Authorization"],
    "headers_set": true,
    "sample_rate": 1,
    "created_at": "2026-07-22T08:30:00Z",
    "updated_at": "2026-07-22T08:30:00Z"
  }
}

Keep $EXPORTER_ID for later updates or deletion. Exporters are enabled by default. Set enabled: false to save the resource without sending telemetry.

Open-Source AISIX Gateway

Add this exporter to observability_exporters in the complete resources file. Environment interpolation keeps the collector token out of the file:

resources.yaml (OTLP exporter)

observability_exporters:
  - name: prod-otlp
    kind: otlp_http
    endpoint: ${OTLP_ENDPOINT}
    headers:
      Authorization: ${OTLP_AUTH_HEADER}
    sample_rate: 1

Validate the assembled complete file:

aisix validate --resources resources.yaml

Start or restart the gateway with OTLP_ENDPOINT and OTLP_AUTH_HEADER in its process environment. Reload a running gateway only if those variables are already available to the process. Exporters are enabled by default; add enabled: false to keep the entry without sending telemetry.

Verify OTLP Delivery

A saved exporter confirms only that AISIX accepted its configuration. Verify the OTLP exporter configured above by capturing the request ID that AISIX returns, finding that ID at the destination, and checking the gateway's delivery signal. If you are checking an existing OTLP exporter with sample_rate below 1, temporarily set the rate to 1 so this request cannot be sampled out.

Send one successful request through a working model alias and capture the ID AISIX returns. This remains accurate whether AISIX accepts caller-supplied IDs or generates its own:

export EXPORTER_NAME="prod-otlp"

if TEST_REQUEST_ID=$(
  set -o pipefail
  curl -fsS -D - -o /dev/null \
    -X POST "$AISIX_PROXY/v1/chat/completions" \
    -H "Authorization: Bearer $AISIX_API_KEY" \
    -H "Content-Type: application/json" \
    -d '{
      "model": "'"${MODEL_ALIAS}"'",
      "messages": [{"role": "user", "content": "Reply with: exporter check"}]
    }' \
    | awk 'tolower($1) == "x-aisix-request-id:" { id = $2; sub(/\r$/, "", id) } END { print id }'
) && test -n "$TEST_REQUEST_ID"; then
  export TEST_REQUEST_ID
  printf 'request ID: %s\n' "$TEST_REQUEST_ID"
else
  unset TEST_REQUEST_ID
  printf 'verification request failed or returned no request ID\n' >&2
  false
fi

Wait for the exporter to flush the batch, then search your OTLP backend for a span whose aisix.request_id attribute equals $TEST_REQUEST_ID. Finding the span confirms end-to-end delivery: the gateway loaded the exporter, authenticated to the destination when required, and the destination accepted the batch.

For AISIX Cloud, the exporter row in the environment's Observability view reports whether delivery is healthy and how many batches online gateways have shipped. You can inspect the same per-gateway heartbeat data through the Admin API:

curl -sS "$AISIX_CP/environments/$ENV_ID/dp_nodes" \
  -H "Authorization: Bearer $AISIX_TOKEN" \
  | jq --arg exporter "$EXPORTER_NAME" \
    '[.data[] | select(.status != "offline") | .exporter_health[]? | select(.name == $exporter)]'

After the next heartbeat, at least one online gateway should report delivered_batches greater than zero and last_error: null. The counters reset when a gateway restarts or the exporter configuration changes. A non-null last_error means delivery is currently failing. The historical fields failed_batches and last_failure_unix remain populated after a later success clears last_error.

For an open-source AISIX gateway, confirm success at the destination. If the record does not arrive, check gateway logs for sink delivery failed; retrying or sink delivery dropped.

Understand Delivery Behavior

Every exporter runs its own delivery pipeline: an in-memory queue in front of a worker that batches records and ships them to the destination. The behavior below is fixed and has no configuration.

Records buffer until the batch reaches 100 records or five seconds pass, whichever comes first.

A destination that refuses a batch transiently — a connection failure, a 429, or a 5xx — is retried on a time budget rather than a fixed number of attempts. The delay starts at 200 milliseconds and doubles up to a 30-second ceiling, and retries continue until 300 seconds after that batch's first attempt. When the budget is spent, the batch is dropped and counted as retries_exhausted in aisix_otlp_fanout_drops_total. A Retry-After header on a 429 or 503 replaces the computed delay, capped at the same 30 seconds, so a destination asking for a longer pause is honored rather than hammered. A budget rather than an attempt count is what lets an exporter survive a receiver outage of a given length: an outage shorter than the budget loses nothing.

While a batch is being retried, new records keep arriving. Each exporter queue holds 8,192 records; when it is full the newest record is refused and counted as queue_full, so the records already queued still have their chance at the destination.

caution

An exporter with full content capture enabled owns its records, prompt and response text included. With content_max_bytes at its default of 131,072, a full 8,192-record queue behind a stalled destination can hold hundreds of megabytes of resident memory. Size content_max_bytes against the gateway's memory limit when you enable full capture on a destination that can stall. See Configure Content Capture.

Understand OTLP Trace Structure

AISIX exports one trace hierarchy for each request that produces OTLP telemetry, rather than one unrelated span for each usage event. For a model request that reaches an upstream, the hierarchy is:

Span	OTLP Kind	Scope
Inbound request	SERVER	Covers the request from arrival until the response finishes or the caller disconnects. A valid inbound traceparent becomes this span's remote parent.
Logical operation	CLIENT	Covers the gateway's upstream operation across retries and failover attempts. Child of the SERVER span.
Upstream attempt	CLIENT	Represents one provider dispatch. Each attempt is a child of the logical operation and carries aisix.attempt_index.

Retries and failover therefore appear as sibling attempt spans under one logical operation. For each usage event, AISIX puts the complete attributes and captured content on the most specific span present: the attempt, otherwise the logical operation, otherwise the SERVER span. The other spans carry the correlation fields needed to join the hierarchy without repeating the whole event.

Not every request has all three levels:

• A cache hit, input guardrail block, or other pre-dispatch result has only the SERVER span because AISIX made no upstream call.
• MCP, A2A, Realtime, job, and passthrough calls that do not use per-attempt tracking have the SERVER span and one CLIENT span for the upstream operation.

Use span kind and parentage rather than the span name alone when counting exported traces. Filter for SERVER spans to count sampled request traces. When sample_rate is below 1, requests that were not selected never appear. Some authentication and malformed-input paths reject requests before a usage event or OTLP span exists, so SERVER spans are not a complete inbound-request count. Use request metrics for gateway request volume. Use CLIENT spans carrying aisix.attempt_index when analyzing individual model attempts.

Continue an Inbound W3C Trace

When a request contains exactly one valid traceparent, AISIX continues that trace and makes its SERVER span a child of the caller's span. A malformed value or multiple traceparent headers are ignored, and AISIX starts a local trace instead of rejecting the request. An accompanying tracestate that passes the gateway's character and length checks is recorded on the SERVER span.

AISIX does not forward the caller's traceparent or tracestate to model providers, passthrough targets, MCP servers, or A2A agents unless forward_client_headers names one of them exactly; a wildcard pattern never matches them. By default the context establishes the caller-to-gateway relationship only. See Upstream Request Headers for the forwarding boundary.

Configure Other Exporters

Use another exporter kind when telemetry should go to object storage or a logging service instead of an OTLP trace backend. With AISIX Cloud, send one of the following objects as the body of POST $AISIX_CP/environments/$ENV_ID/observability_exporters.

Object Storage

For object storage, choose the storage provider, bucket, and key prefix. The default authentication mode uses a credential reference resolved by the gateway:

{
  "name": "request-events-s3",
  "kind": "object_store",
  "provider": "s3",
  "bucket": "acme-aisix-events",
  "prefix": "ai-gateway",
  "region": "us-east-1",
  "credential_ref": "acme_s3"
}

Object storage supports Amazon S3, Google Cloud Storage, Azure Blob, and S3-compatible targets. Cloud identity is supported only for S3 or GCS when the gateway has an attached identity that can write to the bucket. Use credential_ref for Azure Blob and for S3-compatible targets that require static credentials.

For an S3-compatible target such as MinIO, Cloudflare R2, or Alibaba Cloud OSS, set the target endpoint explicitly. Without an endpoint, the exporter uses the native AWS S3 endpoint.

Alibaba Cloud SLS

Configure the endpoint host, project, log store, and credential reference:

{
  "name": "request-events-sls",
  "kind": "aliyun_sls",
  "endpoint": "ap-southeast-3.log.aliyuncs.com",
  "project": "acme-observability",
  "logstore": "ai-gateway",
  "credential_ref": "acme_sls"
}

Datadog

Configure the Datadog site, service name, tags, and credential reference:

{
  "name": "request-events-datadog",
  "kind": "datadog",
  "site": "datadoghq.com",
  "service": "ai-gateway",
  "tags": ["team:platform", "tier:prod"],
  "credential_ref": "acme_datadog"
}

For the resources-file path, add these entries to observability_exporters:

resources.yaml (request exporters)

observability_exporters:
  - name: request-events-s3
    kind: object_store
    provider: s3
    bucket: acme-aisix-events
    prefix: ai-gateway
    region: us-east-1
    credential_ref: acme_s3

  - name: request-events-sls
    kind: aliyun_sls
    endpoint: ap-southeast-3.log.aliyuncs.com
    project: acme-observability
    logstore: ai-gateway
    credential_ref: acme_sls

  - name: request-events-datadog
    kind: datadog
    site: datadoghq.com
    service: ai-gateway
    tags: ["team:platform", "tier:prod"]
    credential_ref: acme_datadog

Keep only the destinations you use, then validate the resources file and start or reload the gateway as described for the OTLP example. Verify these exporters with the same request-ID approach used for OTLP: search object-storage and SLS records by request_id, or Datadog logs by aisix.request_id. The Snowflake guide shows how to inspect object-storage output directly.

SLS, Datadog, and object-storage exporters keep destination credentials out of the exporter resource by using credential references or cloud identity. The gateway resolves those credentials locally when it sends telemetry.

Configure Content Capture

Exporters include request status, token counts, model and provider identifiers, request IDs, finish reason, and timing by default. Prompt and response bodies remain excluded unless full content capture is enabled on an OTLP/HTTP, SLS, or Datadog exporter.

Enable Full Content Capture

Add content_mode and content_max_bytes when creating an exporter. In AISIX Cloud, patch an existing exporter with the same fields:

curl -sS -X PATCH "$AISIX_CP/environments/$ENV_ID/observability_exporters/$EXPORTER_ID" \
  -H "Authorization: Bearer $AISIX_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "content_mode": "full",
    "content_max_bytes": 131072
  }'

In the dashboard, set Content mode to Full, then adjust Max content bytes in the exporter form.

For the resources-file path, add the fields to the exporter entry, then validate and reload the file:

observability_exporters:
  - name: prod-otlp
    kind: otlp_http
    endpoint: ${OTLP_ENDPOINT}
    headers:
      Authorization: ${OTLP_AUTH_HEADER}
    content_mode: full
    content_max_bytes: 131072

Use full content capture only when the destination is approved to receive end-user prompt and response text. AISIX applies the configured byte cap independently to captured prompt and response fields.

Understand Truncated Records

When valid JSON exceeds content_max_bytes, AISIX first tries to reduce the value structurally so the exported field remains valid JSON. If the reduced value still cannot fit the configured cap, AISIX falls back to a UTF-8-safe byte cut. The captured prompt is the serialized request body and follows the same behavior.

Content	Truncation Behavior
Long string	Keeps a prefix and adds the inline marker ...[aisix: truncated, N bytes total].
Base64 data URI	Replaces the encoded data with a size placeholder.
Long array	Keeps head and tail samples around an {"_aisix_truncated": true, "omitted_items": N} element that accounts for every omitted item.
Non-JSON content, or JSON that cannot fit after structural reduction	Cuts the value at a UTF-8 character boundary.

When truncation occurs, OTLP records include aisix.content_truncated: true. Datadog and SLS records include content_truncated: true.

Capture Passthrough Content

For successfully relayed passthrough route traffic, full content capture records the request body as a string, subject to the exporter's content cap and JSON-aware structural truncation. Buffered responses record extracted text when the response matches a supported extraction shape and otherwise record the body as text. Streamed responses record accumulated extracted text; opaque data payloads are retained as text.

Capture Failed Requests

Full content capture applies to the supported AI proxy endpoint types and MCP tool calls summarized below. A2A capture records message content rather than JSON-RPC envelopes. Realtime and job and batch telemetry are not captured. Passthrough requests rejected during authentication, authorization, input guardrails, or rate limiting do not include captured content.

On /v1/chat/completions, /v1/messages, and /v1/responses, a failure that produces a usage event after request parsing records the request body in the prompt field, except for 401 and 403 responses. This includes input guardrail blocks (422), upstream failures other than 401 and 403, and validation failures after parsing, such as an empty messages array. When data masking runs, AISIX captures the post-mask request body. Malformed JSON is rejected before a usage event is created and is not captured.

The following boundaries also apply:

• Caller authentication failures are rejected before a usage event is created.
• When every routing target fails, the last attempt's record carries the prompt.
• A response-side guardrail block never captures the blocked output.

These boundaries keep rejected credentials and blocked output out of exported content while retaining the request context needed to investigate other failures.

Review Captured Content by Endpoint

Endpoint Type	Captured Content
Text generation	Response text.
MCP tools/call	params.arguments after traffic-control and input-guardrail checks, with any data masking applied; the JSON-RPC result when present. Output-guardrail blocks capture neither field. Other MCP methods are not captured.
A2A	Text from request and reply message parts on message/send and message/stream. File parts, data parts, and JSON-RPC envelopes are not captured.
Embeddings, rerank, and image generation	Full response JSON.
Audio transcription	Returned transcript. Uploaded audio is not captured; its SHA-256 checksum is recorded alongside the request's text fields.
Text to speech	Binary speech response is not captured.

Model Fields in Exported Telemetry

Usage telemetry records both the model alias requested by the caller and the model that served an attempt when those values differ. This distinction matters for routed and ensemble traffic, where one caller-facing alias can resolve to several target calls.

Destinations render these values in their own telemetry format. OTLP traces use the following fields:

• gen_ai.request.model contains the caller-requested alias.
• gen_ai.response.model contains the concrete response model version when the provider reports one.
• aisix.model_id identifies the resolved model resource.

Gateway-Protocol Spans

An A2A or MCP call is not a model inference, so its span is not encoded as one. A2A exports as invoke_agent <agent> and MCP as execute_tool <tool>, following the OpenTelemetry generative-AI semantic conventions, and carries the protocol's own detail:

• A2A: gen_ai.agent.name, gen_ai.conversation.id (the A2A context), and aisix.a2a.operation, aisix.a2a.method, aisix.a2a.protocol_version, aisix.a2a.task_id, aisix.a2a.task_state, aisix.a2a.stream_event_count.
• MCP: gen_ai.tool.name and aisix.mcp.server_name.
• Passthrough routes: the span exports as passthrough <route> with aisix.passthrough.route_name, and aisix.client_identity carries the end-user identity when the route's identity_header extracted one.

Model traffic keeps the chat.completions span name and gen_ai.operation.name: chat. A trace query written against those values used to match agent and tool calls as well. It now matches model traffic only, which is what it was meant to describe.

Datadog logs use aisix.requested_model for the caller-requested alias, gen_ai.response.model for the concrete response model version, and aisix.model_id for the resolved model resource.

Object storage and Alibaba Cloud SLS retain usage-event field names such as requested_model, model_id, and provider_model_version.

Request Kind in Exported Telemetry

Every record also carries the kind of work the request asked for — a chat completion, an image generation, a video submission, a tool call. See Tell Request Kinds Apart for the values and what they mean.

Destinations name it differently:

• Object storage and Alibaba Cloud SLS keep the usage-event name, operation.
• Datadog maps it to aisix.operation.
• OTLP carries it as the aisix.operation span attribute, on every span of the request's hierarchy so a trace can be filtered by kind at its root.

OTLP spans also carry the OpenTelemetry gen_ai.operation.name attribute, and the two answer different questions. That one uses OpenTelemetry's own vocabulary, which distinguishes model inference from agent and tool calls. It has a single value for every OpenAI-compatible endpoint, so it cannot separate a conversation from an image or a video. Filter on aisix.operation when the endpoint matters.

Token Counts in Exported Telemetry

Exporters receive the token counts the usage event recorded, which keep the accounting the upstream used rather than the usage block the caller read. See Recorded Token Counts.

Destinations report them as follows:

• Object storage and Alibaba Cloud SLS keep the usage-event fields as recorded, including reasoning_tokens and total_tokens. A zero reasoning_tokens and an unreported total_tokens are omitted.
• Datadog logs carry gen_ai.usage.input_tokens and gen_ai.usage.output_tokens, plus aisix.reasoning_tokens and aisix.total_tokens when present.
• OTLP spans carry gen_ai.usage.input_tokens and gen_ai.usage.output_tokens.

In the OpenTelemetry generative-AI convention, output tokens include reasoning. OTLP and Datadog therefore report gen_ai.usage.output_tokens as completion_tokens + reasoning_tokens when the event counts reasoning beside the completion, and as completion_tokens otherwise. The test is the one the control plane prices by: total_tokens and reasoning_tokens are both greater than zero, and total_tokens equals prompt + completion + reasoning tokens but not prompt + completion tokens. See How Reasoning Tokens Are Billed.

The test applies per event, not per provider. A Gemini thinking model on the Vertex AI adapter normally passes it, and so does an OpenAI-compatible upstream that reports reasoning outside completion_tokens with a total that includes it. For Gemini, the exported output count is the same as before; the OpenAI-compatible case exports completion + reasoning where a gateway at 1.4.0 or earlier exported the completion alone.

AISIX Cloud Control Plane

The dashboard manages the same exporter resources from the target environment's Observability view. The exporter form collects the destination fields, content mode, and any kind-specific options.

Whether an exporter is saved through the API or the dashboard, the control plane projects the configuration to the AISIX gateways attached to that environment.

The following behavior applies to exporters projected to AISIX gateways:

• The AISIX gateway sends request telemetry directly to your destination. The control plane does not proxy exported telemetry.
• Prompt and response content stays on the gateway unless you enable full content capture on an exporter.
• Credential references are resolved by the AISIX gateway. When a destination needs runtime credentials, the dashboard shows the environment variables to configure on the gateway.
• Delivery health comes from gateway heartbeat data and shows whether batches are being shipped or whether the gateway is reporting a delivery error.

For OTLP/HTTP exporters, the dashboard provides presets for Langfuse, Honeycomb, and Grafana Cloud Tempo, and accepts custom OTLP endpoints.

The trace UI URL template is optional. Use it when the Request Logs view should link a request record to an external trace UI. The template must include {request_id} so the control plane can replace it with the request ID from the log record.

Next Steps

Continue with Load Request Telemetry into Snowflake to query object-storage telemetry in Snowflake. Use Metrics and Usage Events to compare exported records with gateway metrics, and Access Logs and Request Correlation to trace individual requests.