Observability Exporters

Observability exporters send usage events from the gateway to destinations used for tracing, logging, storage, or accounting. This guide configures an OTLP/HTTP exporter using AISIX Cloud or the open-source AISIX gateway, then explains destination choices, content capture, and delivery behavior.

Prerequisites

Before starting, prepare the following:

• One of these configuration paths:
  • AISIX Cloud with an environment, an attached gateway, and a write-scoped admin token. For On-Premises, follow the AISIX Cloud Quickstart. To request Hybrid Cloud access, contact API7.
  • An open-source AISIX gateway that loads a declarative resources.yaml file.
• A telemetry destination for the exporter you plan to use.
• A working model alias and caller API key for delivery verification.

Export the gateway and request values used for verification:

# AISIX_PROXY has no trailing slash or endpoint path
# The local quickstarts use http://127.0.0.1:3000
export AISIX_PROXY="YOUR_AISIX_GATEWAY_URL"
export AISIX_API_KEY="YOUR_CALLER_API_KEY"
export MODEL_ALIAS="YOUR_MODEL_ALIAS"

For the AISIX Cloud examples, export the Admin API base URL, admin token, and environment ID:

# AISIX_CP includes /api and has no trailing slash
# The local On-Premises quickstart uses http://localhost:8080/api
export AISIX_CP="YOUR_AISIX_CLOUD_ADMIN_API_URL"
export AISIX_TOKEN="YOUR_ADMIN_TOKEN"
export ENV_ID="YOUR_ENVIRONMENT_ID"

In other deployments, use the reachable AISIX Cloud Admin API URL.

Choose an Exporter Kind

All exporter kinds are available through both configuration paths. AISIX Cloud API requests and resource-file entries select a kind with the kind value, and the dashboard shows a label for the same underlying kind:

kind Value	Dashboard Label	Use When
otlp_http	OTLP/HTTP	You already collect traces through an OTLP/HTTP collector or vendor endpoint.
object_store	Object storage	You want batched NDJSON request events in Amazon S3, S3-compatible storage, Google Cloud Storage, or Azure Blob.
datadog	Datadog	You use Datadog Logs HTTP intake.
aliyun_sls	Alibaba Cloud SLS	You use Alibaba Cloud Simple Log Service as the log destination.

The gateway sends telemetry directly to the selected destination.

Configure an OTLP Exporter

Set the collector endpoint and authorization header used by the example:

# Replace with your values
export OTLP_ENDPOINT="https://collector.example.com/v1/traces"
export OTLP_AUTH_HEADER="Bearer YOUR_COLLECTOR_TOKEN"

The endpoint must be reachable from the gateway process. A receiver running beside a gateway process on the same host can use http://localhost:4318/v1/traces; a containerized gateway needs the receiver's container-network or host address instead. Remove the headers block from the exporter when the receiver does not require authentication.

AISIX Cloud

Create the exporter and capture its ID:

EXPORTER_ID=$(curl -sS -X POST "$AISIX_CP/environments/$ENV_ID/observability_exporters" \
  -H "Authorization: Bearer $AISIX_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "prod-otlp",
    "kind": "otlp_http",
    "endpoint": "'"${OTLP_ENDPOINT}"'",
    "headers": {
      "Authorization": "'"${OTLP_AUTH_HEADER}"'"
    },
    "sample_rate": 1
  }' | jq -r '.observability_exporter.id')

❶ Set static headers only when the OTLP destination requires them. Header values are encrypted at rest and never returned by read operations.

❷ sample_rate: 1 keeps the delivery check deterministic. It is equivalent to omitting the field, which exports every request trace. After verification, lower the value when you need to reduce span volume.

Retrieve the exporter to confirm the stored configuration:

curl -sS "$AISIX_CP/environments/$ENV_ID/observability_exporters/$EXPORTER_ID" \
  -H "Authorization: Bearer $AISIX_TOKEN"

You should see a response similar to the following. Reads expose the configured header names through header_keys and confirm stored values through headers_set; the header values themselves are never returned.

{
  "observability_exporter": {
    "id": "b46a9f5d-6a4d-4bb1-ae2c-4ab1b22f5e80",
    "env_id": "0f2f6a1e-9d33-4a8f-9a6e-2a7b6a9c1d2e",
    "name": "prod-otlp",
    "enabled": true,
    "kind": "otlp_http",
    "endpoint": "https://collector.example.com/v1/traces",
    "header_keys": ["Authorization"],
    "headers_set": true,
    "sample_rate": 1,
    "created_at": "2026-07-22T08:30:00Z",
    "updated_at": "2026-07-22T08:30:00Z"
  }
}

Keep $EXPORTER_ID for later updates or deletion. Exporters are enabled by default. Set enabled: false to save the resource without sending telemetry. Disabling or deleting an exporter stops its delivery pipeline on each gateway once the configuration change arrives: events already queued are delivered first, and the exporter drops out of the gateway's delivery health report. Re-enabling it starts a fresh pipeline with its counters reset.

Open-Source AISIX Gateway

Add this exporter to observability_exporters in the complete resources file. Environment interpolation keeps the collector token out of the file:

resources.yaml (OTLP exporter)

observability_exporters:
  - name: prod-otlp
    kind: otlp_http
    endpoint: ${OTLP_ENDPOINT}
    headers:
      Authorization: ${OTLP_AUTH_HEADER}
    sample_rate: 1

Validate the assembled complete file:

aisix validate --resources resources.yaml

Start or restart the gateway with OTLP_ENDPOINT and OTLP_AUTH_HEADER in its process environment. Reload a running gateway only if those variables are already available to the process. Exporters are enabled by default; add enabled: false to keep the entry without sending telemetry.

Verify OTLP Delivery

A saved exporter confirms only that AISIX accepted its configuration. Verify the OTLP exporter configured above by capturing the request ID that AISIX returns, finding that ID at the destination, and checking the gateway's delivery signal. If you are checking an existing OTLP exporter with sample_rate below 1, temporarily set the rate to 1 so this request cannot be sampled out.

Send one successful request through a working model alias and capture the ID AISIX returns. This remains accurate whether AISIX accepts caller-supplied IDs or generates its own:

export EXPORTER_NAME="prod-otlp"

if TEST_REQUEST_ID=$(
  set -o pipefail
  curl -fsS -D - -o /dev/null \
    -X POST "$AISIX_PROXY/v1/chat/completions" \
    -H "Authorization: Bearer $AISIX_API_KEY" \
    -H "Content-Type: application/json" \
    -d '{
      "model": "'"${MODEL_ALIAS}"'",
      "messages": [{"role": "user", "content": "Reply with: exporter check"}]
    }' \
    | awk 'tolower($1) == "x-aisix-request-id:" { id = $2; sub(/\r$/, "", id) } END { print id }'
) && test -n "$TEST_REQUEST_ID"; then
  export TEST_REQUEST_ID
  printf 'request ID: %s\n' "$TEST_REQUEST_ID"
else
  unset TEST_REQUEST_ID
  printf 'verification request failed or returned no request ID\n' >&2
  false
fi

Wait for the exporter to flush the batch, then search your OTLP backend for a span whose aisix.request_id attribute equals $TEST_REQUEST_ID. Finding the span confirms end-to-end delivery: the gateway loaded the exporter, authenticated to the destination when required, and the destination accepted the batch.

For AISIX Cloud, the exporter row in the environment's Observability view reports whether delivery is healthy and how many batches online gateways have shipped. You can inspect the same per-gateway heartbeat data through the Admin API:

curl -sS "$AISIX_CP/environments/$ENV_ID/dp_nodes" \
  -H "Authorization: Bearer $AISIX_TOKEN" \
  | jq --arg exporter "$EXPORTER_NAME" \
    '[.data[] | select(.status != "offline") | .exporter_health[]? | select(.name == $exporter)]'

After the next heartbeat, at least one online gateway should report delivered_batches greater than zero and last_error: null. The counters reset when a gateway restarts or the exporter configuration changes. A non-null last_error means delivery is currently failing. The historical fields failed_batches and last_failure_unix remain populated after a later success clears last_error.

For an open-source AISIX gateway, confirm success at the destination. If the record does not arrive, check gateway logs for sink delivery failed; retrying or sink delivery dropped.

Understand Delivery Behavior

Every exporter runs its own delivery pipeline: an in-memory queue in front of a worker that batches records and ships them to the destination. The behavior below is fixed and has no configuration.

Records buffer until the batch reaches 100 records or one second passes, whichever comes first.

A destination that refuses a batch transiently — a connection failure, a 429, or a 5xx — is retried on a time budget rather than a fixed number of attempts. The delay starts at 200 milliseconds and doubles up to a 30-second ceiling, and retries continue until 300 seconds after that batch's first attempt. When the budget is spent, the batch is dropped and counted as retries_exhausted in aisix_otlp_fanout_drops_total. A Retry-After header on a 429 or 503 replaces the computed delay, capped at the same 30 seconds, so a destination asking for a longer pause is honored rather than hammered. A budget rather than an attempt count is what lets an exporter survive a receiver outage of a given length: an outage shorter than the budget loses nothing.

While a batch is being retried, new records keep arriving. Each exporter queue holds 8,192 records; when it is full the newest record is refused and counted as queue_full, so the records already queued still have their chance at the destination.

caution

An exporter with full content capture enabled owns its records, prompt and response text included. With content_max_bytes at its default of 131,072, a full 8,192-record queue behind a stalled destination can hold hundreds of megabytes of resident memory. Size content_max_bytes against the gateway's memory limit when you enable full capture on a destination that can stall. See Configure Content Capture.

Understand OTLP Trace Structure

AISIX exports one trace hierarchy for each request that produces OTLP telemetry, rather than one unrelated span for each usage event. For a model request that reaches an upstream, the hierarchy is:

Span	OTLP Kind	Scope
Inbound request	SERVER	Covers the request from arrival until the response finishes or the caller disconnects. A valid inbound traceparent becomes this span's remote parent.
Logical operation	CLIENT	Covers the gateway's upstream operation across retries and failover attempts. Child of the SERVER span.
Upstream attempt	CLIENT	Represents one provider dispatch. Each attempt is a child of the logical operation and carries aisix.attempt_index.

Retries and failover therefore appear as sibling attempt spans under one logical operation. For each usage event, AISIX puts the complete attributes and captured content on the most specific span present: the attempt, otherwise the logical operation, otherwise the SERVER span. The other spans carry the correlation fields needed to join the hierarchy without repeating the whole event.

Not every request has all three levels:

• A cache hit, input guardrail block, or other pre-dispatch result has only the SERVER span because AISIX made no upstream call.
• MCP, A2A, Realtime, job, and passthrough calls that do not use per-attempt tracking have the SERVER span and one CLIENT span for the upstream operation.

Use span kind and parentage rather than the span name alone when counting exported traces. Filter for SERVER spans to count sampled request traces. When sample_rate is below 1, requests that were not selected never appear. Some authentication and malformed-input paths reject requests before a usage event or OTLP span exists, so SERVER spans are not a complete inbound-request count. Use request metrics for gateway request volume. Use CLIENT spans carrying aisix.attempt_index when analyzing individual model attempts.

Continue an Inbound W3C Trace

When a request contains exactly one valid traceparent, AISIX continues that trace and makes its SERVER span a child of the caller's span. A malformed value or multiple traceparent headers are ignored, and AISIX starts a local trace instead of rejecting the request. An accompanying tracestate that passes the gateway's character and length checks is recorded on the SERVER span.

AISIX does not forward the caller's traceparent or tracestate to model providers, passthrough targets, MCP servers, or A2A agents unless forward_client_headers names one of them exactly; a wildcard pattern never matches them. By default the context establishes the caller-to-gateway relationship only. See Upstream Request Headers for the forwarding boundary.

OTLP Span Attribute Reference

OTLP spans follow the OpenTelemetry generative-AI and HTTP semantic conventions as released in semantic-conventions v1.41.1. Attributes the conventions do not define use the aisix. prefix.

Changes from 1.6.0 and earlier

Trace queries, dashboards, and alerts written against a gateway at 1.6.0 or earlier need updating. The exporter configuration does not change.

• gen_ai.system is removed. Use gen_ai.provider.name.
• gen_ai.prompt and gen_ai.completion are removed from OTLP spans. Captured content uses the attributes in Captured Content. Datadog logs keep both fields.
• gen_ai.request.model is the model name sent to the upstream. The alias or Model Group name the caller requested moves to aisix.requested_model.
• gen_ai.usage.input_tokens includes cached input tokens. For usage reported in the Anthropic shape, which keeps cache reads and cache writes beside the input tokens, the value now adds both to the input tokens. The Datadog exporter's gen_ai.usage.input_tokens changes the same way. See Token Counts in Exported Telemetry.
• Span names change. chat.completions is gone, the SERVER span is named after the HTTP route, such as POST /v1/chat/completions, and MCP spans are named tools/call <tool> instead of execute_tool <tool>.
• A successful span leaves its status unset instead of setting OK. A 4xx response no longer marks the SERVER span as an error.
• aisix.client_source_ip and aisix.client_user_agent are renamed to client.address and user_agent.original. They and the http.request.header.<name> attributes move from the operation span to the SERVER span.
• The SERVER span and a logical span that groups attempts no longer carry gen_ai.* attributes. Only the GenAI operation span does, so a backend that turns GenAI spans into model calls counts each call once.
• The resource attribute service.name is the gateway's observability.service_name, which defaults to aisix, instead of the fixed value aisix-dp.

Span Names and Status

Each span in the trace hierarchy has one role. The GenAI operation span is the most specific span present: the attempt span, otherwise the logical span, otherwise the SERVER span.

Span	Name	Error status
SERVER	{method} {route}, such as POST /v1/chat/completions. The route is added only when the request path is itself one of the gateway's fixed routes, so a passthrough path or /v1/videos/{id} is named by the method alone. A nonstandard method is named HTTP in place of the method.	ERROR only for a 5xx status. A 4xx or a 499 leaves the status unset.
GenAI operation	{gen_ai.operation.name} {gen_ai.request.model}, such as chat gpt-4o. MCP spans are tools/call {tool}, A2A spans invoke_agent {agent}, and passthrough spans passthrough {route}. The name is the operation alone when the target is empty or longer than 64 bytes, or when a wildcard model served the request, because the caller chose that model name.	ERROR for any status of 400 or higher except 499, because a caller that closed the request cancelled it deliberately.
Logical span that groups attempts	Same as the GenAI operation span.	Same as the GenAI operation span.

A span with ERROR status also carries error.type, set to the gateway's error class (aisix.error_class) or, when there is none, the status code. A successful span leaves its status unset.

The SERVER span is kind SERVER. The logical and attempt spans are kind CLIENT. A request with only a SERVER span, such as a cache hit, uses that SERVER span as its GenAI operation span.

GenAI Operation Span Attributes

The GenAI operation span carries the usage event:

Attribute	Value
gen_ai.operation.name	chat for /v1/chat/completions, /v1/messages, and /v1/responses; text_completion for /v1/completions; embeddings for /v1/embeddings; invoke_agent for A2A; and execute_tool for MCP. Every other request kind reports its aisix.operation value, such as passthrough.
aisix.operation	The gateway's own request kind, which separates chat, messages, and responses.
gen_ai.provider.name	The provider of the dispatched upstream. Provider keys with the Amazon Bedrock, Google Vertex AI, or Azure OpenAI adapter report aws.bedrock, gcp.vertex_ai, or azure.ai.openai. Other provider keys report openai, anthropic, gcp.gemini, deepseek, x_ai, mistral_ai, groq, cohere, perplexity, or ibm.watsonx.ai for those providers, and any other provider's ID verbatim in lowercase. A passthrough span reports the provider of the route's provider key, and an A2A span reports a2a. MCP spans omit it.
gen_ai.request.model	The model name sent to the upstream. For a wildcard model, it is the concrete name substituted into the model's template.
aisix.requested_model	The model alias or Model Group name the caller requested.
server.address and server.port	The host and port of the base URL the request was sent to. Amazon Bedrock, Google Vertex AI, and Azure OpenAI provider keys report it only when they set api_base. A passthrough span reports the route's target_url host and omits it for a preserve_host route. MCP and A2A spans report the MCP server or agent URL. A request that never reached an upstream, such as a cache hit or a refusal, omits it.
gen_ai.response.model	The concrete model version the provider reported.
gen_ai.response.id	The provider's response ID.
gen_ai.response.finish_reasons	A single-element array with the finish reason.
gen_ai.response.time_to_first_chunk	For a streamed response, the seconds from this attempt's dispatch to the first upstream frame.
http.response.status_code	The status the usage event recorded.
error.type	Set on failure, as described above.

gen_ai.provider.name, gen_ai.request.model, and server.* are omitted when the gateway cannot name the upstream, such as a request that failed before a routing target was selected.

Protocol-specific attributes are also added:

• MCP: mcp.method.name set to tools/call, gen_ai.tool.name, and aisix.mcp.server_name.
• A2A: gen_ai.agent.name, gen_ai.conversation.id (the A2A context), and aisix.a2a.operation, aisix.a2a.method, aisix.a2a.protocol_version, aisix.a2a.task_id, aisix.a2a.task_state, and aisix.a2a.stream_event_count.
• Passthrough routes: aisix.passthrough.route_name, and aisix.client_identity when the route's identity_header extracted an end-user identity.

The span also keeps the gateway's own correlation and attribution attributes when they have values: aisix.request_id, aisix.exporter_name, aisix.api_key_id, aisix.user_id, aisix.model_id, aisix.attempt_index, aisix.attempt_kind, aisix.attempt_model, aisix.error_class, aisix.error_message, aisix.upstream_ttft_ms, aisix.downstream_latency_ms, aisix.audio.duration_seconds, aisix.gateway_embedding_calls, aisix.gateway_embedding_calls_dropped, aisix.jwt_subject, aisix.jwt_provider, and aisix.jwt_claim_mapping.

Request Parameters

On /v1/chat/completions, /v1/messages, /v1/responses, and /v1/completions, the GenAI operation span records the parameters the caller sent, read from the request body in its own protocol. Every attempt of one request carries the same values.

Attribute	Source
gen_ai.request.max_tokens	max_completion_tokens or max_tokens; max_output_tokens on /v1/responses.
gen_ai.request.temperature and gen_ai.request.top_p	temperature and top_p.
gen_ai.request.top_k	top_k on chat completions and messages.
gen_ai.request.stop_sequences	stop on chat completions and completions, or stop_sequences on messages.
gen_ai.request.frequency_penalty, gen_ai.request.presence_penalty, and gen_ai.request.seed	The same-named fields on chat completions and completions.
gen_ai.request.choice.count	n on chat completions and completions, only when it is not 1.
gen_ai.request.stream	true, only when the request streamed.
gen_ai.output.type	json for a json_object or json_schema format and text for a text format, from response_format.type on chat completions or text.format.type on responses. Omitted when the request names no format.
openai.request.service_tier	service_tier, except auto, on spans whose gen_ai.provider.name is openai.

On /v1/embeddings, the span records gen_ai.request.encoding_formats from encoding_format and gen_ai.embeddings.dimension.count from dimensions.

Token Usage Attributes

Every GenAI operation span except MCP carries usage:

Attribute	Value
gen_ai.usage.input_tokens	All input tokens, including cached ones: prompt tokens plus cache creation and cache read tokens that the upstream reported beside them. Always present.
gen_ai.usage.cache_read.input_tokens	Input tokens served from the provider's prompt cache, in either accounting shape. Present when greater than zero.
gen_ai.usage.cache_creation.input_tokens	Input tokens written to the provider's prompt cache: Anthropic's cache creation count, or the OpenAI-shape cache_write_tokens. Present when the cache creation count is greater than zero or the upstream reported cache_write_tokens.
gen_ai.usage.output_tokens	Output tokens, including reasoning. Omitted on embeddings spans.
gen_ai.usage.reasoning.output_tokens	Reasoning tokens. Present when greater than zero.

See Token Counts in Exported Telemetry for how these values relate to the recorded usage event.

Captured Content

With content_mode: full, the GenAI operation span carries the captured content. For /v1/chat/completions, /v1/messages, /v1/responses, and /v1/completions, a message exchange becomes the conventions' structured attributes, each a JSON string in the conventions' message schema:

• gen_ai.input.messages holds the request messages. A completions prompt is converted only when it is a single string.
• gen_ai.system_instructions holds the Anthropic system field or the Responses instructions field. Chat completions system messages stay in gen_ai.input.messages.
• gen_ai.tool.definitions holds the name and type of each declared tool.
• gen_ai.output.messages holds one assistant message with the captured response text and the finish reason.

MCP tool calls use gen_ai.tool.call.arguments and, for a successful call, gen_ai.tool.call.result.

Each structured attribute is held to the exporter's content_max_bytes, and the prompt and response sides are judged separately. Content that is not a message exchange, such as embeddings input, images, audio, jobs, and agent text, stays raw under aisix.content.prompt and aisix.content.response. So does a side that capture truncated, that does not convert, or that would exceed the cap once converted, so a gen_ai.* content attribute is never partial. aisix.content_truncated: true marks a record whose content was truncated.

SERVER Span Attributes

The SERVER span describes the inbound HTTP request:

Attribute	Value
http.request.method	The request method. A nonstandard method reports _OTHER, with the method itself in http.request.method_original.
url.path	The request path, up to 512 bytes.
url.scheme	https on a TLS listener, otherwise http.
http.route	The route, only when the path is itself one of the gateway's fixed routes.
url.query	The query string, up to 1,024 bytes, when the request had one.
http.response.status_code	The final response status.
client.address	The resolved caller address.
user_agent.original	The caller's User-Agent.
http.request.header.<name>	The headers listed in observability.usage_event.request_headers. See Request Headers in Exported Telemetry.
aisix.request_id, aisix.exporter_name, and aisix.operation	The keys that join the request's spans.

url.query replaces the value of a sensitive parameter with REDACTED. The parameter names AWSAccessKeyId, Signature, sig, and X-Goog-Signature match case-sensitively, and key, access_token, token, client_secret, password, api_key, and apikey match case-insensitively. Names are compared after percent-decoding. Every other value is exported verbatim, so do not send other secrets in query strings to a gateway whose OTLP destination should not receive them.

A logical span that groups attempts carries only aisix.request_id, aisix.exporter_name, and aisix.operation.

Resource Attributes

Each export request carries these resource attributes:

• service.name: the gateway's observability.service_name from its startup configuration, which defaults to aisix. A gateway at 1.6.0 or earlier always reports aisix-dp.
• service.version: the gateway build version.

The instrumentation scope is aisix-obs.otlp_http_sink, with the gateway build version.

AISIX does not export attributes whose data the gateway does not retain, including openai.response.service_tier, openai.response.system_fingerprint, gen_ai.agent.id, mcp.session.id, and jsonrpc.request.id.

Configure Other Exporters

Use another exporter kind when telemetry should go to object storage or a logging service instead of an OTLP trace backend. With AISIX Cloud, send one of the following objects as the body of POST $AISIX_CP/environments/$ENV_ID/observability_exporters.

Object Storage

For object storage, choose the storage provider, bucket, and key prefix. The default authentication mode uses a credential reference resolved by the gateway:

{
  "name": "request-events-s3",
  "kind": "object_store",
  "provider": "s3",
  "bucket": "acme-aisix-events",
  "prefix": "ai-gateway",
  "region": "us-east-1",
  "credential_ref": "acme_s3"
}

Object storage supports Amazon S3, Google Cloud Storage, Azure Blob, and S3-compatible targets. Cloud identity is supported only for S3 or GCS when the gateway has an attached identity that can write to the bucket. Use credential_ref for Azure Blob and for S3-compatible targets that require static credentials.

For an S3-compatible target such as MinIO, Cloudflare R2, or Alibaba Cloud OSS, set the target endpoint explicitly. Without an endpoint, the exporter uses the native AWS S3 endpoint.

endpoint overrides the storage provider's native endpoint, and its meaning follows provider:

provider	What endpoint sets
s3	The S3-compatible host to upload to.
gcs	The Cloud Storage XML API base URL, such as a private endpoint. Uploads use the XML API, so the endpoint must serve it; an emulator that accepts only JSON API uploads, such as the stock fake-gcs-server image, cannot receive objects. It takes precedence over a gcs_base_url in the service-account JSON.
azure_blob	The Blob service endpoint.

The value must be an https:// URL, except for loopback and test hosts. endpoint cannot be combined with cloud identity: AISIX Cloud rejects the combination, and an open-source gateway that loads it never uploads and reports a permanent delivery error. The AISIX Cloud console offers the field for S3-compatible targets, Cloud Storage, and Azure Blob, and hides it when cloud identity is selected.

A gcs endpoint needs gateways at 1.5.0 or later, because earlier gateways ignore it and upload to Google's public Cloud Storage endpoint instead. AISIX Cloud therefore refuses to save a gcs exporter with an endpoint with 422 DP_INCOMPATIBLE while any gateway registered in the environment is older than 1.5.0.

Alibaba Cloud SLS

Configure the endpoint host, project, log store, and credential reference:

{
  "name": "request-events-sls",
  "kind": "aliyun_sls",
  "endpoint": "ap-southeast-3.log.aliyuncs.com",
  "project": "acme-observability",
  "logstore": "ai-gateway",
  "credential_ref": "acme_sls"
}

Datadog

Configure the Datadog site, service name, tags, and credential reference:

{
  "name": "request-events-datadog",
  "kind": "datadog",
  "site": "datadoghq.com",
  "service": "ai-gateway",
  "tags": ["team:platform", "tier:prod"],
  "credential_ref": "acme_datadog"
}

For the resources-file path, add these entries to observability_exporters:

resources.yaml (request exporters)

observability_exporters:
  - name: request-events-s3
    kind: object_store
    provider: s3
    bucket: acme-aisix-events
    prefix: ai-gateway
    region: us-east-1
    credential_ref: acme_s3

  - name: request-events-sls
    kind: aliyun_sls
    endpoint: ap-southeast-3.log.aliyuncs.com
    project: acme-observability
    logstore: ai-gateway
    credential_ref: acme_sls

  - name: request-events-datadog
    kind: datadog
    site: datadoghq.com
    service: ai-gateway
    tags: ["team:platform", "tier:prod"]
    credential_ref: acme_datadog

Keep only the destinations you use, then validate the resources file and start or reload the gateway as described for the OTLP example. Verify these exporters with the same request-ID approach used for OTLP: search object-storage and SLS records by request_id, or Datadog logs by aisix.request_id. The Snowflake guide shows how to inspect object-storage output directly.

SLS, Datadog, and object-storage exporters keep destination credentials out of the exporter resource by using credential references or cloud identity. The gateway resolves those credentials locally when it sends telemetry.

Configure Content Capture

Exporters include request status, token counts, model and provider identifiers, request IDs, finish reason, and timing by default. Prompt and response bodies remain excluded unless full content capture is enabled on an OTLP/HTTP, SLS, or Datadog exporter.

Enable Full Content Capture

Add content_mode and content_max_bytes when creating an exporter. In AISIX Cloud, patch an existing exporter with the same fields:

curl -sS -X PATCH "$AISIX_CP/environments/$ENV_ID/observability_exporters/$EXPORTER_ID" \
  -H "Authorization: Bearer $AISIX_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "content_mode": "full",
    "content_max_bytes": 131072
  }'

In the dashboard, set Content mode to Full, then adjust Max content bytes in the exporter form.

For the resources-file path, add the fields to the exporter entry, then validate and reload the file:

observability_exporters:
  - name: prod-otlp
    kind: otlp_http
    endpoint: ${OTLP_ENDPOINT}
    headers:
      Authorization: ${OTLP_AUTH_HEADER}
    content_mode: full
    content_max_bytes: 131072

Use full content capture only when the destination is approved to receive end-user prompt and response text. AISIX applies the configured byte cap independently to captured prompt and response fields.

Understand Truncated Records

When valid JSON exceeds content_max_bytes, AISIX first tries to reduce the value structurally so the exported field remains valid JSON. If the reduced value still cannot fit the configured cap, AISIX falls back to a UTF-8-safe byte cut. The captured prompt is the serialized request body and follows the same behavior.

Content	Truncation Behavior
Long string	Keeps a prefix and adds the inline marker ...[aisix: truncated, N bytes total].
Base64 data URI	Replaces the encoded data with a size placeholder.
Long array	Keeps head and tail samples around an {"_aisix_truncated": true, "omitted_items": N} element that accounts for every omitted item.
Non-JSON content, or JSON that cannot fit after structural reduction	Cuts the value at a UTF-8 character boundary.

When truncation occurs, OTLP records include aisix.content_truncated: true. Datadog and SLS records include content_truncated: true.

Capture Passthrough Content

For successfully relayed passthrough route traffic, full content capture records the request body as a string, subject to the exporter's content cap and JSON-aware structural truncation. Buffered responses record extracted text when the response matches a supported extraction shape and otherwise record the body as text. Streamed responses record accumulated extracted text; opaque data payloads are retained as text.

Capture Failed Requests

Full content capture applies to the supported AI proxy endpoint types and MCP tool calls summarized below. A2A capture records message content rather than JSON-RPC envelopes. Realtime and job and batch telemetry are not captured. Passthrough requests rejected during authentication, authorization, input guardrails, or rate limiting do not include captured content.

On /v1/chat/completions, /v1/messages, and /v1/responses, a failure that produces a usage event after request parsing records the request body in the prompt field, except for 401 and 403 responses. This includes input guardrail blocks (422), upstream failures other than 401 and 403, and validation failures after parsing, such as an empty messages array. When data masking runs, AISIX captures the post-mask request body. Malformed JSON is rejected before a usage event is created and is not captured.

The following boundaries also apply:

• Caller authentication failures are rejected before a usage event is created.
• When every routing target fails, the last attempt's record carries the prompt.
• A response-side guardrail block never captures the blocked output.

These boundaries keep rejected credentials and blocked output out of exported content while retaining the request context needed to investigate other failures.

Review Captured Content by Endpoint

Endpoint Type	Captured Content
Text generation	Response text.
MCP tools/call	params.arguments after traffic-control and input-guardrail checks, with any data masking applied; the JSON-RPC result when present. Output-guardrail blocks capture neither field. Other MCP methods are not captured.
A2A	Text from request and reply message parts on message/send and message/stream. File parts, data parts, and JSON-RPC envelopes are not captured.
Embeddings, rerank, and image generation	Full response JSON.
Audio transcription	Returned transcript. Uploaded audio is not captured; its SHA-256 checksum is recorded alongside the request's text fields.
Text to speech	Binary speech response is not captured.

Model Fields in Exported Telemetry

Usage telemetry records both the model alias requested by the caller and the model that served an attempt when those values differ. This distinction matters for routed and ensemble traffic, where one caller-facing alias can resolve to several target calls.

Destinations render these values in their own telemetry format. OTLP traces use the following fields:

• aisix.requested_model contains the caller-requested alias or Model Group name.
• gen_ai.request.model contains the model name sent to the upstream for that attempt.
• gen_ai.response.model contains the concrete response model version when the provider reports one.
• aisix.model_id identifies the resolved model resource.

A gateway at 1.6.0 or earlier puts the caller-requested alias in gen_ai.request.model.

Gateway-Protocol Spans

An A2A or MCP call is not a model inference, so its span is not encoded as one. Following the OpenTelemetry generative-AI semantic conventions, each protocol has its own span name and operation, and carries the protocol's own detail listed in GenAI Operation Span Attributes:

• A2A exports as invoke_agent <agent> with gen_ai.operation.name: invoke_agent.
• MCP exports as tools/call <tool> with gen_ai.operation.name: execute_tool and mcp.method.name: tools/call. A gateway at 1.6.0 or earlier names it execute_tool <tool>.
• Passthrough routes export as passthrough <route> with gen_ai.operation.name: passthrough.

Model traffic is named after the operation and the upstream model, such as chat gpt-4o, so a trace query on gen_ai.operation.name: chat matches model traffic only. A gateway at 1.6.0 or earlier names every model span chat.completions.

Datadog logs use aisix.requested_model for the caller-requested alias, gen_ai.response.model for the concrete response model version, and aisix.model_id for the resolved model resource.

Object storage and Alibaba Cloud SLS retain usage-event field names such as requested_model, model_id, and provider_model_version.

Request Kind in Exported Telemetry

Every record also carries the kind of work the request asked for — a chat completion, an image generation, a video submission, a tool call. See Tell Request Kinds Apart for the values and what they mean.

Destinations name it differently:

• Object storage and Alibaba Cloud SLS keep the usage-event name, operation.
• Datadog maps it to aisix.operation.
• OTLP carries it as the aisix.operation span attribute, on every span of the request's hierarchy so a trace can be filtered by kind at its root.

OTLP spans also carry the OpenTelemetry gen_ai.operation.name attribute, and the two answer different questions. That one uses OpenTelemetry's own vocabulary where it has a value, which distinguishes model inference from agent and tool calls, and the aisix.operation value otherwise. It reports chat for chat completions, messages, and responses alike, so it cannot separate those three. Filter on aisix.operation when the endpoint matters.

Request Headers in Exported Telemetry

A gateway configured with observability.usage_event.request_headers records the values of those inbound request headers on every usage event, so callers that share one API key can be attributed by headers they send. Recording does not depend on content_mode. See Attribute Events with Request Headers for the configuration and value rules.

Destinations name the values differently:

• Object storage keeps a nested request_headers object in each record.
• Alibaba Cloud SLS carries one request_headers field holding the object as a compact JSON string.
• Datadog nests the object under aisix.request_headers, so each header is aisix.request_headers.<name>.
• OTLP carries one attribute per header on the SERVER span, http.request.header.<name>, as a single-element string array. A gateway at 1.6.0 or earlier puts it on the operation span instead.

Token Counts in Exported Telemetry

Exporters receive the token counts the usage event recorded, which keep the accounting the upstream used rather than the usage block the caller read. See Recorded Token Counts.

Destinations report them as follows:

• Object storage and Alibaba Cloud SLS keep the usage-event fields as recorded, including reasoning_tokens and total_tokens. A zero reasoning_tokens and an unreported total_tokens are omitted.
• Datadog logs carry gen_ai.usage.input_tokens and gen_ai.usage.output_tokens, plus the recorded aisix.cache_creation_tokens, aisix.cache_read_tokens, aisix.reasoning_tokens, and aisix.total_tokens when present.
• OTLP spans carry gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, gen_ai.usage.cache_read.input_tokens, gen_ai.usage.cache_creation.input_tokens, and gen_ai.usage.reasoning.output_tokens. See Token Usage Attributes.

In the OpenTelemetry generative-AI convention, input tokens include every cached token. OTLP and Datadog therefore report gen_ai.usage.input_tokens as prompt_tokens + cache_creation_tokens + cache_read_tokens. The two cache fields are non-zero only for usage reported in the Anthropic shape, such as Anthropic and Amazon Bedrock, which counts cache reads and writes beside the input tokens. An OpenAI-shape upstream already counts its cached tokens inside prompt_tokens, so its exported input count is unchanged. For example, an Anthropic reply with 19 input tokens, 1,200 cache read tokens, and 300 cache creation tokens exports gen_ai.usage.input_tokens as 1,519. A gateway at 1.6.0 or earlier exports 19 for the same reply.

In the OpenTelemetry generative-AI convention, output tokens include reasoning. OTLP and Datadog therefore report gen_ai.usage.output_tokens as completion_tokens + reasoning_tokens when the event counts reasoning beside the completion, and as completion_tokens otherwise. The test is the one the control plane prices by: total_tokens and reasoning_tokens are both greater than zero, and total_tokens equals prompt + completion + reasoning tokens but not prompt + completion tokens. See How Reasoning Tokens Are Billed.

The test applies per event, not per provider. A Gemini thinking model on the Vertex AI adapter normally passes it, and so does an OpenAI-compatible upstream that reports reasoning outside completion_tokens with a total that includes it. For Gemini, the exported output count is the same as before; the OpenAI-compatible case exports completion + reasoning where a gateway at 1.4.0 or earlier exported the completion alone.

AISIX Cloud Control Plane

The dashboard manages the same exporter resources from the target environment's Observability view. The exporter form collects the destination fields, content mode, and any kind-specific options.

Whether an exporter is saved through the API or the dashboard, the control plane projects the configuration to the AISIX gateways attached to that environment.

The following behavior applies to exporters projected to AISIX gateways:

• The AISIX gateway sends request telemetry directly to your destination. The control plane does not proxy exported telemetry.
• Prompt and response content stays on the gateway unless you enable full content capture on an exporter.
• Credential references are resolved by the AISIX gateway. When a destination needs runtime credentials, the dashboard shows the environment variables to configure on the gateway.
• Delivery health comes from gateway heartbeat data and shows whether batches are being shipped or whether the gateway is reporting a delivery error.

For OTLP/HTTP exporters, the dashboard provides presets for Langfuse, Honeycomb, and Grafana Cloud Tempo, and accepts custom OTLP endpoints.

The trace UI URL template is optional. Use it when the Request Logs view should link a request record to an external trace UI. The template must include {request_id} so the control plane can replace it with the request ID from the log record.

Next Steps

Continue with Load Request Telemetry into Snowflake to query object-storage telemetry in Snowflake. Use Metrics and Usage Events to compare exported records with gateway metrics, and Access Logs and Request Correlation to trace individual requests.