# Export Request Traces to ClickHouse AISIX gateways export request traces over OTLP/HTTP. You can send those traces directly to a ClickStack ingestion endpoint or run an OpenTelemetry Collector that writes them to an existing ClickHouse database. ## Choose an Ingestion Path[​](#choose-an-ingestion-path "Direct link to Choose an Ingestion Path") Choose the path that matches the ClickHouse service you operate: | Destination | Path | | ---------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | ClickStack | Send traces to the OTLP/HTTP endpoint supplied by ClickStack. Its packaged Collector handles batching, schema management, and ClickHouse writes. Skip to [Configure the AISIX Exporter](#configure-the-aisix-exporter). | | Existing ClickHouse database | Deploy the custom Collector configuration in this guide. The Collector receives traces from AISIX and writes them with the ClickHouse exporter. | For ClickStack, use the complete traces endpoint and authentication header shown by your deployment. A ClickStack OTLP/HTTP endpoint normally ends in `/v1/traces`. Self-managed ClickStack can use an ingestion key or [OIDC bearer token](https://clickhouse.com/blog/whats-new-in-clickstack-august-2026), depending on how its packaged Collector is configured. See the [ClickStack Collector guidance](https://clickhouse.com/blog/whats-new-in-clickstack-january-2026) for the deployment options. The remaining Collector deployment steps are for an existing ClickHouse database that does not already expose a suitable OTLP endpoint: ## Prerequisites[​](#prerequisites "Direct link to Prerequisites") Before deploying the custom Collector path, prepare the following: * An AISIX Enterprise or AISIX Cloud environment with an attached gateway, or an AISIX OSS deployment that loads a declarative [`resources.yaml`](https://docs.api7.ai/ai-gateway/reference/resources-file.md) file. * An existing ClickHouse database and a write account. Decide whether the Collector may create its database and tables. * A working request through the gateway for verification. * Network access from the gateway to the Collector and from the Collector to ClickHouse. * Docker Compose or access to the Kubernetes namespace where the gateway runs. This guide pins OpenTelemetry Collector Contrib 0.163.2. The ClickHouse exporter is included in that distribution, and its [v0.163.0 component reference](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/v0.163.0/exporter/clickhouseexporter/README.md) classifies traces support as beta. Review its stability and configuration before using it for production telemetry. ## Configure the Collector[​](#configure-the-collector "Direct link to Configure the Collector") The complete configuration below is for a new dedicated Collector. If you already operate a compatible Collector, merge these component definitions and pipeline references into its configuration without replacing unrelated components, then use its normal rollout process. Create a working directory: ``` mkdir aisix-clickhouse cd aisix-clickhouse ``` Create the following complete Collector configuration and replace the uppercase placeholders: collector.yaml ``` extensions: bearertokenauth: token: 'COLLECTOR_TOKEN' receivers: otlp: protocols: http: endpoint: '0.0.0.0:4318' auth: authenticator: bearertokenauth exporters: clickhouse: endpoint: 'tcp://CLICKHOUSE_HOST:9000?dial_timeout=10s' username: 'CLICKHOUSE_USER' password: 'CLICKHOUSE_PASSWORD' database: 'TARGET_DATABASE' traces_table_name: 'TARGET_TRACES_TABLE' create_schema: true sending_queue: num_consumers: 10 batch: min_size: 5000 flush_timeout: 5s service: extensions: [bearertokenauth] pipelines: traces: receivers: [otlp] exporters: [clickhouse] ``` Replace the placeholders as follows: | Placeholder or field | Value | | ------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `COLLECTOR_TOKEN` | A long random token. For example, generate one with `openssl rand -hex 32`. Use the same value in the AISIX exporter header. | | `CLICKHOUSE_HOST` and port `9000` | The ClickHouse native endpoint reachable from the Collector. Use the port and TLS settings supplied by your database operator. | | `CLICKHOUSE_USER` and `CLICKHOUSE_PASSWORD` | A ClickHouse account that can insert traces and, when schema creation is enabled, create the target database and tables. | | `TARGET_DATABASE` | The database that stores the trace table, such as `aisix`. | | `TARGET_TRACES_TABLE` | The trace table name, such as `otel_traces`. | | `create_schema` | Keep `true` to let the exporter create a compatible database and table. Set it to `false` only after your database operator creates a schema compatible with this exporter version. | The endpoint in this example uses the ClickHouse native protocol without TLS. Configure the [exporter's TLS settings](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/v0.163.0/exporter/clickhouseexporter/README.md) when traffic crosses an untrusted network or your database requires TLS. Keep credentials out of source control. The internal `sending_queue.batch` block follows the ClickHouse exporter's current batching recommendation. It flushes a partial batch after five seconds, so low-volume verification does not wait for 5,000 rows. Do not add the standalone `batch` processor for this pipeline. The queue is in memory and can lose pending traces if the Collector restarts. For restart durability, configure `sending_queue.storage` with a [storage extension](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/v0.163.0/extension/storage/filestorage), add that extension to `service.extensions`, and mount its directory on durable storage. ## Deploy the Collector[​](#deploy-the-collector "Direct link to Deploy the Collector") Both examples run the Collector beside the gateway and make the OTLP/HTTP receiver available as `otel-collector:4318`. * Docker Compose * Kubernetes Find the gateway container and its Docker networks: ``` docker ps --format 'table {{.Names}}\t{{.Image}}' docker inspect GATEWAY_CONTAINER --format '{{json .NetworkSettings.Networks}}' ``` Replace `GATEWAY_CONTAINER` with the container name returned by `docker ps`. Select a user-defined network attached to the gateway. Create the following file beside `collector.yaml`, then replace `GATEWAY_NETWORK` with that network name: compose.yaml ``` services: otel-collector: image: ghcr.io/open-telemetry/opentelemetry-collector-releases/opentelemetry-collector-contrib:0.163.2 command: ['--config=/etc/otelcol-contrib/config.yaml'] volumes: - ./collector.yaml:/etc/otelcol-contrib/config.yaml:ro networks: [gateway] restart: unless-stopped networks: gateway: external: true name: GATEWAY_NETWORK ``` Validate the configuration, start the Collector, and inspect its logs: ``` docker compose run --rm otel-collector validate --config=/etc/otelcol-contrib/config.yaml docker compose up -d docker compose logs --tail=100 otel-collector ``` The gateway can now reach the receiver at `http://otel-collector:4318/v1/traces`. Ensure the selected network also lets the Collector reach ClickHouse. Deploy the Collector in the gateway namespace so the gateway can resolve the short Service name. Replace every `GATEWAY_NAMESPACE` value below with that namespace. Create a Secret from `collector.yaml`: ``` kubectl -n GATEWAY_NAMESPACE create secret generic otel-collector-config \ --from-file=collector.yaml=./collector.yaml ``` This Secret contains the ClickHouse password and Collector token. Restrict access with your cluster's normal Secret and RBAC controls. Create the Collector Deployment: collector-deployment.yaml ``` apiVersion: apps/v1 kind: Deployment metadata: name: otel-collector namespace: GATEWAY_NAMESPACE spec: replicas: 1 selector: matchLabels: app: otel-collector template: metadata: labels: app: otel-collector spec: containers: - name: collector image: ghcr.io/open-telemetry/opentelemetry-collector-releases/opentelemetry-collector-contrib:0.163.2 args: ['--config=/etc/otelcol-contrib/collector.yaml'] ports: - name: otlp-http containerPort: 4318 volumeMounts: - name: config mountPath: /etc/otelcol-contrib readOnly: true volumes: - name: config secret: secretName: otel-collector-config ``` Apply the Deployment and expose its receiver: ``` kubectl apply -f collector-deployment.yaml kubectl -n GATEWAY_NAMESPACE expose deployment otel-collector \ --name=otel-collector --port=4318 --target-port=4318 kubectl -n GATEWAY_NAMESPACE rollout status deployment/otel-collector kubectl -n GATEWAY_NAMESPACE logs deployment/otel-collector --tail=100 ``` The gateway can now reach the receiver at `http://otel-collector:4318/v1/traces`. Allow gateway-to-Collector and Collector-to-ClickHouse traffic in the applicable NetworkPolicies. If you deploy the Collector in another namespace, use its namespace-qualified Service address. For the OpenTelemetry Collector Helm chart, merge these component settings into its `config` value rather than the AISIX chart values. These deployment examples use one Collector replica and do not set resource limits. Size the Collector, queue, and availability model for your trace volume before using this path in production. ## Configure the AISIX Exporter[​](#configure-the-aisix-exporter "Direct link to Configure the AISIX Exporter") Export the gateway URL and request values used for verification: ``` # AISIX_PROXY has no trailing slash or endpoint path # The local quickstarts use http://127.0.0.1:3000 export AISIX_PROXY="YOUR_AISIX_GATEWAY_URL" export AISIX_API_KEY="YOUR_CALLER_API_KEY" export MODEL_ALIAS="YOUR_MODEL_ALIAS" ``` For the custom Collector deployment above, export the receiver URL and authentication header: ``` export OTLP_ENDPOINT="http://otel-collector:4318/v1/traces" export OTLP_AUTH_HEADER="Bearer YOUR_COLLECTOR_TOKEN" ``` For ClickStack, replace these values with its complete OTLP/HTTP traces endpoint and the exact authorization value required by its ingestion setup. The gateway accepts `http://` and `https://` exporter endpoints on any host. Gateways 1.7.0 and earlier accept `http://` only for the `otel-collector` host and the loopback hosts `localhost` and `127.0.0.1`. With `http://`, the authorization header and exported telemetry cross the network in plaintext. Use it only on a network you trust, especially when full content capture is enabled. Follow [Configure an OTLP Exporter](https://docs.api7.ai/ai-gateway/observability/exporters.md#configure-an-otlp-exporter) to create the exporter through AISIX Enterprise, AISIX Cloud, or AISIX OSS. Keep `sample_rate` at `1` until delivery is verified. A saved exporter confirms only that AISIX accepted the configuration. ## Verify Delivery[​](#verify-delivery "Direct link to Verify Delivery") Send a successful request through an existing model alias and save the response headers: ``` curl -sS -D aisix-response.headers -o /dev/null \ -X POST "$AISIX_PROXY/v1/chat/completions" \ -H "Authorization: Bearer $AISIX_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "model": "'"${MODEL_ALIAS}"'", "messages": [{"role": "user", "content": "Reply with: ClickHouse trace check"}] }' export TEST_REQUEST_ID=$(awk 'tolower($1) == "x-aisix-request-id:" {id=$2; sub(/\r$/, "", id)} END {print id}' aisix-response.headers) printf 'request ID: %s\n' "$TEST_REQUEST_ID" ``` If the ID is empty, confirm that the request passed through AISIX and succeeded before checking the export path. For ClickStack, search its trace view for this request ID. For the custom Collector path, wait at least five seconds for the configured partial batch to flush, then query the target table: ``` SELECT Timestamp, SpanName, SpanKind, TraceId, SpanId, ParentSpanId, SpanAttributes['aisix.request_id'] AS request_id, SpanAttributes['http.response.status_code'] AS status FROM aisix.otel_traces WHERE Timestamp >= now() - INTERVAL 15 MINUTE AND SpanAttributes['aisix.request_id'] = 'YOUR_RETURNED_REQUEST_ID' ORDER BY Timestamp ASC; ``` Replace the request ID, database, and table names with your values. A typical model request that reaches an upstream produces a SERVER span, a logical CLIENT span, and an attempt CLIENT span. Other outcomes can produce fewer spans, so finding the request ID is the delivery check; a fixed row count is not. ## Troubleshoot Delivery[​](#troubleshoot-delivery "Direct link to Troubleshoot Delivery") For ClickStack, check its ingestion status or packaged Collector logs, then confirm that the AISIX endpoint and authorization header match the values ClickStack provides. For the custom Collector path, inspect both the gateway and Collector logs before changing the configuration: * Docker Compose * Kubernetes Replace `GATEWAY_CONTAINER` with the gateway container name used during deployment: ``` docker compose ps docker compose logs --since=10m otel-collector docker logs --since=10m GATEWAY_CONTAINER ``` Replace `GATEWAY_NAMESPACE` with the gateway namespace and `AISIX_POD` with the gateway Pod name: ``` kubectl -n GATEWAY_NAMESPACE get pods,svc,endpointslices kubectl -n GATEWAY_NAMESPACE logs deployment/otel-collector --since=10m kubectl -n GATEWAY_NAMESPACE logs AISIX_POD --since=10m ``` | Symptom | Check | | ------------------------------------------------- | ----------------------------------------------------------------------------------------------------- | | Receiver returns `401` or `403` | The Collector token, the AISIX `Authorization` header, and the space after `Bearer`. | | DNS or connection failure | Service names, namespaces, Docker networks, NetworkPolicies, and the ClickHouse host and port. | | Database authentication, schema, or insert error | ClickHouse credentials, permissions, TLS settings, and schema compatibility with the pinned exporter. | | Exporter is configured but no current rows appear | The target gateway, sample rate, request ID, batching delay, and query time range. | | Repeated Collector restarts | The pinned image, unresolved placeholders, configuration syntax, resource limits, and workload logs. | For AISIX Enterprise or AISIX Cloud, the exporter row in the environment's Observability view also shows delivery health reported by online gateways. For AISIX OSS, use the gateway logs and the destination result. ## Stop Exporting[​](#stop-exporting "Direct link to Stop Exporting") Disable or remove the AISIX exporter through the same configuration path used to create it. For ClickStack, no change to its packaged Collector is required. For the custom Collector path, keep a shared Collector running for its other pipelines. If the ClickHouse pipeline is no longer needed, remove only its receiver and exporter references through your normal Collector rollout process; existing database records are unaffected. ## Next Steps[​](#next-steps "Direct link to Next Steps") See [Observability Exporters](https://docs.api7.ai/ai-gateway/observability/exporters.md) for sampling, content capture, and gateway delivery behavior. Use [OTLP Trace Reference](https://docs.api7.ai/ai-gateway/reference/otlp-traces.md) for trace structure and attributes, and the [ClickHouse exporter reference](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/v0.163.0/exporter/clickhouseexporter/README.md) for schema, retention, queue, and database options.