Export Request Traces to ClickHouse

AISIX gateways export request traces over OTLP/HTTP. You can send those traces directly to a ClickStack ingestion endpoint or run an OpenTelemetry Collector that writes them to an existing ClickHouse database.

Choose an Ingestion Path

Choose the path that matches the ClickHouse service you operate:

Destination	Path
ClickStack	Send traces to the OTLP/HTTP endpoint supplied by ClickStack. Its packaged Collector handles batching, schema management, and ClickHouse writes. Skip to Configure the AISIX Exporter.
Existing ClickHouse database	Deploy the custom Collector configuration in this guide. The Collector receives traces from AISIX and writes them with the ClickHouse exporter.

For ClickStack, use the complete traces endpoint and authentication header shown by your deployment. A ClickStack OTLP/HTTP endpoint normally ends in /v1/traces. Self-managed ClickStack can use an ingestion key or OIDC bearer token, depending on how its packaged Collector is configured. See the ClickStack Collector guidance for the deployment options.

The remaining Collector deployment steps are for an existing ClickHouse database that does not already expose a suitable OTLP endpoint:

Prerequisites

Before deploying the custom Collector path, prepare the following:

• An AISIX Enterprise or AISIX Cloud environment with an attached gateway, or an AISIX OSS deployment that loads a declarative resources.yaml file.
• An existing ClickHouse database and a write account. Decide whether the Collector may create its database and tables.
• A working request through the gateway for verification.
• Network access from the gateway to the Collector and from the Collector to ClickHouse.
• Docker Compose or access to the Kubernetes namespace where the gateway runs.

This guide pins OpenTelemetry Collector Contrib 0.163.2. The ClickHouse exporter is included in that distribution, and its v0.163.0 component reference classifies traces support as beta. Review its stability and configuration before using it for production telemetry.

Configure the Collector

The complete configuration below is for a new dedicated Collector. If you already operate a compatible Collector, merge these component definitions and pipeline references into its configuration without replacing unrelated components, then use its normal rollout process.

Create a working directory:

mkdir aisix-clickhouse
cd aisix-clickhouse

Create the following complete Collector configuration and replace the uppercase placeholders:

collector.yaml

extensions:
  bearertokenauth:
    token: 'COLLECTOR_TOKEN'

receivers:
  otlp:
    protocols:
      http:
        endpoint: '0.0.0.0:4318'
        auth:
          authenticator: bearertokenauth

exporters:
  clickhouse:
    endpoint: 'tcp://CLICKHOUSE_HOST:9000?dial_timeout=10s'
    username: 'CLICKHOUSE_USER'
    password: 'CLICKHOUSE_PASSWORD'
    database: 'TARGET_DATABASE'
    traces_table_name: 'TARGET_TRACES_TABLE'
    create_schema: true
    sending_queue:
      num_consumers: 10
      batch:
        min_size: 5000
        flush_timeout: 5s

service:
  extensions: [bearertokenauth]
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [clickhouse]

Replace the placeholders as follows:

Placeholder or field	Value
COLLECTOR_TOKEN	A long random token. For example, generate one with openssl rand -hex 32. Use the same value in the AISIX exporter header.
CLICKHOUSE_HOST and port 9000	The ClickHouse native endpoint reachable from the Collector. Use the port and TLS settings supplied by your database operator.
CLICKHOUSE_USER and CLICKHOUSE_PASSWORD	A ClickHouse account that can insert traces and, when schema creation is enabled, create the target database and tables.
TARGET_DATABASE	The database that stores the trace table, such as aisix.
TARGET_TRACES_TABLE	The trace table name, such as otel_traces.
create_schema	Keep true to let the exporter create a compatible database and table. Set it to false only after your database operator creates a schema compatible with this exporter version.

The endpoint in this example uses the ClickHouse native protocol without TLS. Configure the exporter's TLS settings when traffic crosses an untrusted network or your database requires TLS. Keep credentials out of source control.

The internal sending_queue.batch block follows the ClickHouse exporter's current batching recommendation. It flushes a partial batch after five seconds, so low-volume verification does not wait for 5,000 rows. Do not add the standalone batch processor for this pipeline.

The queue is in memory and can lose pending traces if the Collector restarts. For restart durability, configure sending_queue.storage with a storage extension, add that extension to service.extensions, and mount its directory on durable storage.

Deploy the Collector

Both examples run the Collector beside the gateway and make the OTLP/HTTP receiver available as otel-collector:4318.

Docker Compose

Find the gateway container and its Docker networks:

docker ps --format 'table {{.Names}}\t{{.Image}}'
docker inspect GATEWAY_CONTAINER --format '{{json .NetworkSettings.Networks}}'

Replace GATEWAY_CONTAINER with the container name returned by docker ps. Select a user-defined network attached to the gateway. Create the following file beside collector.yaml, then replace GATEWAY_NETWORK with that network name:

compose.yaml

services:
  otel-collector:
    image: ghcr.io/open-telemetry/opentelemetry-collector-releases/opentelemetry-collector-contrib:0.163.2
    command: ['--config=/etc/otelcol-contrib/config.yaml']
    volumes:
      - ./collector.yaml:/etc/otelcol-contrib/config.yaml:ro
    networks: [gateway]
    restart: unless-stopped

networks:
  gateway:
    external: true
    name: GATEWAY_NETWORK

Validate the configuration, start the Collector, and inspect its logs:

docker compose run --rm otel-collector validate --config=/etc/otelcol-contrib/config.yaml
docker compose up -d
docker compose logs --tail=100 otel-collector

The gateway can now reach the receiver at http://otel-collector:4318/v1/traces. Ensure the selected network also lets the Collector reach ClickHouse.

Kubernetes

Deploy the Collector in the gateway namespace so the gateway can resolve the short Service name. Replace every GATEWAY_NAMESPACE value below with that namespace.

Create a Secret from collector.yaml:

kubectl -n GATEWAY_NAMESPACE create secret generic otel-collector-config \
  --from-file=collector.yaml=./collector.yaml

This Secret contains the ClickHouse password and Collector token. Restrict access with your cluster's normal Secret and RBAC controls.

Create the Collector Deployment:

collector-deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: otel-collector
  namespace: GATEWAY_NAMESPACE
spec:
  replicas: 1
  selector:
    matchLabels:
      app: otel-collector
  template:
    metadata:
      labels:
        app: otel-collector
    spec:
      containers:
        - name: collector
          image: ghcr.io/open-telemetry/opentelemetry-collector-releases/opentelemetry-collector-contrib:0.163.2
          args: ['--config=/etc/otelcol-contrib/collector.yaml']
          ports:
            - name: otlp-http
              containerPort: 4318
          volumeMounts:
            - name: config
              mountPath: /etc/otelcol-contrib
              readOnly: true
      volumes:
        - name: config
          secret:
            secretName: otel-collector-config

Apply the Deployment and expose its receiver:

kubectl apply -f collector-deployment.yaml
kubectl -n GATEWAY_NAMESPACE expose deployment otel-collector \
  --name=otel-collector --port=4318 --target-port=4318
kubectl -n GATEWAY_NAMESPACE rollout status deployment/otel-collector
kubectl -n GATEWAY_NAMESPACE logs deployment/otel-collector --tail=100

The gateway can now reach the receiver at http://otel-collector:4318/v1/traces. Allow gateway-to-Collector and Collector-to-ClickHouse traffic in the applicable NetworkPolicies.

If you deploy the Collector in another namespace, use its namespace-qualified Service address. For the OpenTelemetry Collector Helm chart, merge these component settings into its config value rather than the AISIX chart values.

These deployment examples use one Collector replica and do not set resource limits. Size the Collector, queue, and availability model for your trace volume before using this path in production.

Configure the AISIX Exporter

Export the gateway URL and request values used for verification:

# AISIX_PROXY has no trailing slash or endpoint path
# The local quickstarts use http://127.0.0.1:3000
export AISIX_PROXY="YOUR_AISIX_GATEWAY_URL"
export AISIX_API_KEY="YOUR_CALLER_API_KEY"
export MODEL_ALIAS="YOUR_MODEL_ALIAS"

For the custom Collector deployment above, export the receiver URL and authentication header:

export OTLP_ENDPOINT="http://otel-collector:4318/v1/traces"
export OTLP_AUTH_HEADER="Bearer YOUR_COLLECTOR_TOKEN"

For ClickStack, replace these values with its complete OTLP/HTTP traces endpoint and the exact authorization value required by its ingestion setup.

The gateway accepts http:// and https:// exporter endpoints on any host. Gateways 1.7.0 and earlier accept http:// only for the otel-collector host and the loopback hosts localhost and 127.0.0.1. With http://, the authorization header and exported telemetry cross the network in plaintext. Use it only on a network you trust, especially when full content capture is enabled.

Follow Configure an OTLP Exporter to create the exporter through AISIX Enterprise, AISIX Cloud, or AISIX OSS. Keep sample_rate at 1 until delivery is verified. A saved exporter confirms only that AISIX accepted the configuration.

Verify Delivery

Send a successful request through an existing model alias and save the response headers:

curl -sS -D aisix-response.headers -o /dev/null \
  -X POST "$AISIX_PROXY/v1/chat/completions" \
  -H "Authorization: Bearer $AISIX_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "'"${MODEL_ALIAS}"'",
    "messages": [{"role": "user", "content": "Reply with: ClickHouse trace check"}]
  }'

export TEST_REQUEST_ID=$(awk 'tolower($1) == "x-aisix-request-id:" {id=$2; sub(/\r$/, "", id)} END {print id}' aisix-response.headers)
printf 'request ID: %s\n' "$TEST_REQUEST_ID"

If the ID is empty, confirm that the request passed through AISIX and succeeded before checking the export path.

For ClickStack, search its trace view for this request ID. For the custom Collector path, wait at least five seconds for the configured partial batch to flush, then query the target table:

SELECT
  Timestamp,
  SpanName,
  SpanKind,
  TraceId,
  SpanId,
  ParentSpanId,
  SpanAttributes['aisix.request_id'] AS request_id,
  SpanAttributes['http.response.status_code'] AS status
FROM aisix.otel_traces
WHERE Timestamp >= now() - INTERVAL 15 MINUTE
  AND SpanAttributes['aisix.request_id'] = 'YOUR_RETURNED_REQUEST_ID'
ORDER BY Timestamp ASC;

Replace the request ID, database, and table names with your values. A typical model request that reaches an upstream produces a SERVER span, a logical CLIENT span, and an attempt CLIENT span. Other outcomes can produce fewer spans, so finding the request ID is the delivery check; a fixed row count is not.

Troubleshoot Delivery

For ClickStack, check its ingestion status or packaged Collector logs, then confirm that the AISIX endpoint and authorization header match the values ClickStack provides.

For the custom Collector path, inspect both the gateway and Collector logs before changing the configuration:

Docker Compose

Replace GATEWAY_CONTAINER with the gateway container name used during deployment:

docker compose ps
docker compose logs --since=10m otel-collector
docker logs --since=10m GATEWAY_CONTAINER

Kubernetes

Replace GATEWAY_NAMESPACE with the gateway namespace and AISIX_POD with the gateway Pod name:

kubectl -n GATEWAY_NAMESPACE get pods,svc,endpointslices
kubectl -n GATEWAY_NAMESPACE logs deployment/otel-collector --since=10m
kubectl -n GATEWAY_NAMESPACE logs AISIX_POD --since=10m

Symptom	Check
Receiver returns 401 or 403	The Collector token, the AISIX Authorization header, and the space after Bearer.
DNS or connection failure	Service names, namespaces, Docker networks, NetworkPolicies, and the ClickHouse host and port.
Database authentication, schema, or insert error	ClickHouse credentials, permissions, TLS settings, and schema compatibility with the pinned exporter.
Exporter is configured but no current rows appear	The target gateway, sample rate, request ID, batching delay, and query time range.
Repeated Collector restarts	The pinned image, unresolved placeholders, configuration syntax, resource limits, and workload logs.

For AISIX Enterprise or AISIX Cloud, the exporter row in the environment's Observability view also shows delivery health reported by online gateways. For AISIX OSS, use the gateway logs and the destination result.

Stop Exporting

Disable or remove the AISIX exporter through the same configuration path used to create it. For ClickStack, no change to its packaged Collector is required. For the custom Collector path, keep a shared Collector running for its other pipelines. If the ClickHouse pipeline is no longer needed, remove only its receiver and exporter references through your normal Collector rollout process; existing database records are unaffected.

Next Steps

See Observability Exporters for sampling, content capture, and gateway delivery behavior. Use OTLP Trace Reference for trace structure and attributes, and the ClickHouse exporter reference for schema, retention, queue, and database options.