Version: latest

Distributed Tracing

Distributed tracing gives you end-to-end visibility into a request's journey across API7 Gateway and your backend services. API7 Gateway uses the opentelemetry plugin to generate spans and export them over OTLP/HTTP to any compatible collector — Jaeger, Tempo, the OpenTelemetry Collector, Honeycomb, Datadog, New Relic, AWS X-Ray, and others.

This guide walks through a complete end-to-end setup. For the full list of plugin parameters and metadata fields, see the opentelemetry plugin reference.

How it works

The opentelemetry plugin has two layers of configuration:

Plugin metadata is shared by every route in a gateway group. It controls the collector endpoint, request timeout, custom headers, batch span processor, and trace-ID source. It is set once per gateway group via /apisix/admin/plugin_metadata/opentelemetry.
Per-route plugin config is set on a route, service, or global rule. It controls the sampling strategy and any extra span attributes for the matched traffic.

You must configure the plugin metadata at least once per gateway group before any route starts producing useful spans.

Prerequisites

A running API7 Gateway deployment with at least one online data plane.
An OTLP/HTTP-compatible collector reachable from the data plane (the example below uses Jaeger's built-in OTLP receiver on port 4318).

Step 1: Configure the collector endpoint

Set the collector address in the plugin metadata. Tune the batch span processor and request timeout for production traffic — the default endpoint of 127.0.0.1:4318 and 3-second timeout are rarely what you want.

Admin API
ADC

curl -k "https://localhost:7443/apisix/admin/plugin_metadata/opentelemetry?gateway_group_id={group_id}" -X PUT \
  -H "X-API-KEY: ${API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "trace_id_source": "x-request-id",
    "resource": {
      "service.name": "api7-gateway",
      "deployment.environment": "production"
    },
    "collector": {
      "address": "jaeger:4318",
      "request_timeout": 3
    },
    "set_ngx_var": true
  }'

adc.yaml
plugin_metadata:
  opentelemetry:
    trace_id_source: x-request-id
    resource:
      service.name: api7-gateway
      deployment.environment: production
    collector:
      address: "jaeger:4318"
      request_timeout: 3
    set_ngx_var: true

adc sync -f adc.yaml

set_ngx_var: true exposes the trace ID and span ID as NGINX variables ($opentelemetry_trace_id, $opentelemetry_span_id), which is needed for trace–log correlation.

If your collector requires authentication, add it under collector.request_headers. For all available metadata fields and their defaults, see the plugin metadata reference.

Step 2: Enable tracing on a route

Enable the plugin and pick a sampling strategy. The default sampler is always_off, so you must set one explicitly to start capturing traces. The example below applies the plugin globally so every route in the gateway group is covered:

Admin API
ADC

# Create the service with the upstream
curl -k "https://localhost:7443/apisix/admin/services/tracing-demo?gateway_group_id={group_id}" -X PUT \
  -H "X-API-KEY: ${API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "tracing-demo",
    "upstream": {
      "type": "roundrobin",
      "scheme": "http",
      "nodes": [
        { "host": "httpbin.org", "port": 80, "weight": 1 }
      ]
    }
  }'

# Create the route under that service
curl -k "https://localhost:7443/apisix/admin/routes/tracing-demo?gateway_group_id={group_id}" -X PUT \
  -H "X-API-KEY: ${API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "tracing-demo",
    "service_id": "tracing-demo",
    "paths": ["/anything"]
  }'

# Enable tracing for every request via a global rule
curl -k "https://localhost:7443/apisix/admin/global_rules/opentelemetry?gateway_group_id={group_id}" -X PUT \
  -H "X-API-KEY: ${API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "plugins": {
      "opentelemetry": {
        "sampler": {
          "name": "trace_id_ratio",
          "options": { "fraction": 1.0 }
        },
        "additional_attributes": ["consumer_name", "request_uri"]
      }
    }
  }'

adc.yaml
services:
  - name: tracing-demo
    routes:
      - name: tracing-demo
        uris:
          - /anything
    upstream:
      type: roundrobin
      nodes:
        - host: httpbin.org
          port: 80
          weight: 1

global_rules:
  opentelemetry:
    sampler:
      name: trace_id_ratio
      options:
        fraction: 1.0
    additional_attributes:
      - consumer_name
      - request_uri

adc sync -f adc.yaml

For all per-route fields (samplers, additional attributes, header attributes), see the plugin parameters reference. Values in additional_attributes are APISIX variable names such as consumer_name, request_uri, or route_name. The plugin already attaches common OpenTelemetry semantic attributes (http.route, http.status_code, apisix.route_name, etc.) automatically.

Choose a sampling strategy

Strategy	When to use
`always_on`	Development, staging, or low-volume routes where you want every request traced.
`trace_id_ratio`	Production. Pick a fraction such as `0.01` (1%) to keep collector load and storage cost bounded. The decision is deterministic across services that share the same trace ID.
`parent_base`	When upstream services or a service mesh already make sampling decisions; the gateway should respect them and only fall back to a root sampler for unsampled root requests.
`always_off`	The default. Use to keep the plugin attached but temporarily stop emitting spans.

A common production pattern is parent_base with a low-fraction trace_id_ratio root sampler, so cross-service traces stay intact while keeping volume low for unsampled root traffic:

{
  "sampler": {
    "name": "parent_base",
    "options": {
      "root": {
        "name": "trace_id_ratio",
        "options": { "fraction": 0.01 }
      }
    }
  }
}

Step 3: Verify

Send a request through the gateway:

curl -i "http://127.0.0.1:9080/anything"

Open the collector's UI (for Jaeger, http://127.0.0.1:16686) and search for the service api7-gateway. You should see a span named after the request method and path (for example, GET /anything), with attributes such as http.status_code, apisix.route_name=tracing-demo, and request_uri.

Correlate traces with access logs

When set_ngx_var: true is in the plugin metadata (as in Step 1), include the trace and span IDs in the access log format so each log line links back to the corresponding span:

conf/config.yaml
nginx_config:
  http:
    access_log_format: '{"time": "$time_iso8601","trace_id": "$opentelemetry_trace_id","span_id": "$opentelemetry_span_id","remote_addr": "$remote_addr","status": "$status"}'
    access_log_format_escape: json

The same variables can be referenced from logger plugins such as http-logger or kafka-logger inside their log_format field.

Next steps

Monitor Metrics — correlate trace data with Prometheus metrics and latency dashboards.
Centralized Logging — ship trace-correlated access logs to your log backend.
opentelemetry plugin reference — full list of parameters and plugin metadata fields.

How it works​

Prerequisites​

Step 1: Configure the collector endpoint​

Step 2: Enable tracing on a route​

Choose a sampling strategy​

Step 3: Verify​

Correlate traces with access logs​

Next steps​