Skip to main content
API7.ai Logo
Version: 3.10.x

Configure Distributed Tracing

Distributed tracing shows how requests move through API7 Gateway and backend services, helping you locate latency and failures. API7 Gateway uses the opentelemetry plugin to create spans and export them to an OTLP/HTTP endpoint, such as an OpenTelemetry Collector or a compatible observability backend.

This guide configures a local Jaeger instance for evaluation, enables tracing globally, and correlates the exported traces with access logs.

How It Works​

The opentelemetry plugin uses two configuration layers:

  • Plugin metadata configures the collector endpoint, request timeout, resource attributes, custom headers, batch span processor, and trace ID source for a gateway group.
  • Plugin configuration selects a sampling strategy and adds span attributes. You can apply the plugin to a route, service, or global rule.

Plugin metadata must be configured before the plugin can create and export spans.

Prerequisites​

  • A running API7 Gateway deployment with at least one online data plane.
  • Docker to run Jaeger for this local evaluation.
  • A Dashboard token assigned to the API_KEY environment variable.
  • If you use ADC, install and configure ADC, including the ADC_BACKEND, ADC_SERVER, ADC_TOKEN, and ADC_GATEWAY_GROUP environment variables.

Start Jaeger​

This local example assumes that the data plane runs in Docker. For another deployment type, use an OTLP/HTTP endpoint reachable from the data plane, skip the Docker commands in this section, and substitute its address when configuring plugin metadata.

Create a Docker network and connect the data plane container to it. Replace <data-plane-container> with the container name:

docker network create gateway-tracing-net
docker network connect gateway-tracing-net <data-plane-container>

Start a pinned Jaeger instance on the same network. The image accepts OTLP traces on port 4318 and provides the Jaeger UI on port 16686:

docker run -d --name jaeger \
  --network gateway-tracing-net \
  -p 127.0.0.1:16686:16686 \
  cr.jaegertracing.io/jaegertracing/jaeger:2.20.0

Wait for the Jaeger UI to become available:

until curl -fsS "http://127.0.0.1:16686/" > /dev/null; do
  sleep 1
done
Local evaluation setup

This Jaeger instance uses transient in-memory storage without authentication or TLS. For production, use a secured collector or observability backend with durable storage. If a suitable OTLP/HTTP endpoint is already available, skip this section and substitute its address below.

Configure the Collector Endpoint​

Configure plugin metadata to send spans to Jaeger and expose the trace and span IDs as NGINX variables. The variables are used later to correlate traces with access logs.

curl -k "https://localhost:7443/apisix/admin/plugin_metadata/opentelemetry?gateway_group_id={gateway_group_id}" -X PUT \
  -H "X-API-KEY: ${API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "resource": {
      "service.name": "api7-gateway",
      "deployment.environment": "development"
    },
    "collector": {
      "address": "jaeger:4318",
      "request_timeout": 3
    },
    "set_ngx_var": true
  }'

If the collector requires authentication, configure its headers under collector.request_headers. For all metadata fields and defaults, see the plugin metadata reference.

Enable Tracing Globally​

Create a sample service and route, then apply opentelemetry in a global rule. The rule traces requests to every route in the gateway group. To trace selected traffic instead, configure the plugin on individual routes or services.

The examples use the fixed IDs tracing-demo and opentelemetry. Confirm that these IDs are unused, or replace them consistently throughout the configuration before applying it.

Create a service with an upstream:

curl -k "https://localhost:7443/apisix/admin/services/tracing-demo?gateway_group_id={gateway_group_id}" -X PUT \
  -H "X-API-KEY: ${API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "tracing-demo",
    "upstream": {
      "type": "roundrobin",
      "scheme": "http",
      "nodes": [
        { "host": "httpbin.org", "port": 80, "weight": 1 }
      ]
    }
  }'

Create a route under the service:

curl -k "https://localhost:7443/apisix/admin/routes/tracing-demo?gateway_group_id={gateway_group_id}" -X PUT \
  -H "X-API-KEY: ${API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "tracing-demo",
    "service_id": "tracing-demo",
    "paths": ["/anything"]
  }'

Enable tracing with a global rule:

curl -k "https://localhost:7443/apisix/admin/global_rules/opentelemetry?gateway_group_id={gateway_group_id}" -X PUT \
  -H "X-API-KEY: ${API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "plugins": {
      "opentelemetry": {
        "sampler": {
          "name": "trace_id_ratio",
          "options": { "fraction": 1.0 }
        },
        "additional_attributes": ["request_uri"]
      }
    }
  }'

Values in additional_attributes are APISIX variables, such as request_uri or route_name. The plugin automatically adds common attributes, including the matched route, route name, status code, and response source.

Choose a Sampling Strategy​

StrategyWhen to use
always_onDevelopment, staging, or low-volume traffic where every request should be traced.
trace_id_ratioProduction traffic where a sampling fraction such as 0.01 keeps collector load and storage bounded. The decision is deterministic for services that share the same trace ID.
parent_baseTraffic where an upstream service or service mesh makes the sampling decision. The gateway respects a parent decision and uses a root sampler when no parent span exists.
always_offThe default. Use it to keep the plugin attached without emitting spans.

A common production strategy is parent_base with a low-ratio root sampler. Existing distributed traces remain intact while new root traces are sampled at a controlled rate:

{
  "sampler": {
    "name": "parent_base",
    "options": {
      "root": {
        "name": "trace_id_ratio",
        "options": { "fraction": 0.01 }
      }
    }
  }
}

For all plugin parameters, see the opentelemetry plugin reference.

Verify Trace Export​

Send a request through the gateway:

curl -i "http://127.0.0.1:9080/anything"

Wait for the asynchronously exported trace to become available:

trace_ready=false
for attempt in $(seq 1 20); do
  if curl -fsS \
    "http://127.0.0.1:16686/api/traces?service=api7-gateway&limit=10&lookback=1h" | \
    grep -q '"traceID"'; then
    trace_ready=true
    break
  fi
  sleep 1
done
[ "$trace_ready" = true ]

Navigate to the Jaeger UI at http://127.0.0.1:16686 and search for the api7-gateway service. The trace should include a span named GET /anything with attributes such as the status code, route name, response source, and request URI.

Correlate Traces with Access Logs​

Add the trace and span ID variables to the access log format. This makes it possible to find the trace associated with a log entry.

Add the following configuration to the data plane's conf/config.yaml:

conf/config.yaml
nginx_config:
  http:
    enable_access_log: true
    access_log_format: '{"time":"$time_iso8601","trace_id":"$opentelemetry_trace_id","span_id":"$opentelemetry_span_id","status":$status,"request":"$request"}'
    access_log_format_escape: json

Reload or restart the data plane for the change to take effect.

After the data plane is ready, send another request:

curl -i "http://127.0.0.1:9080/anything"

The access log should contain nonempty trace and span IDs similar to the following:

{
  "time": "2026-09-16T10:30:00+00:00",
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "span_id": "00f067aa0ba902b7",
  "status": 200,
  "request": "GET /anything HTTP/1.1"
}

The same variables can be used in the log_format field of logger plugins such as http-logger and kafka-logger.

Next Steps​