# Configure Distributed Tracing

Distributed tracing shows how requests move through API7 Gateway and backend services, helping you locate latency and failures. API7 Gateway uses the [`opentelemetry`](https://docs.api7.ai/hub/opentelemetry.md) plugin to create spans and export them to an OTLP/HTTP endpoint, such as an OpenTelemetry Collector or a compatible observability backend.

This guide configures a local Jaeger instance for evaluation, enables tracing globally, and correlates the exported traces with access logs.

## How It Works[​](#how-it-works "Direct link to How It Works")

The `opentelemetry` plugin uses two configuration layers:

* **Plugin metadata** configures the collector endpoint, request timeout, resource attributes, custom headers, batch span processor, and trace ID source for a gateway group.
* **Plugin configuration** selects a sampling strategy and adds span attributes. You can apply the plugin to a route, service, or global rule.

Plugin metadata must be configured before the plugin can create and export spans.

## Prerequisites[​](#prerequisites "Direct link to Prerequisites")

* A running API7 Gateway deployment with at least one online data plane.
* [Docker](https://docs.docker.com/get-docker/) to run Jaeger for this local evaluation.
* A [Dashboard token](https://docs.api7.ai/api7-gateway/reference/obtain-dashboard-token.md) assigned to the `API_KEY` environment variable.
* If you use ADC, [install and configure ADC](https://docs.api7.ai/api7-gateway/getting-started/quick-start.md), including the `ADC_BACKEND`, `ADC_SERVER`, `ADC_TOKEN`, and `ADC_GATEWAY_GROUP` environment variables.

## Start Jaeger[​](#start-jaeger "Direct link to Start Jaeger")

This local example assumes that the data plane runs in Docker. For another deployment type, use an OTLP/HTTP endpoint reachable from the data plane, skip the Docker commands in this section, and substitute its address when configuring plugin metadata.

Create a Docker network and connect the data plane container to it. Replace `<data-plane-container>` with the container name:

```
docker network create gateway-tracing-net

docker network connect gateway-tracing-net <data-plane-container>
```

Start a pinned Jaeger instance on the same network. The image accepts OTLP traces on port `4318` and provides the Jaeger UI on port `16686`:

```
docker run -d --name jaeger \

  --network gateway-tracing-net \

  -p 127.0.0.1:16686:16686 \

  cr.jaegertracing.io/jaegertracing/jaeger:2.20.0
```

Wait for the Jaeger UI to become available:

```
until curl -fsS "http://127.0.0.1:16686/" > /dev/null; do

  sleep 1

done
```

Local evaluation setup

This Jaeger instance uses transient in-memory storage without authentication or TLS. For production, use a secured collector or observability backend with durable storage. If a suitable OTLP/HTTP endpoint is already available, skip this section and substitute its address below.

## Configure the Collector Endpoint[​](#configure-the-collector-endpoint "Direct link to Configure the Collector Endpoint")

Configure plugin metadata to send spans to Jaeger and expose the trace and span IDs as NGINX variables. The variables are used later to correlate traces with access logs.

* Admin API
* ADC

```
curl -k "https://localhost:7443/apisix/admin/plugin_metadata/opentelemetry?gateway_group_id={gateway_group_id}" -X PUT \

  -H "X-API-KEY: ${API_KEY}" \

  -H "Content-Type: application/json" \

  -d '{

    "resource": {

      "service.name": "api7-gateway",

      "deployment.environment": "development"

    },

    "collector": {

      "address": "jaeger:4318",

      "request_timeout": 3

    },

    "set_ngx_var": true

  }'
```

Add the plugin metadata to your existing ADC configuration so other plugin metadata remains in the desired state:

adc.yaml

```
plugin_metadata:

  opentelemetry:

    resource:

      service.name: api7-gateway

      deployment.environment: development

    collector:

      address: "jaeger:4318"

      request_timeout: 3

    set_ngx_var: true
```

Preview the plugin-metadata reconciliation and confirm that it contains no unintended updates or deletions:

```
adc diff -f adc.yaml --include-resource-type plugin_metadata
```

Synchronize the reviewed configuration:

```
adc sync -f adc.yaml --include-resource-type plugin_metadata
```

If the collector requires authentication, configure its headers under `collector.request_headers`. For all metadata fields and defaults, see the [plugin metadata reference](https://docs.api7.ai/hub/opentelemetry/configuration.md#plugin-metadata).

## Enable Tracing Globally[​](#enable-tracing-globally "Direct link to Enable Tracing Globally")

Create a sample service and route, then apply `opentelemetry` in a global rule. The rule traces requests to every route in the gateway group. To trace selected traffic instead, configure the plugin on individual routes or services.

The examples use the fixed IDs `tracing-demo` and `opentelemetry`. Confirm that these IDs are unused, or replace them consistently throughout the configuration before applying it.

* Admin API
* ADC

Create a service with an upstream:

```
curl -k "https://localhost:7443/apisix/admin/services/tracing-demo?gateway_group_id={gateway_group_id}" -X PUT \

  -H "X-API-KEY: ${API_KEY}" \

  -H "Content-Type: application/json" \

  -d '{

    "name": "tracing-demo",

    "upstream": {

      "type": "roundrobin",

      "scheme": "http",

      "nodes": [

        { "host": "httpbin.org", "port": 80, "weight": 1 }

      ]

    }

  }'
```

Create a route under the service:

```
curl -k "https://localhost:7443/apisix/admin/routes/tracing-demo?gateway_group_id={gateway_group_id}" -X PUT \

  -H "X-API-KEY: ${API_KEY}" \

  -H "Content-Type: application/json" \

  -d '{

    "name": "tracing-demo",

    "service_id": "tracing-demo",

    "paths": ["/anything"]

  }'
```

Enable tracing with a global rule:

```
curl -k "https://localhost:7443/apisix/admin/global_rules/opentelemetry?gateway_group_id={gateway_group_id}" -X PUT \

  -H "X-API-KEY: ${API_KEY}" \

  -H "Content-Type: application/json" \

  -d '{

    "plugins": {

      "opentelemetry": {

        "sampler": {

          "name": "trace_id_ratio",

          "options": { "fraction": 1.0 }

        },

        "additional_attributes": ["request_uri"]

      }

    }

  }'
```

Add the service, route, and global rule to your existing ADC configuration so other services and global rules remain in the desired state:

adc.yaml

```
services:

  - name: tracing-demo

    routes:

      - name: tracing-demo

        uris:

          - /anything

    upstream:

      type: roundrobin

      nodes:

        - host: httpbin.org

          port: 80

          weight: 1



global_rules:

  opentelemetry:

    sampler:

      name: trace_id_ratio

      options:

        fraction: 1.0

    additional_attributes:

      - request_uri
```

Preview the reconciliation and confirm that it contains no unintended updates or deletions:

```
adc diff -f adc.yaml \

  --include-resource-type service \

  --include-resource-type global_rule
```

Synchronize the reviewed configuration. The resource-type filters also keep the plugin metadata unchanged:

```
adc sync -f adc.yaml \

  --include-resource-type service \

  --include-resource-type global_rule
```

Values in `additional_attributes` are [APISIX variables](https://docs.api7.ai/api7-gateway/reference/built-in-variables.md), such as `request_uri` or `route_name`. The plugin automatically adds common attributes, including the matched route, route name, status code, and response source.

### Choose a Sampling Strategy[​](#choose-a-sampling-strategy "Direct link to Choose a Sampling Strategy")

| Strategy         | When to use                                                                                                                                                                    |
| ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `always_on`      | Development, staging, or low-volume traffic where every request should be traced.                                                                                              |
| `trace_id_ratio` | Production traffic where a sampling fraction such as `0.01` keeps collector load and storage bounded. The decision is deterministic for services that share the same trace ID. |
| `parent_base`    | Traffic where an upstream service or service mesh makes the sampling decision. The gateway respects a parent decision and uses a root sampler when no parent span exists.      |
| `always_off`     | The default. Use it to keep the plugin attached without emitting spans.                                                                                                        |

A common production strategy is `parent_base` with a low-ratio root sampler. Existing distributed traces remain intact while new root traces are sampled at a controlled rate:

```
{

  "sampler": {

    "name": "parent_base",

    "options": {

      "root": {

        "name": "trace_id_ratio",

        "options": { "fraction": 0.01 }

      }

    }

  }

}
```

For all plugin parameters, see the [`opentelemetry` plugin reference](https://docs.api7.ai/hub/opentelemetry/configuration.md#parameters).

## Verify Trace Export[​](#verify-trace-export "Direct link to Verify Trace Export")

Send a request through the gateway:

```
curl -i "http://127.0.0.1:9080/anything"
```

Wait for the asynchronously exported trace to become available:

```
trace_ready=false

for attempt in $(seq 1 20); do

  if curl -fsS \

    "http://127.0.0.1:16686/api/traces?service=api7-gateway&limit=10&lookback=1h" | \

    grep -q '"traceID"'; then

    trace_ready=true

    break

  fi

  sleep 1

done

[ "$trace_ready" = true ]
```

Navigate to the Jaeger UI at <http://127.0.0.1:16686> and search for the `api7-gateway` service. The trace should include a span named `GET /anything` with attributes such as the status code, route name, response source, and request URI.

## Correlate Traces with Access Logs[​](#correlate-traces-with-access-logs "Direct link to Correlate Traces with Access Logs")

Add the trace and span ID variables to the access log format. This makes it possible to find the trace associated with a log entry.

* Host or Docker
* Kubernetes

Add the following configuration to the data plane's `conf/config.yaml`:

conf/config.yaml

```
nginx_config:

  http:

    enable_access_log: true

    access_log_format: '{"time":"$time_iso8601","trace_id":"$opentelemetry_trace_id","span_id":"$opentelemetry_span_id","status":$status,"request":"$request"}'

    access_log_format_escape: json
```

Reload or restart the data plane for the change to take effect.

Add the following values to the data plane Helm configuration:

values.yaml

```
logs:

  enableAccessLog: true

  accessLogFormat: '{"time":"$time_iso8601","trace_id":"$opentelemetry_trace_id","span_id":"$opentelemetry_span_id","status":$status,"request":"$request"}'

  accessLogFormatEscape: json
```

Apply the values. Replace the placeholders with the Helm release name and namespace:

```
helm upgrade <release-name> api7/gateway \

  --version "~3.10.0" \

  --namespace <namespace> \

  -f values.yaml
```

After the data plane is ready, send another request:

```
curl -i "http://127.0.0.1:9080/anything"
```

The access log should contain nonempty trace and span IDs similar to the following:

```
{

  "time": "2026-09-16T10:30:00+00:00",

  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",

  "span_id": "00f067aa0ba902b7",

  "status": 200,

  "request": "GET /anything HTTP/1.1"

}
```

The same variables can be used in the `log_format` field of logger plugins such as [`http-logger`](https://docs.api7.ai/hub/http-logger.md) and [`kafka-logger`](https://docs.api7.ai/hub/kafka-logger.md).

## Next Steps[​](#next-steps "Direct link to Next Steps")

* [Monitor Metrics](https://docs.api7.ai/api7-gateway/observability/metrics.md) to correlate traces with Prometheus metrics and latency dashboards.
* [Configure Centralized Logging](https://docs.api7.ai/api7-gateway/observability/logging.md) to send trace-correlated access logs to a log backend.
* Review the [`opentelemetry` plugin reference](https://docs.api7.ai/hub/opentelemetry/configuration.md) for all plugin and metadata fields.