Trace API Traffic

Traces are one of the three pillars of observability, along with metrics and logs. A trace tracks the journey of a request as it traverses through various parts of a system. It is an effective mechanism that helps developers and administrators monitor system performance, identify bottlenecks, and improve user experience.

This tutorial guides you through creating tracing using the opentelemetry plugin, which instruments the Gateway and sends traces to the OpenTelemetry collector based on the OpenTelemetry specification, in binary-encoded OLTP over HTTP. Below is an interactive demo providing a hands-on introduction to the following use cases.

Prerequisites

1. Have a running API on the gateway group and create a route.

Install OpenTelemetry

While the steps in this section use specific OpenTelemetry collectors, you may use other distributions of collectors, such as SigNoz OpenTelemetry Collector, which all handle telemetry data but vary in focus, features, and flexibility for integration.

Docker

Start an OpenTelemetry collector instance:

docker run -d --name otel-collector -p 4318:4318 otel/opentelemetry-collector-contrib

Kubernetes

Install cert-manager as a dependency:

kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.16.1/cert-manager.yaml

When cert-manager is ready, install OpenTelemetry operator:

kubectl apply -f https://github.com/open-telemetry/opentelemetry-operator/releases/latest/download/opentelemetry-operator.yaml

Install an OpenTelemetry instance:

kubectl apply -f - <<EOF
apiVersion: opentelemetry.io/v1beta1
kind: OpenTelemetryCollector
metadata:
  namespace: api7
  name: simplest
spec:
  config:
    receivers:
      otlp:
        protocols:
          grpc:
            endpoint: 0.0.0.0:4317
          http:
            endpoint: 0.0.0.0:4318
    processors:
      memory_limiter:
        check_interval: 1s
        limit_percentage: 75
        spike_limit_percentage: 15
      batch:
        send_batch_size: 10000
        timeout: 10s

    exporters:
      debug: {}

    service:
      pipelines:
        traces:
          receivers: [otlp]
          processors: [memory_limiter, batch]
          exporters: [debug]
EOF

Configure OpenTelemetry Metadata

In this section, you will be configuring the metadata for opentelemetry plugin, which specifies the connection interface and other parameters.

Dashboard

1. Select Plugin Settings of your gateway group from the side navigation bar.

2. Select the Plugin Metadata tab, then click Add Plugin Metadata.

3. Search for the opentelemetry plugin, then click Add Metadata.

4. In the dialog box that appeared, add the following configuration to the JSON Editor and replace the collector address with your IP:

   {
     "trace_id_source": "x-request-id",
     "resource": {
       "service.name": "API7"
     },
     "collector": {
       "address": "192.168.2.106:4318",
       "request_timeout": 3,
       "request_headers": {
         "Authorization": "token"
       }
     },
     "batch_span_processor": {
       "drop_on_queue_full": false,
       "max_queue_size": 1024,
       "batch_timeout": 2,
       "inactive_timeout": 1,
       "max_export_batch_size": 16
     },
     "set_ngx_var": true
   }

5. Click Save.

ADC

To use ADC to update plugin metadata, create the following configuration:

otel-metadata.yaml

plugin_metadata:
  opentelemetry:
    batch_span_processor:
      batch_timeout: 2
      drop_on_queue_full: false
      inactive_timeout: 1
      max_export_batch_size: 16
      max_queue_size: 1024
    collector:
      address: 192.168.2.106:4318
      request_headers:
        Authorization: token
      request_timeout: 3
    resource:
      service.name: API7
    set_ngx_var: true
    trace_id_source: x-request-id

Synchronize the configuration to API7 Enterprise:

adc sync -f otel-metadata.yaml

Configure OpenTelemetry as a Global Plugin

In this example, you will be configuring opentelemetry as a global plugin that traces all requests.

Dashboard

1. Select Plugin Settings of your gateway group from the side navigation bar.
2. Select the Plugin Global Rules tab, then click Add Plugin.
3. Search for the opentelemetry plugin, then click Add.
4. In the dialog box that appeared, add the following configuration to the JSON Editor:

  {
    "opentelemetry": {
      "sampler": {
        "name": "always_on"
      }
    }
  }

5. Click Add.

ADC

To use ADC to configure a global plugin, create the following configuration:

otel-global-plugin.yaml

global_rules:
  opentelemetry:
    _meta:
      disable: false
    opentelemetry:
      sampler:
        name: always_on

Synchronize the configuration to API7 Enterprise:

adc sync -f otel-global-plugin.yaml otel-metadata.yaml

Verify

Suppose you have completed steps in launch your first API to create a sample route.

Send a few requests to the route:

curl "http://127.0.0.1:9080/ip"

In the OpenTelemetry collector's log, you should see traces collected from the gateway traffic:

{"resource": {"service.instance.id": "c4a2503a-f1e7-4077-9e1e-f139da86442f", "service.name": "otelcol", "service.version": "0.136.0"}, "otelcol.component.id": "debug", "otelcol.component.kind": "exporter", "otelcol.signal": "traces", "resource spans": 1, "spans": 1}

Configure OpenTelemetry on a Route

In this example, you will be configuring opentelemetry plugin on a single route, which only traces requests visiting this route.

Dashboard

1. Click into your service to create a new route or update the existing /ip route.
2. Click Add Plugin.
3. Search for the opentelemetry plugin, then click Add.
4. In the dialog box that appeared, add the following configuration to the JSON Editor:

  {
    "opentelemetry": {
      "sampler": {
        "name": "always_on"
      }
    }
  }

5. Click Add.

ADC

To use ADC to update plugin on a route, create the following configuration:

otel-route.yaml

services:
  - name: httpbin
    upstream:
      name: httpbin
      scheme: http
      type: roundrobin
      nodes:
        - host: httpbin.org
          port: 80
          weight: 100
    routes:
      - uris:
          - /ip
        name: get-ip
        methods:
          - GET
        plugins:
          opentelemetry:
            _meta:
              disable: false
            opentelemetry:
              sampler:
                name: always_on

Synchronize the configuration to API7 Enterprise:

adc sync -f otel-route.yaml otel-metadata.yaml

Verify

Send a few requests to the route:

curl "http://127.0.0.1:9080/ip"

In the OpenTelemetry collector's log, you should see traces collected from the gateway traffic:

{"resource": {"service.instance.id": "c4a2503a-f1e7-4077-9e1e-f139da86442f", "service.name": "otelcol", "service.version": "0.136.0"}, "otelcol.component.id": "debug", "otelcol.component.kind": "exporter", "otelcol.signal": "traces", "resource spans": 1, "spans": 1}

Use Trace Variables in Gateway Access Logging

In this example, you will be updating the Gateway's configuration to configure the access log to show OpenTelemetry tracing information. Make sure the set_ngx_var has been set to true in the plugin metadata.

Docker

Update the logging format configurations in the gateway's configuration file:

docker exec <api7-ee-gateway-container-name> /bin/sh -c "echo '
nginx_config:
  http:
    enable_access_log: true
    access_log_format: \"{\\\"time\\\": \\\"\$time_iso8601\\\",\\\"opentelemetry_context_traceparent\\\": \\\"\$opentelemetry_context_traceparent\\\",\\\"opentelemetry_trace_id\\\": \\\"\$opentelemetry_trace_id\\\",\\\"opentelemetry_span_id\\\": \\\"\$opentelemetry_span_id\\\",\\\"remote_addr\\\": \\\"\$remote_addr\\\"}\"
    access_log_format_escape: json
' > /usr/local/apisix/conf/config.yaml"

Reload the container for configuration changes to take effect:

docker exec <api7-ee-gateway-container-name> apisix reload

Kubernetes

To update the gateway's logging format, first export all values (including defaults):

helm get values -n api7 api7-ee-3-gateway --all > values.yaml

In the values file, update the following section values as such:

values.yaml

logs:
  enableAccessLog: true
  accessLogFormat: '{"time": "$time_iso8601","opentelemetry_context_traceparent": "$opentelemetry_context_traceparent","opentelemetry_trace_id": "$opentelemetry_trace_id","opentelemetry_span_id": "$opentelemetry_span_id","remote_addr": "$remote_addr"}'
  accessLogFormatEscape: json

Upgrade the release:

helm upgrade --install -n api7 api7-ee-3-gateway api7/gateway -f values.yaml

Verify

Suppose you have completed steps in launch your first API to create a sample route.

Send a few requests to the route:

curl "http://127.0.0.1:9080/ip"

In the Gateway's log, you should see access logs similar to the following:

{"time": "2025-10-10T10:00:33+00:00","opentelemetry_context_traceparent": "00-dd43f6275874022ef5fe6c470b4fee89-c099993cd4ad537a-01","opentelemetry_trace_id": "dd43f6275874022ef5fe6c470b4fee89","opentelemetry_span_id": "c099993cd4ad537a","remote_addr": "127.0.0.1"}
{"time": "2025-10-10T10:01:17+00:00","opentelemetry_context_traceparent": "00-830ec1895f16da34063e3a5c5b96c1fb-06ca189db60cba2a-01","opentelemetry_trace_id": "830ec1895f16da34063e3a5c5b96c1fb","opentelemetry_span_id": "06ca189db60cba2a","remote_addr": "127.0.0.1"}