Trace API Traffic

Traces are one of the three pillars of observability, along with metrics and logs. A trace tracks the journey of a request as it traverses through various parts of a system. It is an effective mechanism that helps developers and administrators monitor system performance, identify bottlenecks, and improve user experience.

This tutorial guides you through creating tracing using the opentelemetry plugin, which instruments the Gateway and sends traces to the OpenTelemetry collector based on the OpenTelemetry specification, in binary-encoded OLTP over HTTP. Below is an interactive demo providing a hands-on introduction to the following use cases.

Prerequisites

1. Have a running API on the gateway group and create a route.

Install OpenTelemetry

While the steps in this section use specific OpenTelemetry collectors, you may use other distributions of collectors, such as SigNoz OpenTelemetry Collector, which all handle telemetry data but vary in focus, features, and flexibility for integration.

Docker

Start an OpenTelemetry collector instance:

docker run -d --name otel-collector -p 4318:4318 otel/opentelemetry-collector-contrib

Kubernetes

Install cert-manager as a dependency:

kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.16.1/cert-manager.yaml

When cert-manager is ready, install OpenTelemetry operator:

kubectl apply -f https://github.com/open-telemetry/opentelemetry-operator/releases/latest/download/opentelemetry-operator.yaml

Install an OpenTelemetry instance:

kubectl apply -f - <<EOF
apiVersion: opentelemetry.io/v1beta1
kind: OpenTelemetryCollector
metadata:
  namespace: api7
  name: simplest
spec:
  config:
    receivers:
      otlp:
        protocols:
          grpc:
            endpoint: 0.0.0.0:4317
          http:
            endpoint: 0.0.0.0:4318
    processors:
      memory_limiter:
        check_interval: 1s
        limit_percentage: 75
        spike_limit_percentage: 15
      batch:
        send_batch_size: 10000
        timeout: 10s

    exporters:
      debug: {}

    service:
      pipelines:
        traces:
          receivers: [otlp]
          processors: [memory_limiter, batch]
          exporters: [debug]
EOF

Configure OpenTelemetry Metadata

In this section, you will be configuring the metadata for opentelemetry plugin, which specifies the connection interface and other parameters.

Dashboard

1. Select Plugin Settings of your gateway group from the side navigation bar.

2. Select the Plugin Metadata tab, then click Add Plugin Metadata.

3. Search for the opentelemetry plugin, then click Add Metadata.

4. In the dialog box that appeared, add the following configuration to the JSON Editor and replace the collector address with your IP:

   {
     "trace_id_source": "x-request-id",
     "resource": {
       "service.name": "API7"
     },
     "collector": {
       "address": "192.168.2.106:4318",
       "request_timeout": 3,
       "request_headers": {
         "Authorization": "token"
       }
     },
     "batch_span_processor": {
       "drop_on_queue_full": false,
       "max_queue_size": 1024,
       "batch_timeout": 2,
       "inactive_timeout": 1,
       "max_export_batch_size": 16
     },
     "set_ngx_var": true
   }

5. Click Save.

ADC

To use ADC to update plugin metadata, create the following configuration:

otel-metadata.yaml

plugin_metadata:
  opentelemetry:
    batch_span_processor:
      batch_timeout: 2
      drop_on_queue_full: false
      inactive_timeout: 1
      max_export_batch_size: 16
      max_queue_size: 1024
    collector:
      address: 192.168.2.106:4318
      request_headers:
        Authorization: token
      request_timeout: 3
    resource:
      service.name: API7
    set_ngx_var: true
    trace_id_source: x-request-id

Synchronize the configuration to API7 Enterprise:

adc sync -f otel-metadata.yaml

Ingress Controller

Gateway API

APISIX CRD

Update your GatewayProxy manifest to include the opentelemetry plugin metadata configuration:

gateway-proxy.yaml

apiVersion: apisix.apache.org/v1alpha1
kind: GatewayProxy
metadata:
  namespace: api7
  name: apisix
spec:
  provider:
    type: ControlPlane
    controlPlane:
      # your control plane connection configuration
      # ....
  pluginMetadata:
    opentelemetry: {
      "trace_id_source": "x-request-id",
      "resource": {
        "service.name": "API7"
      },
      "collector": {
        "address": "simplest-collector:4318",
        "request_timeout": 3,
        "request_headers": {
          "Authorization": "token"
        }
      },
      "batch_span_processor": {
        "drop_on_queue_full": false,
        "max_queue_size": 1024,
        "batch_timeout": 2,
        "inactive_timeout": 1,
        "max_export_batch_size": 16
      },
      "set_ngx_var": true
    }

Apply the configuration to your cluster:

kubectl apply -f gateway-proxy.yaml

Go to the dashboard and under Plugin Settings > Plugin Metadata, you should see the plugin metadata updated with opentelemetry plugin and its metadata.

Configure OpenTelemetry as a Global Plugin

In this example, you will be configuring opentelemetry as a global plugin that traces all requests.

Dashboard

1. Select Plugin Settings of your gateway group from the side navigation bar.
2. Select the Plugin Global Rules tab, then click Add Plugin.
3. Search for the opentelemetry plugin, then click Add.
4. In the dialog box that appeared, add the following configuration to the JSON Editor:

  {
    "opentelemetry": {
      "sampler": {
        "name": "always_on"
      }
    }
  }

5. Click Add.

ADC

To use ADC to configure a global plugin, create the following configuration:

otel-global-plugin.yaml

global_rules:
  opentelemetry:
    _meta:
      disable: false
    opentelemetry:
      sampler:
        name: always_on

Synchronize the configuration to API7 Enterprise:

adc sync -f otel-global-plugin.yaml otel-metadata.yaml

Ingress Controller

Gateway API

Update your GatewayProxy manifest to enable opentelemetry as a global plugin:

global-rule-otel.yaml

apiVersion: apisix.apache.org/v1alpha1
kind: GatewayProxy
metadata:
  namespace: api7
  name: apisix
spec:
  provider:
    type: ControlPlane
    controlPlane:
      endpoints:
        - https://xxx.xxx.xxx.xxx:7443  # update with your CP endpoint
      auth:
        type: AdminKey
        adminKey:
          value: xxxxxxxxxxx            # update with your admin key
  plugins:
  - name: opentelemetry
    enabled: true
    config:
      sampler:
        name: always_on
  pluginMetadata:
    opentelemetry: {
      "trace_id_source": "x-request-id",
      "resource": {
        "service.name": "API7"
      },
      "collector": {
        "address": "simplest-collector:4318",
        "request_timeout": 3,
        "request_headers": {
          "Authorization": "token"
        }
      },
      "batch_span_processor": {
        "drop_on_queue_full": false,
        "max_queue_size": 1024,
        "batch_timeout": 2,
        "inactive_timeout": 1,
        "max_export_batch_size": 16
      },
      "set_ngx_var": true
    }

Apply the configuration to your cluster:

kubectl apply -f gateway-proxy.yaml

Once applied, you should see the opentelemetry plugin appear under the Plugin Settings on the dashboard.

APISIX CRD

Create a Kubernetes manifest file for a global opentelemetry plugin:

global-otel.yaml

apiVersion: apisix.apache.org/v2
kind: ApisixGlobalRule
metadata:
  namespace: api7
  name: apisix-global-rule-logging
spec:
  ingressClassName: apisix
  plugins:
  - name: opentelemetry
    config:
      sampler:
        name: always_on

Apply the configuration to your cluster:

kubectl apply -f global-otel.yaml

Verify

Suppose you have completed steps in launch your first API to create a sample route.

Send a few requests to the route:

curl "http://127.0.0.1:9080/ip"

In the OpenTelemetry collector's log, you should see traces collected from the gateway traffic:

{"resource": {"service.instance.id": "c4a2503a-f1e7-4077-9e1e-f139da86442f", "service.name": "otelcol", "service.version": "0.136.0"}, "otelcol.component.id": "debug", "otelcol.component.kind": "exporter", "otelcol.signal": "traces", "resource spans": 1, "spans": 1}

Configure OpenTelemetry on a Route

In this example, you will be configuring opentelemetry plugin on a single route, which only traces requests visiting this route.

Dashboard

1. Click into your service to create a new route or update the existing /ip route.
2. Click Add Plugin.
3. Search for the opentelemetry plugin, then click Add.
4. In the dialog box that appeared, add the following configuration to the JSON Editor:

  {
    "opentelemetry": {
      "sampler": {
        "name": "always_on"
      }
    }
  }

5. Click Add.

ADC

To use ADC to update plugin on a route, create the following configuration:

otel-route.yaml

services:
  - name: httpbin
    upstream:
      name: httpbin
      scheme: http
      type: roundrobin
      nodes:
        - host: httpbin.org
          port: 80
          weight: 100
    routes:
      - uris:
          - /ip
        name: get-ip
        methods:
          - GET
        plugins:
          opentelemetry:
            _meta:
              disable: false
            opentelemetry:
              sampler:
                name: always_on

Synchronize the configuration to API7 Enterprise:

adc sync -f otel-route.yaml otel-metadata.yaml

Ingress Controller

Gateway API

API7 Ingress Controller currently does not support enabling plugins on a single route. Alternatively, you could create a new service and enable the opentelemetry plugin on the service.

Create a Kubernetes manifest file for a route (that also creates a service) and enable opentelemetry:

service-otel.yaml

apiVersion: apisix.apache.org/v1alpha1
kind: PluginConfig
metadata:
  namespace: api7
  name: otel-plugin-config
spec:
  plugins:
    - name: opentelemetry
      config:
        sampler:
          name: always_on
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  namespace: api7
  name: get-ip
spec:
  parentRefs:
  - name: apisix
  rules:
  - matches: 
    - path:
        type: Exact
        value: /ip
    filters:
    - type: ExtensionRef
      extensionRef:
        group: apisix.apache.org
        kind: PluginConfig
        name: otel-plugin-config
    backendRefs:
    - name: httpbin
      port: 80

Apply the configuration to your cluster:

kubectl apply -f service-otel.yaml

Once applied, you should see the opentelemetry plugin appears under the service plugins.

APISIX CRD

Create a Kubernetes manifest file for a route (that also creates a service) and enable opentelemetry:

route-otel.yaml

apiVersion: apisix.apache.org/v2
kind: ApisixRoute
metadata:
  namespace: api7
  name: get-ip
spec:
  ingressClassName: apisix
  http:
    - name: get-ip
      match:
        paths:
          - /ip
      plugins:
      - name: opentelemetry
        enable: true
        config:
          sampler:
            name: always_on
      backends:
      - serviceName: httpbin
        servicePort: 80

Apply the configuration to your cluster:

kubectl apply -f route-otel.yaml

Verify

Send a few requests to the route:

curl "http://127.0.0.1:9080/ip"

In the OpenTelemetry collector's log, you should see traces collected from the gateway traffic:

{"resource": {"service.instance.id": "c4a2503a-f1e7-4077-9e1e-f139da86442f", "service.name": "otelcol", "service.version": "0.136.0"}, "otelcol.component.id": "debug", "otelcol.component.kind": "exporter", "otelcol.signal": "traces", "resource spans": 1, "spans": 1}

Use Trace Variables in Gateway Access Logging

In this example, you will be updating the Gateway's configuration to configure the access log to show OpenTelemetry tracing information. Make sure the set_ngx_var has been set to true in the plugin metadata.

Docker

Update the logging format configurations in the gateway's configuration file:

docker exec <api7-ee-gateway-container-name> /bin/sh -c "echo '
nginx_config:
  http:
    enable_access_log: true
    access_log_format: \"{\\\"time\\\": \\\"\$time_iso8601\\\",\\\"opentelemetry_context_traceparent\\\": \\\"\$opentelemetry_context_traceparent\\\",\\\"opentelemetry_trace_id\\\": \\\"\$opentelemetry_trace_id\\\",\\\"opentelemetry_span_id\\\": \\\"\$opentelemetry_span_id\\\",\\\"remote_addr\\\": \\\"\$remote_addr\\\"}\"
    access_log_format_escape: json
' > /usr/local/apisix/conf/config.yaml"

Reload the container for configuration changes to take effect:

docker exec <api7-ee-gateway-container-name> apisix reload

Kubernetes

To update the gateway's logging format, first export all values (including defaults):

helm get values -n api7 api7-ee-3-gateway --all > values.yaml

In the values file, update the following section values as such:

values.yaml

logs:
  enableAccessLog: true
  accessLogFormat: '{"time": "$time_iso8601","opentelemetry_context_traceparent": "$opentelemetry_context_traceparent","opentelemetry_trace_id": "$opentelemetry_trace_id","opentelemetry_span_id": "$opentelemetry_span_id","remote_addr": "$remote_addr"}'
  accessLogFormatEscape: json

Upgrade the release:

helm upgrade --install -n api7 api7-ee-3-gateway api7/gateway -f values.yaml

Verify

Suppose you have completed steps in launch your first API to create a sample route.

Send a few requests to the route:

curl "http://127.0.0.1:9080/ip"

In the Gateway's log, you should see access logs similar to the following:

{"time": "2025-10-10T10:00:33+00:00","opentelemetry_context_traceparent": "00-dd43f6275874022ef5fe6c470b4fee89-c099993cd4ad537a-01","opentelemetry_trace_id": "dd43f6275874022ef5fe6c470b4fee89","opentelemetry_span_id": "c099993cd4ad537a","remote_addr": "127.0.0.1"}
{"time": "2025-10-10T10:01:17+00:00","opentelemetry_context_traceparent": "00-830ec1895f16da34063e3a5c5b96c1fb-06ca189db60cba2a-01","opentelemetry_trace_id": "830ec1895f16da34063e3a5c5b96c1fb","opentelemetry_span_id": "06ca189db60cba2a","remote_addr": "127.0.0.1"}