Skip to main content
API7.ai Logo

OpenTelemetry

OpenTelemetry is a vendor-neutral observability framework for generating, collecting, and exporting telemetry. The opentelemetry plugin instruments gateway requests and exports sampled traces as binary Protobuf over OTLP/HTTP to an OpenTelemetry Collector. The collector can process the traces and forward them to a compatible observability backend.

Examples​

The examples below configure trace export and expose trace identifiers to gateway access logs.

Enable the Plugin​

In API7 Gateway, opentelemetry is available in Dashboard and Admin API by default. For APISIX deployments, load the plugin in the gateway static configuration before configuring routes that use it.

For APISIX host or Docker deployments, keep the existing plugin list in config.yaml and add opentelemetry:

config.yaml
plugins:
  # Keep the complete plugin list used by your gateway.
  - opentelemetry

Reload the gateway for changes to take effect.

Send Traces to an OpenTelemetry Collector​

The following example sends traces to an OpenTelemetry Collector that writes detailed span data to its container log.

Local evaluation setup

The collector configuration below uses the debug exporter without authentication or TLS. For production, secure the collector endpoint and configure an exporter for your observability backend.

Start the collector for your environment:

Create the collector configuration:

otel-collector-config.yaml
receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318

exporters:
  debug:
    verbosity: detailed

extensions:
  health_check:
    endpoint: 0.0.0.0:13133

service:
  extensions: [health_check]
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [debug]

Set GATEWAY_CONTAINER to the name of the running APISIX or API7 Gateway container. Create a dedicated network and connect the gateway to it:

export GATEWAY_CONTAINER=replace-with-gateway-container-name

docker network create gateway-otel-net
docker network connect gateway-otel-net "$GATEWAY_CONTAINER"

Start the collector on the same network. Publish only its health endpoint on the loopback interface:

docker run -d --name otel-collector \
  --network gateway-otel-net \
  -p 127.0.0.1:13133:13133 \
  -v "$PWD/otel-collector-config.yaml:/etc/otelcol/config.yaml:ro" \
  ghcr.io/open-telemetry/opentelemetry-collector-releases/opentelemetry-collector:0.160.0

Wait for the collector to become ready:

until curl -fsS "http://127.0.0.1:13133/" > /dev/null; do
  sleep 1
done

For API7 Gateway, use ADC or follow Configure Distributed Tracing for the managed Admin API workflow.

Configure the plugin metadata with the collector address:

curl "http://127.0.0.1:9180/apisix/admin/plugin_metadata/opentelemetry" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d '{
    "resource": {
      "service.name": "APISIX"
    },
    "collector": {
      "address": "otel-collector:4318"
    }
  }'

The service.name resource attribute identifies the gateway in trace backends. Replace APISIX with the service name used for your deployment.

Create a route with the opentelemetry plugin:

The examples use the fixed identifiers otel-tracing-route and httpbin. Confirm that they are unused, or replace them consistently throughout the configuration before applying it.

curl "http://127.0.0.1:9180/apisix/admin/routes/otel-tracing-route" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d '{
    "uri": "/anything",
    "plugins": {
      "opentelemetry": {
        "sampler": {
          "name": "always_on"
        }
      }
    },
    "upstream": {
      "type": "roundrobin",
      "nodes": {
        "httpbin.org": 1
      }
    }
  }'

The always_on sampler traces every request for this example. For production traffic, choose a sampling strategy that matches the required trace coverage and overhead.

Send a request to the route:

curl "http://127.0.0.1:9080/anything"

You should receive an HTTP/1.1 200 OK response.

Wait for the exported span, then inspect the collector log for your environment:

trace_ready=false
for attempt in $(seq 1 20); do
  if docker logs --since 1m otel-collector 2>&1 | \
    grep -q 'Name.*GET /anything'; then
    trace_ready=true
    break
  fi
  sleep 1
done
[ "$trace_ready" = true ]

docker logs --since 1m otel-collector

The log should contain span information similar to the following:

2026-09-15T06:44:59.753Z info ResourceSpans #0
Resource SchemaURL:
Resource attributes:
     -> telemetry.sdk.language: Str(lua)
     -> telemetry.sdk.name: Str(opentelemetry-lua)
     -> telemetry.sdk.version: Str(0.1.1)
     -> hostname: Str(9b3ccdfa09dc)
     -> service.name: Str(APISIX)
ScopeSpans #0
ScopeSpans SchemaURL:
InstrumentationScope opentelemetry-lua
Span #0
    Trace ID       : 224f7d84b4f77125b22f5d9a601021dc
    Parent ID      :
    ID             : c5597cecf6ba7128
    Name           : GET /anything
    Kind           : Server
    Start time     : 2026-09-15 06:44:52.040692992 +0000 UTC
    End time       : 2026-09-15 06:44:53.746045952 +0000 UTC
    Status code    : Unset
    Status message :
Attributes:
     -> net.host.name: Str(127.0.0.1)
     -> http.method: Str(GET)
     -> http.scheme: Str(http)
     -> http.target: Str(/anything)
     -> http.user_agent: Str(curl/8.7.1)
     -> http.request.method: Str(GET)
     -> url.scheme: Str(http)
     -> url.path: Str(/anything)
     -> user_agent.original: Str(curl/8.7.1)
     -> apisix.route_id: Str(otel-tracing-route)
     -> apisix.route_name: Empty()
     -> http.route: Str(/anything)
     -> apisix.response_source: Str(upstream)
     -> http.status_code: Int(200)
     -> http.response.status_code: Int(200)

To visualize traces, configure the collector with an exporter for a tracing backend such as Jaeger, Zipkin, or Grafana Tempo. See the OpenTelemetry exporter documentation for available options.

Introduced in API7 Enterprise 3.9.10 and APISIX 3.17.0, the apisix.response_source attribute classifies the origin of the HTTP response:

  • apisix: the response was generated by the gateway, such as a plugin rejection, authentication failure, or route-not-found error.
  • nginx: the response was generated by the NGINX proxy layer, such as a connection refusal or upstream timeout.
  • upstream: the response came from the upstream service.

This attribute enables more precise error attribution in trace analysis, for example, distinguishing gateway-side rejections from real upstream errors.

Use Trace Variables in Logs​

This example continues with the collector and /anything route configured in Send Traces to an OpenTelemetry Collector.

The plugin can populate the following built-in variables for use in logger plugins and access logs:

  • opentelemetry_context_traceparent: W3C traceparent value generated from the request span context
  • opentelemetry_trace_id: trace ID of the request span
  • opentelemetry_span_id: span ID of the request span

Update the plugin metadata to populate these variables while retaining the collector configuration:

curl "http://127.0.0.1:9180/apisix/admin/plugin_metadata/opentelemetry" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d '{
    "resource": {
      "service.name": "APISIX"
    },
    "collector": {
      "address": "otel-collector:4318"
    },
    "set_ngx_var": true
  }'

Configure the gateway access-log format for its deployment method.

Add or update this section in the gateway configuration file to use the opentelemetry plugin variables:

config.yaml
nginx_config:
  http:
    enable_access_log: true
    access_log_format: '{"time": "$time_iso8601","opentelemetry_context_traceparent": "$opentelemetry_context_traceparent","opentelemetry_trace_id": "$opentelemetry_trace_id","opentelemetry_span_id": "$opentelemetry_span_id","remote_addr": "$remote_addr"}'
    access_log_format_escape: json

Reload the gateway for configuration changes to take effect.

Send a request through the traced route:

curl "http://127.0.0.1:9080/anything"

You should see an access log entry similar to the following:

{"time": "2026-09-15T07:03:11+00:00","opentelemetry_context_traceparent": "00-d5698166ff3e63b6717fa57d8aebcce0-12b2f5c59eb9d2e3-01","opentelemetry_trace_id": "d5698166ff3e63b6717fa57d8aebcce0","opentelemetry_span_id": "12b2f5c59eb9d2e3","remote_addr": "192.168.158.1"}