OpenTelemetry

The opentelemetry plugin instruments APISIX and sends traces to OpenTelemetry collector based on the OpenTelemetry specification, in binary-encoded OLTP over HTTP.

Examples​

The examples below demonstrate how you can work with the opentelemetry plugin for different scenarios.

Enable opentelemetry Plugin​

If you are using API7 Enterprise, you may skip this section as there is no need to manually enable the plugin.

By default, the opentelemetry plugin is disabled in APISIX. To enable, add the plugin to your configuration file as such:

plugins:
  - ...
  - opentelemetry

Reload APISIX for changes to take effect.

See static configurations for other available options you can configure in config.yaml.

Send Traces to OpenTelemetry​

The following example demonstrates how to trace requests to a route and send traces to OpenTelemetry.

Start an OpenTelemetry collector instance in Docker:

docker run -d --name otel-collector -p 4318:4318 otel/opentelemetry-collector-contrib

The collector should start listening on 127.0.0.1:4318. If you would like to update the collector, you can configure the plugin metadata as such:

curl "http://127.0.0.1:9180/apisix/admin/plugin_metadata/opentelemetry" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d '{
    "collector": {
      "address": "127.0.0.1:4318"
    }
  }'

Create a route with opentelemetry plugin:

curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d '{
    "id": "otel-tracing-route",
    "uri": "/anything",
    "plugins": {
      "opentelemetry": {
        "sampler": {
          "name": "always_on"
        }
      }
    },
    "upstream": {
      "type": "roundrobin",
      "nodes": {
        "httpbin.org": 1
      }
    }
  }'

Send a request to the route:

curl "http://127.0.0.1:9080/anything"

You should receive an HTTP/1.1 200 OK response.

In OpenTelemetry collector's log, you should see information similar to the following:

2024-02-18T17:14:03.825Z info ResourceSpans #0
Resource SchemaURL: 
Resource attributes:
     -> telemetry.sdk.language: Str(lua)
     -> telemetry.sdk.name: Str(opentelemetry-lua)
     -> telemetry.sdk.version: Str(0.1.1)
     -> hostname: Str(e34673e24631)
     -> service.name: Str(APISIX)
ScopeSpans #0
ScopeSpans SchemaURL: 
InstrumentationScope opentelemetry-lua 
Span #0
    Trace ID       : fbd0a38d4ea4a128ff1a688197bc58b0
    Parent ID      : 
    ID             : af3dc7642104748a
    Name           : GET /anything
    Kind           : Server
    Start time     : 2024-02-18 17:14:03.763244032 +0000 UTC
    End time       : 2024-02-18 17:14:03.920229888 +0000 UTC
    Status code    : Unset
    Status message : 
Attributes:
     -> net.host.name: Str(127.0.0.1)
     -> http.method: Str(GET)
     -> http.scheme: Str(http)
     -> http.target: Str(/anything)
     -> http.user_agent: Str(curl/7.64.1)
     -> apisix.route_id: Str(otel-tracing-route)
     -> apisix.route_name: Empty()
     -> http.route: Str(/anything)
     -> http.status_code: Int(200)
{"kind": "exporter", "data_type": "traces", "name": "debug"}

To visualize these traces, you can export your telemetry to backend services, such as Zipkin and Prometheus. See exporters for more details.

Using Trace Variables in Logging​

The following example demonstrates how to configure the opentelemetry plugin to set the following built-in variables, which can be used in logger plugins or access logs:

• opentelemetry_context_traceparent: trace parent ID
• opentelemetry_trace_id: trace ID of the current span
• opentelemetry_span_id: span ID of the current span

Configure the plugin metadata to set set_ngx_var as true:

curl "http://127.0.0.1:9180/apisix/admin/plugin_metadata/opentelemetry" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d '{
    "set_ngx_var": true
  }'

Update the access log format in configuration file to use the opentelemetry plugin variables as such:

nginx_config:
  http:
    enable_access_log: true
    access_log_format: '{"time": "$time_iso8601","opentelemetry_context_traceparent": "$opentelemetry_context_traceparent","opentelemetry_trace_id": "$opentelemetry_trace_id","opentelemetry_span_id": "$opentelemetry_span_id","remote_addr": "$remote_addr"}'
    access_log_format_escape: json

Reload APISIX for configuration changes to take effect.

You should see access log entries similar to the following when you generate requests:

{"time": "18/Feb/2024:15:09:00 +0000","opentelemetry_context_traceparent": "00-fbd0a38d4ea4a128ff1a688197bc58b0-8f4b9d9970a02629-01","opentelemetry_trace_id": "fbd0a38d4ea4a128ff1a688197bc58b0","opentelemetry_span_id": "af3dc7642104748a","remote_addr": "172.10.0.1"}