OpenTelemetry

The opentelemetry plugin instruments APISIX and sends traces to OpenTelemetry collector based on the OpenTelemetry specification, in binary-encoded OLTP over HTTP.

Examples​

The examples below demonstrate how you can work with the opentelemetry plugin for different scenarios.

Enable opentelemetry Plugin​

By default, the opentelemetry plugin is disabled in APISIX. To enable, add the plugin to your configuration file as such:

plugins:
  - ...
  - opentelemetry

Configurations in config.yaml take precedence over the default configurations in config-default.yaml. For changes to take effect, reload APISIX.

See static configurations for other available options you can configure in config.yaml.

Send Traces to OpenTelemetry​

The following example demonstrates how to trace requests to a route and send traces to OpenTelemetry.

Start an OpenTelemetry collector instance in Docker:

docker run -d --name otel-collector -p 4318:4318 otel/opentelemetry-collector-contrib

Create a route with opentelemetry plugin:

curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d '{
    "id": "otel-tracing-route",
    "uri": "/anything",
    "plugins": {
      "opentelemetry": {
        "sampler": {
          "name": "always_on"
        }
      }
    },
    "upstream": {
      "type": "roundrobin",
      "nodes": {
        "httpbin.org": 1
      }
    }
  }'

Send a request to the route:

curl "http://127.0.0.1:9080/anything"

You should receive an HTTP/1.1 200 OK response.

In OpenTelemetry collector's log, you should see information similar to the following:

2024-02-18T17:14:03.825Z info ResourceSpans #0
Resource SchemaURL: 
Resource attributes:
     -> telemetry.sdk.language: Str(lua)
     -> telemetry.sdk.name: Str(opentelemetry-lua)
     -> telemetry.sdk.version: Str(0.1.1)
     -> hostname: Str(e34673e24631)
     -> service.name: Str(APISIX)
ScopeSpans #0
ScopeSpans SchemaURL: 
InstrumentationScope opentelemetry-lua 
Span #0
    Trace ID       : fbd0a38d4ea4a128ff1a688197bc58b0
    Parent ID      : 
    ID             : af3dc7642104748a
    Name           : GET /anything
    Kind           : Server
    Start time     : 2024-02-18 17:14:03.763244032 +0000 UTC
    End time       : 2024-02-18 17:14:03.920229888 +0000 UTC
    Status code    : Unset
    Status message : 
Attributes:
     -> net.host.name: Str(127.0.0.1)
     -> http.method: Str(GET)
     -> http.scheme: Str(http)
     -> http.target: Str(/anything)
     -> http.user_agent: Str(curl/7.64.1)
     -> apisix.route_id: Str(otel-tracing-route)
     -> apisix.route_name: Empty()
     -> http.route: Str(/anything)
     -> http.status_code: Int(200)
{"kind": "exporter", "data_type": "traces", "name": "debug"}

To visualize these traces, you can export your telemetry to backend services, such as Zipkin and Prometheus. See exporters for more details.

Using Trace Variables in Logging​

The following example demonstrates how to configure the opentelemetry plugin to set the following built-in variables, which can be used in logger plugins or access logs:

• opentelemetry_context_traceparent: trace parent ID
• opentelemetry_trace_id: trace ID of the current span
• opentelemetry_span_id: span ID of the current span

Update the configuration file as such:

http:
    enable_access_log: true
    access_log_format: '{"time": "$time_iso8601","opentelemetry_context_traceparent": "$opentelemetry_context_traceparent","opentelemetry_trace_id": "$opentelemetry_trace_id","opentelemetry_span_id": "$opentelemetry_span_id","remote_addr": "$remote_addr"}'
    access_log_format_escape: json
plugin_attr:
  opentelemetry:
    set_ngx_var: true

❶ access_log_format: customize the access log format to use the opentelemetry plugin variables.

❷ set_ngx_var: set opentelemetry variables.

Reload APISIX for configuration changes to take effect.

You should see access log entries similar to the following when you generate requests:

{"time": "18/Feb/2024:15:09:00 +0000","opentelemetry_context_traceparent": "00-fbd0a38d4ea4a128ff1a688197bc58b0-8f4b9d9970a02629-01","opentelemetry_trace_id": "fbd0a38d4ea4a128ff1a688197bc58b0","opentelemetry_span_id": "af3dc7642104748a","remote_addr": "172.10.0.1"}