OpenTelemetry
The opentelemetry
plugin instruments APISIX and sends traces to OpenTelemetry collector based on the OpenTelemetry specification, in binary-encoded OLTP over HTTP.
Examples
The examples below demonstrate how you can work with the opentelemetry
plugin for different scenarios.
Enable opentelemetry
Plugin
If you are using API7 Enterprise, you may skip this section as there is no need to manually enable the plugin.
By default, the opentelemetry
plugin is disabled in APISIX. To enable, add the plugin to your configuration file as such:
plugins:
- ...
- opentelemetry
Reload APISIX for changes to take effect.
See static configurations for other available options you can configure in config.yaml
.
Send Traces to OpenTelemetry
The following example demonstrates how to trace requests to a route and send traces to OpenTelemetry.
Start an OpenTelemetry collector instance in Docker:
docker run -d --name otel-collector -p 4318:4318 otel/opentelemetry-collector-contrib
Create a route with opentelemetry
plugin:
curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
-H "X-API-KEY: ${ADMIN_API_KEY}" \
-d '{
"id": "otel-tracing-route",
"uri": "/anything",
"plugins": {
"opentelemetry": {
"sampler": {
"name": "always_on"
}
}
},
"upstream": {
"type": "roundrobin",
"nodes": {
"httpbin.org": 1
}
}
}'
Send a request to the route:
curl "http://127.0.0.1:9080/anything"
You should receive an HTTP/1.1 200 OK
response.
In OpenTelemetry collector's log, you should see information similar to the following:
2024-02-18T17:14:03.825Z info ResourceSpans #0
Resource SchemaURL:
Resource attributes:
-> telemetry.sdk.language: Str(lua)
-> telemetry.sdk.name: Str(opentelemetry-lua)
-> telemetry.sdk.version: Str(0.1.1)
-> hostname: Str(e34673e24631)
-> service.name: Str(APISIX)
ScopeSpans #0
ScopeSpans SchemaURL:
InstrumentationScope opentelemetry-lua
Span #0
Trace ID : fbd0a38d4ea4a128ff1a688197bc58b0
Parent ID :
ID : af3dc7642104748a
Name : GET /anything
Kind : Server
Start time : 2024-02-18 17:14:03.763244032 +0000 UTC
End time : 2024-02-18 17:14:03.920229888 +0000 UTC
Status code : Unset
Status message :
Attributes:
-> net.host.name: Str(127.0.0.1)
-> http.method: Str(GET)
-> http.scheme: Str(http)
-> http.target: Str(/anything)
-> http.user_agent: Str(curl/7.64.1)
-> apisix.route_id: Str(otel-tracing-route)
-> apisix.route_name: Empty()
-> http.route: Str(/anything)
-> http.status_code: Int(200)
{"kind": "exporter", "data_type": "traces", "name": "debug"}
To visualize these traces, you can export your telemetry to backend services, such as Zipkin and Prometheus. See exporters for more details.
Using Trace Variables in Logging
The following example demonstrates how to configure the opentelemetry
plugin to set the following built-in variables, which can be used in logger plugins or access logs:
opentelemetry_context_traceparent
: trace parent IDopentelemetry_trace_id
: trace ID of the current spanopentelemetry_span_id
: span ID of the current span
Update the configuration file as such:
nginx_config:
http:
enable_access_log: true
access_log_format: '{"time": "$time_iso8601","opentelemetry_context_traceparent": "$opentelemetry_context_traceparent","opentelemetry_trace_id": "$opentelemetry_trace_id","opentelemetry_span_id": "$opentelemetry_span_id","remote_addr": "$remote_addr"}'
access_log_format_escape: json
plugin_attr:
opentelemetry:
set_ngx_var: true
❶ access_log_format
: customize the access log format to use the opentelemetry
plugin variables.
❷ set_ngx_var
: set opentelemetry
variables.
Reload APISIX for configuration changes to take effect.
If you are using API7 Enterprise, you should configure the parameters under plugin_attr
in the dashboard's Plugin Settings > Plugin Metadata, in JSON format. See Configure OpenTelemetry Metadata for more details.
You should see access log entries similar to the following when you generate requests:
{"time": "18/Feb/2024:15:09:00 +0000","opentelemetry_context_traceparent": "00-fbd0a38d4ea4a128ff1a688197bc58b0-8f4b9d9970a02629-01","opentelemetry_trace_id": "fbd0a38d4ea4a128ff1a688197bc58b0","opentelemetry_span_id": "af3dc7642104748a","remote_addr": "172.10.0.1"}