zipkin

Zipkin is an open-source distributed tracing system. The zipkin plugin instruments APISIX and sends traces to Zipkin based on the Zipkin API specification.

The plugin can also sends traces to other compatible collectors, such as Jaeger and Apache SkyWalking, both of which support Zipkin v1 and v2 APIs.

Examples​

The examples below shows different use cases for using the zipkin plugin.

Send Traces to Zipkin​

The following example demonstrates how to trace requests to a route and send traces to Zipkin using Zipkin API v2. You will also understand the differences between span version 2 and span version 1.

Start a Zipkin instance in Docker:

docker run -d --name zipkin -p 9411:9411 openzipkin/zipkin

Create a route with zipkin and use the default span version 2:

curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d '{
    "id": "zipkin-tracing-route",
    "uri": "/anything",
    "plugins": {
      "zipkin": {
        "endpoint": "http://127.0.0.1:9411/api/v2/spans",
        "sample_ratio": 1,
        "span_version": 2
      }
    },
    "upstream": {
      "type": "roundrobin",
      "nodes": {
        "httpbin.org": 1
      }
    }
  }'

❶ Adjust the IP address as needed for Zipkin HTTP endpoint.

❷ Configure the sample ratio to 1 to trace every request.

❸ Set span version to 2.

Send a request to the route:

curl "http://127.0.0.1:9080/anything"

You should receive an HTTP/1.1 200 OK response similar to the following:

{
  "args": {}, 
  "data": "", 
  "files": {}, 
  "form": {}, 
  "headers": {
    "Accept": "*/*", 
    "Host": "127.0.0.1", 
    "User-Agent": "curl/7.64.1", 
    "X-Amzn-Trace-Id": "Root=1-65af2926-497590027bcdb09e34752b78", 
    "X-B3-Parentspanid": "347dddedf73ec176", 
    "X-B3-Sampled": "1", 
    "X-B3-Spanid": "429afa01d0b0067c", 
    "X-B3-Traceid": "aea58f4b490766eccb08275acd52a13a", 
    "X-Forwarded-Host": "127.0.0.1"
  }, 
  ...
}

Navigate to the Zipkin web UI at http://127.0.0.1:9411/zipkin and click Run Query, you should see a trace corresponding to the request:

Click Show to see more tracing details:

Note that with span version 2, every traced request creates the following spans:

request
├── proxy
└── response

where proxy represents the time from the beginning of the request to the beginning of header_filter, and response represents the time from the beginning of header_filter to the beginning of log.

Now, update the plugin on the route to use span version 1:

curl "http://127.0.0.1:9180/apisix/admin/routes/zipkin-tracing-route" -X PATCH \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d '{
    "plugins": {
      "zipkin": {
        "span_version": 1
      }
    }
  }'

Send another request to the route:

curl "http://127.0.0.1:9080/anything"

In the Zipkin web UI, you should a new trace with details similar to the following:

Note that with the older span version 1, every traced request creates the following spans:

request
├── rewrite
├── access
└── proxy
    └── body_filter

Send Traces to Jaeger​

The following example demonstrates how to trace requests to a route and send traces to Jaeger.

Start a Jaeger instance in Docker:

docker run -d --name jaeger \
  -e COLLECTOR_ZIPKIN_HOST_PORT=9411 \
  -p 16686:16686 \
  -p 9411:9411 \
  jaegertracing/all-in-one

Create a route with zipkin:

curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d '{
    "id": "zipkin-tracing-route",
    "uri": "/anything",
    "plugins": {
      "zipkin": {
        "endpoint": "http://127.0.0.1:9411/api/v2/spans",
        "sample_ratio": 1
      }
    },
    "upstream": {
      "type": "roundrobin",
      "nodes": {
        "httpbin.org": 1
      }
    }
  }'

❶ Adjust the IP address as needed for Zipkin HTTP endpoint.

❷ Configure the sample ratio to 1 to trace every request.

Send a request to the route:

curl "http://127.0.0.1:9080/anything"

You should receive an HTTP/1.1 200 OK response.

Navigate to the Jaeger web UI at http://127.0.0.1:16686, select APISIX as the service, and click Find Traces, you should see a trace corresponding to the request:

Similarly, you should find more span details once you click into a trace:

Using Trace Variables in Logging​

The following example demonstrates how to configure the zipkin plugin to set the following built-in variables, which can be used in logger plugins or access logs:

• zipkin_context_traceparent: trace parent ID
• zipkin_trace_id: trace ID of the current span
• zipkin_span_id: span ID of the current span

Update the configuration file as such:

nginx_config:
  http:
    enable_access_log: true
    access_log_format: '{"time": "$time_iso8601","zipkin_context_traceparent": "$zipkin_context_traceparent","zipkin_trace_id": "$zipkin_trace_id","zipkin_span_id": "$zipkin_span_id","remote_addr": "$remote_addr"}'
    access_log_format_escape: json
plugin_attr:
  zipkin:
    set_ngx_var: true

❶ access_log_format: customize the access log format to use the zipkin plugin variables.

❷ set_ngx_var: set zipkin variables.

Reload APISIX for configuration changes to take effect.

You should see access log entries similar to the following when you generate requests:

{"time": "23/Jan/2024:06:28:00 +0000","zipkin_context_traceparent": "00-61bce33055c56f5b9bec75227befd142-13ff3c7370b29925-01","zipkin_trace_id": "61bce33055c56f5b9bec75227befd142","zipkin_span_id": "13ff3c7370b29925","remote_addr": "172.28.0.1"}