Trace Requests with Zipkin

Distributed traces show how requests move through a system and where they spend time. APISIX can instrument gateway requests with the zipkin plugin and export sampled spans through the Zipkin v2 HTTP API.

This guide configures the plugin globally, proxies a request through a sample route, and verifies the resulting trace in Zipkin.

Prerequisites

• Install Docker.
• Install cURL to send requests and verify the trace.
• Complete the APISIX Getting Started tutorial to start the Docker Quickstart. The Quickstart creates the apisix-quickstart-net network used below.

Start Zipkin

Start a pinned Zipkin instance on the same Docker network as APISIX. Publish the Zipkin UI only on the loopback interface:

docker run -d --name zipkin \
  --network apisix-quickstart-net \
  -p 127.0.0.1:9411:9411 \
  openzipkin/zipkin:3.6.1

Wait for Zipkin to become ready:

until curl -fsS "http://127.0.0.1:9411/health" > /dev/null; do
  sleep 1
done

Local evaluation setup

This Zipkin instance uses transient in-memory storage without authentication or TLS. For production, configure durable storage and secure the collector endpoint.

Configure APISIX

Configure zipkin in a global rule so the plugin applies to all routed requests, then create a sample route. To limit tracing to selected traffic, configure the plugin on individual routes or services instead.

The examples use the fixed identifiers zipkin, zipkin-tracing-route, and httpbin. Confirm that they are unused, or replace them consistently throughout the configuration before applying it.

Admin API

Create a global rule that sends every sampled span to the Zipkin container:

curl "http://127.0.0.1:9180/apisix/admin/global_rules/zipkin" -X PUT \
  -d '{
    "plugins": {
      "zipkin": {
        "endpoint": "http://zipkin:9411/api/v2/spans",
        "sample_ratio": 1
      }
    }
  }'

Create the sample route:

curl "http://127.0.0.1:9180/apisix/admin/routes/zipkin-tracing-route" -X PUT \
  -d '{
    "uri": "/anything",
    "upstream": {
      "type": "roundrobin",
      "nodes": {
        "httpbin.org:80": 1
      }
    }
  }'

ADC

Add the global rule and sample service to your existing ADC configuration so other services and global rules remain in the desired state:

adc.yaml

global_rules:
  zipkin:
    endpoint: "http://zipkin:9411/api/v2/spans"
    sample_ratio: 1
services:
  - name: httpbin
    routes:
      - uris:
          - /anything
        name: zipkin-tracing-route
    upstream:
      type: roundrobin
      nodes:
        - host: httpbin.org
          port: 80
          weight: 1

Preview the reconciliation and confirm that it contains no unintended updates or deletions:

adc diff -f adc.yaml \
  --include-resource-type service \
  --include-resource-type global_rule

Synchronize the reviewed configuration:

adc sync -f adc.yaml \
  --include-resource-type service \
  --include-resource-type global_rule

❶ Send spans to the Zipkin v2 HTTP endpoint reachable from the APISIX container.

❷ Sample every request for this local example unless the request carries a B3 decision not to sample. Use a lower sample ratio for high-throughput production traffic when full sampling is unnecessary.

Verify the Trace

Send a request through APISIX:

curl "http://127.0.0.1:9080/anything"

You should receive an HTTP/1.1 200 OK response. The JSON body should show the B3 tracing headers that APISIX added to the request sent upstream:

{
  "args": {},
  "data": "",
  "files": {},
  "form": {},
  "headers": {
    "Accept": "*/*",
    "Host": "127.0.0.1",
    "User-Agent": "curl/8.7.1",
    "X-Amzn-Trace-Id": "Root=1-6aa8b24c-29cf898f18bbdf415ccd7b42",
    "X-B3-Parentspanid": "e1a7df5f617b3a04",
    "X-B3-Sampled": "1",
    "X-B3-Spanid": "c3a11943cd04ec70",
    "X-B3-Traceid": "b06793db23230f51187a9638e08fb772",
    "X-Forwarded-Host": "127.0.0.1:9080"
  },
  "json": null,
  "method": "GET",
  "url": "http://127.0.0.1:9080/anything"
}

Wait for the asynchronously exported trace to become available:

trace_ready=false
for attempt in $(seq 1 20); do
  if curl -fsS \
    "http://127.0.0.1:9411/api/v2/traces?serviceName=apisix&limit=10" | \
    grep -q '"traceId"'; then
    trace_ready=true
    break
  fi
  sleep 1
done
[ "$trace_ready" = true ]

Navigate to the Zipkin UI at http://127.0.0.1:9411/zipkin and select Run Query. You should see the request trace:

Open the trace to inspect its spans:

For the successfully proxied request in this example, the default span hierarchy contains:

apisix.request
├── apisix.proxy
└── apisix.response_span

The apisix.proxy span covers the beginning of the request through the start of the NGINX header_filter phase. The apisix.response_span span covers the start of header_filter through the start of the log phase.

The request span also includes an apisix.response_source tag that identifies whether the response came from APISIX, NGINX, or the upstream service. For the full plugin configuration and span-version comparison, see the zipkin plugin reference.

Next Steps

• Send traces to Jaeger through Jaeger v2's built-in Zipkin receiver.
• Use trace variables in logs to correlate access logs with request traces.