Trace Requests with Zipkin

Traces are one of the three pillars of observability, along with metrics and logs. A trace tracks the journey of a request as it traverses through various parts of a system. It is an effective mechanism that helps developers and administrators monitor system performance, identify bottlenecks, and improve user experience.

This guide will walk you through how to trace requests to APISIX using the zipkin plugin and send traces to Zipkin.

Prerequisite(s)

• Install Docker.
• Install cURL to send requests to the services for validation.
• Follow the Getting Started tutorial to start a new APISIX instance in Docker.

Start Zipkin Instance

Start a Zipkin instance:

Docker

docker run -d --name zipkin -p 9411:9411 openzipkin/zipkin

Kubernetes

Create a Kubernetes manifest file for Zipkin:

zipkin.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: ingress-apisix
  name: zipkin
spec:
  replicas: 1
  selector:
    matchLabels:
      app: zipkin
  template:
    metadata:
  namespace: ingress-apisix
      labels:
        app: zipkin
    spec:
      containers:
        - name: zipkin
          image: openzipkin/zipkin
          ports:
            - containerPort: 9411
---
apiVersion: v1
kind: Service
metadata:
  namespace: ingress-apisix
  name: zipkin
spec:
  selector:
    app: zipkin
  ports:
    - name: http
      port: 9411
      targetPort: 9411
  type: ClusterIP

Apply the configuration to your cluster:

kubectl apply -f zipkin.yaml

Port forward the Zipkin web UI's service port to your local machine's port:

kubectl port-forward svc/zipkin 9411:9411 &

Configure APISIX

Enable zipkin globally and create a sample route to generate traces. Alternatively, you can enable the plugin on a route.

Admin API

Configure zipkin to be a global plugin:

curl "http://127.0.0.1:9180/apisix/admin/global_rules/zipkin" -X PUT -d '
{
  "plugins": {
    "zipkin": {
      "endpoint": "http://192.168.42.145:9411/api/v2/spans",
      "sample_ratio": 1
    }
  }
}'

❶ Set to the /spans Zipkin endpoint. Update with your IP address.

❷ Configure the sample ratio to 1 to trace every request.

Create a sample route where requests through the route will be traced:

curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT -d '
{
  "id": "zipkin-tracing-route",
  "uri": "/anything",
  "upstream": {
    "type": "roundrobin",
    "nodes": {
      "httpbin.org": 1
    }
  }
}'

ADC

Configure zipkin to be a global plugin and create a sample route where requests through the route will be traced:

adc-route.yaml

global_rules:
  zipkin:
    endpoint: "http://192.168.42.145:9411/api/v2/spans"
    sample_ratio: 1
services:
  - name: httpbin Service
    routes:
      - uris:
          - /anything
        name: zipkin-tracing-route
    upstream:
      type: roundrobin
      nodes:
        - host: httpbin.org
          weight: 1

❶ Set to the /spans Zipkin endpoint.

❷ Configure the sample ratio to 1 to trace every request.

Synchronize the configuration to APISIX:

adc sync -f adc-route.yaml

Ingress Controller

Gateway API

Create a Kubernetes manifest file to enable zipkin globally:

global-zipkin.yaml

apiVersion: apisix.apache.org/v1alpha1
kind: GatewayProxy
metadata:
  namespace: ingress-apisix
  name: apisix-config
spec:
  plugins:
  - name: zipkin
    enabled: true
    config:
      endpoint: http://zipkin:9411/api/v2/spans
      sample_ratio: 1

❶ Set to the /spans Zipkin endpoint.

❷ Configure the sample ratio to 1 to trace every request.

Create another Kubernetes manifest file for a sample route on which you will collect traces:

httpbin-route.yaml

apiVersion: v1
kind: Service
metadata:
  namespace: ingress-apisix
  name: httpbin-external-domain
spec:
  type: ExternalName
  externalName: httpbin.org
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  namespace: ingress-apisix
  name: zipkin-tracing-route
spec:
  parentRefs:
  - name: apisix
  rules:
  - matches:
    - path:
        type: Exact
        value: /anything
    backendRefs:
    - name: httpbin-external-domain
      port: 80

APISIX CRD

Create a Kubernetes manifest file to enable zipkin globally:

global-zipkin.yaml

apiVersion: apisix.apache.org/v2
kind: ApisixGlobalRule
metadata:
  namespace: ingress-apisix
  name: global-zipkin
spec:
  ingressClassName: apisix
  plugins:
    - name: zipkin
      enable: true
      config:
        config:
        endpoint: http://zipkin:9411/api/v2/spans
        sample_ratio: 1

❶ Set to the /spans Zipkin endpoint.

❷ Configure the sample ratio to 1 to trace every request.

Create another Kubernetes manifest file for a sample route on which you will collect traces:

httpbin-route.yaml

apiVersion: apisix.apache.org/v2
kind: ApisixUpstream
metadata:
  namespace: ingress-apisix
  name: httpbin-external-domain
spec:
  ingressClassName: apisix
  externalNodes:
  - type: Domain
    name: httpbin.org
---
apiVersion: apisix.apache.org/v2
kind: ApisixRoute
metadata:
  namespace: ingress-apisix
  name: zipkin-tracing-route
spec:
  ingressClassName: apisix
  http:
    - name: zipkin-tracing-route
      match:
        paths:
          - /anything
      upstreams:
      - name: httpbin-external-domain

Apply the configuration to your cluster:

kubectl apply -f global-zipkin.yaml -f httpbin-route.yaml

Trace Requests

Send a request to the route:

curl "http://127.0.0.1:9080/anything"

You should receive an HTTP/1.1 200 OK response similar to the following, with trace information in the headers:

{
  "args": {}, 
  "data": "", 
  "files": {}, 
  "form": {}, 
  "headers": {
    "Accept": "*/*", 
    "Host": "127.0.0.1", 
    "User-Agent": "curl/7.64.1", 
    "X-Amzn-Trace-Id": "Root=1-65b1dd40-339dc8832b4b78d36703cbc0", 
    "X-B3-Parentspanid": "6ca01ad46bdb0198", 
    "X-B3-Sampled": "1", 
    "X-B3-Spanid": "f02ab1a9b2d5c3e4", 
    "X-B3-Traceid": "ef5b16781c7ad00ea2e3efa6f784031a", 
    "X-Forwarded-Host": "127.0.0.1"
  }, 
  ...
}

Navigate to the Zipkin web UI at http://127.0.0.1:9411/zipkin and click Run Query, you should see a trace corresponding to the request:

Click Show to see more tracing details:

By default, the plugin uses span version 2, where every traced request has proxy and response spans. More specifically:

• proxy represents the time from the beginning of the request to the beginning of header_filter
• response represents the time from the beginning of header_filter to the beginning of log

To understand the differences between different span versions, see the plugin doc.

Next Steps

You have now integrated Zipkin with APISIX for tracing. In addition to sending traces to Zipkin, the zipkin plugin can also send traces to Jaeger and other compatible collectors. See the zipkin plugin doc for more details.