zipkin

Zipkin is an open-source distributed tracing system. The zipkin plugin instruments APISIX and sends traces to Zipkin based on the Zipkin API specification.

The plugin can also send traces to other compatible collectors, such as Jaeger and Apache SkyWalking, both of which support Zipkin v1 and v2 APIs.

Examples

The examples below show different use cases for the zipkin plugin.

Send Traces to Zipkin

The following example demonstrates how to trace requests to a route and send traces to Zipkin using Zipkin API v2. You will also understand the differences between span version 2 and span version 1.

Start a Zipkin instance:

Docker
Kubernetes

docker run -d --name zipkin -p 9411:9411 openzipkin/zipkin

zipkin-server.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: aic
  name: zipkin
spec:
  replicas: 1
  selector:
    matchLabels:
      app: zipkin
  template:
    metadata:
      labels:
        app: zipkin
    spec:
      containers:
      - name: zipkin
        image: openzipkin/zipkin
        ports:
        - containerPort: 9411
---
apiVersion: v1
kind: Service
metadata:
  namespace: aic
  name: zipkin
spec:
  selector:
    app: zipkin
  ports:
  - port: 9411
    targetPort: 9411
  type: ClusterIP

Apply the manifest:

kubectl apply -f zipkin-server.yaml

Create a route with zipkin and use the default span version 2:

Admin API
ADC
Ingress Controller

curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d '{
    "id": "zipkin-tracing-route",
    "uri": "/anything",
    "plugins": {
      "zipkin": {
        "endpoint": "http://127.0.0.1:9411/api/v2/spans",
        "sample_ratio": 1,
        "span_version": 2
      }
    },
    "upstream": {
      "type": "roundrobin",
      "nodes": {
        "httpbin.org": 1
      }
    }
  }'

adc.yaml
services:
  - name: httpbin
    routes:
      - uris:
          - /anything
        name: zipkin-tracing-route
        plugins:
          zipkin:
            endpoint: "http://127.0.0.1:9411/api/v2/spans"
            sample_ratio: 1
            span_version: 2
    upstream:
      type: roundrobin
      nodes:
        - host: httpbin.org
          port: 80
          weight: 1

Synchronize the configuration to the gateway:

adc sync -f adc.yaml

Gateway API
APISIX CRD

zipkin-ic.yaml
apiVersion: v1
kind: Service
metadata:
  namespace: aic
  name: httpbin-external-domain
spec:
  type: ExternalName
  externalName: httpbin.org
---
apiVersion: apisix.apache.org/v1alpha1
kind: PluginConfig
metadata:
  namespace: aic
  name: zipkin-plugin-config
spec:
  plugins:
    - name: zipkin
      config:
        endpoint: "http://zipkin.aic.svc.cluster.local:9411/api/v2/spans"
        sample_ratio: 1
        span_version: 2
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  namespace: aic
  name: zipkin-route
spec:
  parentRefs:
    - name: apisix
  rules:
    - matches:
        - path:
            type: Exact
            value: /anything
      filters:
        - type: ExtensionRef
          extensionRef:
            group: apisix.apache.org
            kind: PluginConfig
            name: zipkin-plugin-config
      backendRefs:
        - name: httpbin-external-domain
          port: 80

zipkin-ic.yaml
apiVersion: apisix.apache.org/v2
kind: ApisixUpstream
metadata:
  namespace: aic
  name: httpbin-external-domain
spec:
  ingressClassName: apisix
  externalNodes:
  - type: Domain
    name: httpbin.org
---
apiVersion: apisix.apache.org/v2
kind: ApisixRoute
metadata:
  namespace: aic
  name: zipkin-route
spec:
  ingressClassName: apisix
  http:
    - name: zipkin-route
      match:
        paths:
          - /anything
      upstreams:
      - name: httpbin-external-domain
      plugins:
      - name: zipkin
        enable: true
        config:
          endpoint: "http://zipkin.aic.svc.cluster.local:9411/api/v2/spans"
          sample_ratio: 1
          span_version: 2

Apply the configuration to your cluster:

kubectl apply -f zipkin-ic.yaml

❶ Adjust the endpoint URL as needed.

❷ Configure the sample ratio to 1 to trace every request.

❸ Set span version to 2.

Send a request to the route:

curl "http://127.0.0.1:9080/anything"

You should receive an HTTP/1.1 200 OK response similar to the following:

{
  "args": {}, 
  "data": "", 
  "files": {}, 
  "form": {}, 
  "headers": {
    "Accept": "*/*", 
    "Host": "127.0.0.1", 
    "User-Agent": "curl/7.64.1", 
    "X-Amzn-Trace-Id": "Root=1-65af2926-497590027bcdb09e34752b78", 
    "X-B3-Parentspanid": "347dddedf73ec176", 
    "X-B3-Sampled": "1", 
    "X-B3-Spanid": "429afa01d0b0067c", 
    "X-B3-Traceid": "aea58f4b490766eccb08275acd52a13a", 
    "X-Forwarded-Host": "127.0.0.1"
  }, 
  ...
}

Navigate to the Zipkin web UI at http://127.0.0.1:9411/zipkin and click Run Query, you should see a trace corresponding to the request:

trace-from-request

Click Show to see more tracing details:

v2-trace-spans

Note that with span version 2, every traced request creates the following spans:

request
├── proxy
└── response

where proxy represents the time from the beginning of the request to the beginning of header_filter, and response represents the time from the beginning of header_filter to the beginning of log.

In API7 Enterprise (from version 3.9.10), the request span includes an apisix.response_source tag that classifies the response origin as apisix (generated by APISIX, such as plugin rejections), nginx (NGINX proxy errors), or upstream (real response from the upstream service). This tag is not available in APISIX yet.

Now, update the plugin on the route to use span version 1:

Admin API
ADC
Ingress Controller

curl "http://127.0.0.1:9180/apisix/admin/routes/zipkin-tracing-route" -X PATCH \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d '{
    "plugins": {
      "zipkin": {
        "span_version": 1
      }
    }
  }'

Update adc.yaml to set span_version to 1:

adc.yaml
services:
  - name: httpbin
    routes:
      - uris:
          - /anything
        name: zipkin-tracing-route
        plugins:
          zipkin:
            endpoint: "http://127.0.0.1:9411/api/v2/spans"
            sample_ratio: 1
            span_version: 1
    upstream:
      type: roundrobin
      nodes:
        - host: httpbin.org
          port: 80
          weight: 1

Synchronize the configuration to the gateway:

adc sync -f adc.yaml

Gateway API
APISIX CRD

Update zipkin-ic.yaml to set span_version to 1:

zipkin-ic.yaml
apiVersion: v1
kind: Service
metadata:
  namespace: aic
  name: httpbin-external-domain
spec:
  type: ExternalName
  externalName: httpbin.org
---
apiVersion: apisix.apache.org/v1alpha1
kind: PluginConfig
metadata:
  namespace: aic
  name: zipkin-plugin-config
spec:
  plugins:
    - name: zipkin
      config:
        endpoint: "http://zipkin.aic.svc.cluster.local:9411/api/v2/spans"
        sample_ratio: 1
        span_version: 1
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  namespace: aic
  name: zipkin-route
spec:
  parentRefs:
    - name: apisix
  rules:
    - matches:
        - path:
            type: Exact
            value: /anything
      filters:
        - type: ExtensionRef
          extensionRef:
            group: apisix.apache.org
            kind: PluginConfig
            name: zipkin-plugin-config
      backendRefs:
        - name: httpbin-external-domain
          port: 80

Update zipkin-ic.yaml to set span_version to 1:

zipkin-ic.yaml
apiVersion: apisix.apache.org/v2
kind: ApisixUpstream
metadata:
  namespace: aic
  name: httpbin-external-domain
spec:
  ingressClassName: apisix
  externalNodes:
  - type: Domain
    name: httpbin.org
---
apiVersion: apisix.apache.org/v2
kind: ApisixRoute
metadata:
  namespace: aic
  name: zipkin-route
spec:
  ingressClassName: apisix
  http:
    - name: zipkin-route
      match:
        paths:
          - /anything
      upstreams:
      - name: httpbin-external-domain
      plugins:
      - name: zipkin
        enable: true
        config:
          endpoint: "http://zipkin.aic.svc.cluster.local:9411/api/v2/spans"
          sample_ratio: 1
          span_version: 1

Reapply the configuration:

kubectl apply -f zipkin-ic.yaml

Send another request to the route:

curl "http://127.0.0.1:9080/anything"

In the Zipkin web UI, you should see a new trace with details similar to the following:

v1-trace-spans

Note that with the older span version 1, every traced request creates the following spans:

request
├── rewrite
├── access
└── proxy
    └── body_filter

Send Traces to Jaeger

The following example demonstrates how to trace requests to a route and send traces to Jaeger.

Start a Jaeger instance:

Docker
Kubernetes

docker run -d --name jaeger \
  -e COLLECTOR_ZIPKIN_HOST_PORT=9411 \
  -p 16686:16686 \
  -p 9411:9411 \
  jaegertracing/all-in-one

jaeger-server.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: aic
  name: jaeger
spec:
  replicas: 1
  selector:
    matchLabels:
      app: jaeger
  template:
    metadata:
      labels:
        app: jaeger
    spec:
      containers:
      - name: jaeger
        image: jaegertracing/all-in-one
        env:
        - name: COLLECTOR_ZIPKIN_HOST_PORT
          value: "9411"
        ports:
        - containerPort: 16686
        - containerPort: 9411
---
apiVersion: v1
kind: Service
metadata:
  namespace: aic
  name: jaeger
spec:
  selector:
    app: jaeger
  ports:
  - name: ui
    port: 16686
    targetPort: 16686
  - name: zipkin
    port: 9411
    targetPort: 9411
  type: ClusterIP

Apply the manifest:

kubectl apply -f jaeger-server.yaml

Create a route with zipkin:

Admin API
ADC
Ingress Controller

curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d '{
    "id": "zipkin-tracing-route",
    "uri": "/anything",
    "plugins": {
      "zipkin": {
        "endpoint": "http://127.0.0.1:9411/api/v2/spans",
        "sample_ratio": 1
      }
    },
    "upstream": {
      "type": "roundrobin",
      "nodes": {
        "httpbin.org": 1
      }
    }
  }'

adc.yaml
services:
  - name: httpbin
    routes:
      - uris:
          - /anything
        name: zipkin-tracing-route
        plugins:
          zipkin:
            endpoint: "http://127.0.0.1:9411/api/v2/spans"
            sample_ratio: 1
    upstream:
      type: roundrobin
      nodes:
        - host: httpbin.org
          port: 80
          weight: 1

Synchronize the configuration to the gateway:

adc sync -f adc.yaml

Gateway API
APISIX CRD

zipkin-jaeger-ic.yaml
apiVersion: v1
kind: Service
metadata:
  namespace: aic
  name: httpbin-external-domain
spec:
  type: ExternalName
  externalName: httpbin.org
---
apiVersion: apisix.apache.org/v1alpha1
kind: PluginConfig
metadata:
  namespace: aic
  name: zipkin-jaeger-plugin-config
spec:
  plugins:
    - name: zipkin
      config:
        endpoint: "http://jaeger.aic.svc.cluster.local:9411/api/v2/spans"
        sample_ratio: 1
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  namespace: aic
  name: zipkin-jaeger-route
spec:
  parentRefs:
    - name: apisix
  hostnames:
    - "jaeger.example.com"
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /
      filters:
        - type: ExtensionRef
          extensionRef:
            group: apisix.apache.org
            kind: PluginConfig
            name: zipkin-jaeger-plugin-config
      backendRefs:
        - name: httpbin-external-domain
          port: 80

zipkin-jaeger-ic.yaml
apiVersion: apisix.apache.org/v2
kind: ApisixUpstream
metadata:
  namespace: aic
  name: httpbin-external-domain
spec:
  ingressClassName: apisix
  externalNodes:
  - type: Domain
    name: httpbin.org
---
apiVersion: apisix.apache.org/v2
kind: ApisixRoute
metadata:
  namespace: aic
  name: zipkin-jaeger-route
spec:
  ingressClassName: apisix
  http:
    - name: zipkin-jaeger-route
      match:
        hosts:
          - "jaeger.example.com"
        paths:
          - /*
      upstreams:
      - name: httpbin-external-domain
      plugins:
      - name: zipkin
        enable: true
        config:
          endpoint: "http://jaeger.aic.svc.cluster.local:9411/api/v2/spans"
          sample_ratio: 1

Apply the configuration to your cluster:

kubectl apply -f zipkin-jaeger-ic.yaml

❶ Adjust the endpoint URL as needed.

❷ Configure the sample ratio to 1 to trace every request.

Send a request to the route:

Admin API
ADC
Ingress Controller

curl "http://127.0.0.1:9080/anything"

curl "http://127.0.0.1:9080/anything"

curl "http://127.0.0.1:9080/anything" -H "Host: jaeger.example.com"

You should receive an HTTP/1.1 200 OK response.

Navigate to the Jaeger web UI at http://127.0.0.1:16686, select APISIX as the service, and click Find Traces, you should see a trace corresponding to the request:

jaeger-traces

Similarly, you should find more span details once you click into a trace:

jaeger-details

Using Trace Variables in Logging

The following example demonstrates how to configure the zipkin plugin to set the following built-in variables, which can be used in logger plugins or access logs:

zipkin_context_traceparent: trace parent ID
zipkin_trace_id: trace ID of the current span
zipkin_span_id: span ID of the current span

Update the configuration file as such:

conf/config.yaml
nginx_config:
  http:
    enable_access_log: true
    access_log_format: '{"time": "$time_iso8601","zipkin_context_traceparent": "$zipkin_context_traceparent","zipkin_trace_id": "$zipkin_trace_id","zipkin_span_id": "$zipkin_span_id","remote_addr": "$remote_addr"}'
    access_log_format_escape: json
plugin_attr:
  zipkin:
    set_ngx_var: true

❶ access_log_format: customize the access log format to use the zipkin plugin variables.

❷ set_ngx_var: set zipkin variables.

Reload APISIX for configuration changes to take effect.

Ingress Controller

If you are using the Ingress Controller with Helm, update the access log format and zipkin plugin attributes in the Helm values file and run helm upgrade. For more information, see the Helm chart values reference.

You should see access log entries similar to the following when you generate requests:

{"time": "23/Jan/2024:06:28:00 +0000","zipkin_context_traceparent": "00-61bce33055c56f5b9bec75227befd142-13ff3c7370b29925-01","zipkin_trace_id": "61bce33055c56f5b9bec75227befd142","zipkin_span_id": "13ff3c7370b29925","remote_addr": "172.28.0.1"}

Examples​

Send Traces to Zipkin​

Send Traces to Jaeger​

Using Trace Variables in Logging​

Examples

Send Traces to Zipkin

Send Traces to Jaeger

Using Trace Variables in Logging