Version: 3.13.0

Canary Deployment

A canary deployment is a strategy for rolling out a new release by gradually increasing the traffic directed to it. This strategy can help test the new release with real traffic to identify and fix any issues before making it generally available.

Canary deployments using Apache APISIX

This guide will walk you through configuring canary deployments in APISIX using the traffic-split plugin.

Configure Canary Deployment

You will be using httpbin.org and mock.api7.ai as the old and new services.

First, direct all traffic to your old service. To do this, create a route with the following configuration:

Admin API
ADC
Ingress Controller

curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d '{
    "uri": "/headers",
    "id": "canary-deployment",
    "plugins": {
      "traffic-split": {
        "rules": [
          {
            "weighted_upstreams": [
              {
                "upstream": {
                  "type": "roundrobin",
                  "scheme": "https",
                  "pass_host": "node",
                  "nodes": {
                    "httpbin.org:443":1
                  }
                },
                "weight": 100
              },
              {
                "weight": 0
              }
            ]
          }
        ]
      }
    },
    "upstream": {
      "type": "roundrobin",
      "scheme": "https",
      "pass_host": "node",
      "nodes": {
        "mock.api7.ai:443":1
      }
    }
  }'

❶ 100% of the requests should be routed to httpbin.org.

❷ 0% of the requests should be routed to mock.api7.ai.

apisix.yaml
routes:
  - uris:
      - /headers
    name: canary-deployment
    plugins:
      traffic-split:
        rules:
          - weighted_upstreams:
              - weight: 100
                upstream:
                  type: roundrobin
                  pass_host: node
                  nodes:
                    httpbin.org:443: 1
                  scheme: https
              - weight: 0
    upstream:
      type: roundrobin
      pass_host: node
      nodes:
        mock.api7.ai:443: 1
      scheme: https

❶ 100% of the requests should be routed to httpbin.org.

❷ 0% of the requests should be routed to mock.api7.ai.

Synchronize the configuration to APISIX:

adc sync -f apisix.yaml

Gateway API
APISIX CRD

canary-deployment.yaml
apiVersion: apisix.apache.org/v1alpha1
kind: PluginConfig
metadata:
  namespace: ingress-apisix
  name: traffic-split-plugin-config
spec:
  plugins:
    - name: traffic-split
      config:
        rules:
          - weighted_upstreams:
            - upstream:
                scheme: https
                pass_host: node
                nodes:
                  httpbin.org:443: 1
              weight: 100
            - weight: 0
---
apiVersion: v1
kind: Service
metadata:
  namespace: ingress-apisix
  name: mockapi7-external-domain
spec:
  type: ExternalName
  externalName: mock.api7.ai
---
apiVersion: apisix.apache.org/v1alpha1
kind: BackendTrafficPolicy
metadata:
  namespace: ingress-apisix
  name: passhost-node
spec:
  targetRefs:
  - name: mockapi7-external-domain
    kind: Service
    group: ""
  passHost: node
  scheme: https
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  namespace: ingress-apisix
  name: canary-deployment
spec:
  parentRefs:
  - name: apisix
  rules:
  - matches:
    - path:
        type: Exact
        value: /headers
    filters:
    - type: ExtensionRef
      extensionRef:
        group: apisix.apache.org
        kind: PluginConfig
        name: traffic-split-plugin-config
    backendRefs:
    - name: mockapi7-external-domain
      port: 443
      weight: 1

canary-deployment.yaml
apiVersion: apisix.apache.org/v2
kind: ApisixUpstream
metadata:
  namespace: ingress-apisix
  name: mockapi7-external-domain
spec:
  ingressClassName: apisix
  scheme: https
  passHost: node
  externalNodes:
  - type: Domain
    name: mock.api7.ai
    weight: 1
    port: 443
---
apiVersion: apisix.apache.org/v2
kind: ApisixPluginConfig
metadata:
  namespace: ingress-apisix
  name: traffic-split-plugin-config
spec:
  ingressClassName: apisix
  plugins:
    - name: traffic-split
      enable: true
      config:
        rules:
          - weighted_upstreams:
            - upstream:
                scheme: https
                pass_host: node
                nodes:
                  httpbin.org:443: 1
              weight: 100
            - weight: 0
---
apiVersion: apisix.apache.org/v2
kind: ApisixRoute
metadata:
  namespace: ingress-apisix
  name: canary-deployment
spec:
  ingressClassName: apisix
  http:
    - name: canary-deployment
      match:
        paths:
          - /headers
      plugin_config_name: traffic-split-plugin-config
      upstreams:
        - name: mockapi7-external-domain

❶ 100% of the requests should be routed to httpbin.org.

❷ 0% of the requests should be routed to mock.api7.ai.

Apply the configuration to your cluster:

kubectl apply -f canary-deployment.yaml

So, if you send 100 requests, APISIX will direct them all to httpbin.org:

resp=$(seq 100 | xargs -I{} curl "http://127.0.0.1:9080/headers" -sL) && \
  count_httpbin=$(echo "$resp" | grep "httpbin.org" | wc -l) && \
  count_mockapi7=$(echo "$resp" | grep "mock.api7.ai" | wc -l) && \
  echo httpbin.org: $count_httpbin, mock.api7.ai: $count_mockapi7

You will get the following response:

httpbin.org: 100, mock.api7.ai: 0

Next, update the configuration to direct 5% of the requests to the new service:

Admin API
ADC
Ingress Controller

curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d '{
    "uri": "/headers",
    "id": "canary-deployment"
    "plugins": {
      "traffic-split": {
        "rules": [
          {
            "weighted_upstreams": [
              {
                "upstream": {
                  "type": "roundrobin",
                  "scheme": "https",
                  "pass_host": "node",
                  "nodes": {
                    "httpbin.org:443":1
                  }
                },
                "weight": 95
              },
              {
                "weight": 5
              }
            ]
          }
        ]
      }
    },
    "upstream": {
      "type": "roundrobin",
      "scheme": "https",
      "pass_host": "node",
      "nodes": {
        "mock.api7.ai:443":1
      }
    }
  }'

apisix.yaml
routes:
  - uris:
      - /headers
    name: canary-deployment
    plugins:
      traffic-split:
        rules:
          - weighted_upstreams:
              - weight: 95
                upstream:
                  type: roundrobin
                  pass_host: node
                  nodes:
                    httpbin.org:443: 1
                  scheme: https
              - weight: 5
    upstream:
      type: roundrobin
      pass_host: node
      nodes:
        mock.api7.ai:443: 1
      scheme: https

Synchronize the configuration to APISIX:

adc sync -f apisix.yaml

Gateway API
APISIX CRD

canary-deployment.yaml
apiVersion: apisix.apache.org/v1alpha1
kind: PluginConfig
metadata:
  namespace: ingress-apisix
  name: traffic-split-plugin-config
spec:
  plugins:
    - name: traffic-split
      config:
        rules:
          - weighted_upstreams:
            - upstream:
                scheme: https
                pass_host: node
                nodes:
                  httpbin.org:443: 1
              weight: 95
            - weight: 5
---
apiVersion: v1
kind: Service
metadata:
  namespace: ingress-apisix
  name: mockapi7-external-domain
spec:
  type: ExternalName
  externalName: mock.api7.ai
---
apiVersion: apisix.apache.org/v1alpha1
kind: BackendTrafficPolicy
metadata:
  namespace: ingress-apisix
  name: passhost-node
spec:
  targetRefs:
  - name: mockapi7-external-domain
    kind: Service
    group: ""
  passHost: node
  scheme: https
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  namespace: ingress-apisix
  name: canary-deployment
spec:
  parentRefs:
  - name: apisix
  rules:
  - matches:
    - path:
        type: Exact
        value: /headers
    filters:
    - type: ExtensionRef
      extensionRef:
        group: apisix.apache.org
        kind: PluginConfig
        name: traffic-split-plugin-config
    backendRefs:
    - name: mockapi7-external-domain
      port: 443
      weight: 1

canary-deployment.yaml
apiVersion: apisix.apache.org/v2
kind: ApisixUpstream
metadata:
  namespace: ingress-apisix
  name: mockapi7-external-domain
spec:
  ingressClassName: apisix
  scheme: https
  passHost: node
  externalNodes:
  - type: Domain
    name: mock.api7.ai
    weight: 1
    port: 443
---
apiVersion: apisix.apache.org/v2
kind: ApisixPluginConfig
metadata:
  namespace: ingress-apisix
  name: traffic-split-plugin-config
spec:
  ingressClassName: apisix
  plugins:
    - name: traffic-split
      enable: true
      config:
        rules:
          - weighted_upstreams:
            - upstream:
                scheme: https
                pass_host: node
                nodes:
                  httpbin.org:443: 1
              weight: 95
            - weight: 5
---
apiVersion: apisix.apache.org/v2
kind: ApisixRoute
metadata:
  namespace: ingress-apisix
  name: canary-deployment
spec:
  ingressClassName: apisix
  http:
    - name: canary-deployment
      match:
        paths:
          - /headers
      plugin_config_name: traffic-split-plugin-config
      upstreams:
        - name: mockapi7-external-domain

Apply the configuration to your cluster:

kubectl apply -f canary-deployment.yaml

Now, if you send 100 requests, 5 of them will be directed to mock.api7.ai:

resp=$(seq 100 | xargs -I{} curl "http://127.0.0.1:9080/headers" -sL) && \
  count_httpbin=$(echo "$resp" | grep "httpbin.org" | wc -l) && \
  count_mockapi7=$(echo "$resp" | grep "mock.api7.ai" | wc -l) && \
  echo httpbin.org: $count_httpbin, mock.api7.ai: $count_mockapi7

The response will be as follows:

httpbin.org: 95, mock.api7.ai: 5

The idea behind a canary deployment is to test the new release with a small subset of the requests to minimize the impact of any issues. Once this release is tested, gradually increase the percentage of requests sent to the new release until all requests are routed to it:

Admin API
ADC
Ingress Controller

curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d '{
    "uri": "/headers",
    "id": "canary-deployment"
    "plugins": {
      "traffic-split": {
        "rules": [
          {
            "weighted_upstreams": [
              {
                "upstream": {
                  "type": "roundrobin",
                  "scheme": "https",
                  "pass_host": "node",
                  "nodes": {
                    "httpbin.org:443":1
                  }
                },
                "weight": 0
              },
              {
                "weight": 100
              }
            ]
          }
        ]
      }
    },
    "upstream": {
      "type": "roundrobin",
      "scheme": "https",
      "pass_host": "node",
      "nodes": {
        "mock.api7.ai:443":1
      }
    }
  }'

apisix.yaml
routes:
  - uris:
      - /headers
    name: canary-deployment
    plugins:
      traffic-split:
        rules:
          - weighted_upstreams:
              - weight: 0
                upstream:
                  type: roundrobin
                  pass_host: node
                  nodes:
                    httpbin.org:443: 1
                  scheme: https
              - weight: 100
    upstream:
      type: roundrobin
      pass_host: node
      nodes:
        mock.api7.ai:443: 1
      scheme: https

Synchronize the configuration to APISIX:

adc sync -f apisix.yaml

Gateway API
APISIX CRD

canary-deployment.yaml
apiVersion: apisix.apache.org/v1alpha1
kind: PluginConfig
metadata:
  namespace: ingress-apisix
  name: traffic-split-plugin-config
spec:
  plugins:
    - name: traffic-split
      config:
        rules:
          - weighted_upstreams:
            - upstream:
                scheme: https
                pass_host: node
                nodes:
                  httpbin.org:443: 1
              weight: 0
            - weight: 100
---
apiVersion: v1
kind: Service
metadata:
  namespace: ingress-apisix
  name: mockapi7-external-domain
spec:
  type: ExternalName
  externalName: mock.api7.ai
---
apiVersion: apisix.apache.org/v1alpha1
kind: BackendTrafficPolicy
metadata:
  namespace: ingress-apisix
  name: passhost-node
spec:
  targetRefs:
  - name: mockapi7-external-domain
    kind: Service
    group: ""
  passHost: node
  scheme: https
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  namespace: ingress-apisix
  name: canary-deployment
spec:
  parentRefs:
  - name: apisix
  rules:
  - matches:
    - path:
        type: Exact
        value: /headers
    filters:
    - type: ExtensionRef
      extensionRef:
        group: apisix.apache.org
        kind: PluginConfig
        name: traffic-split-plugin-config
    backendRefs:
    - name: mockapi7-external-domain
      port: 443
      weight: 1

canary-deployment.yaml
apiVersion: apisix.apache.org/v2
kind: ApisixUpstream
metadata:
  namespace: ingress-apisix
  name: mockapi7-external-domain
spec:
  ingressClassName: apisix
  scheme: https
  passHost: node
  externalNodes:
  - type: Domain
    name: mock.api7.ai
    weight: 1
    port: 443
---
apiVersion: apisix.apache.org/v2
kind: ApisixPluginConfig
metadata:
  namespace: ingress-apisix
  name: traffic-split-plugin-config
spec:
  ingressClassName: apisix
  plugins:
    - name: traffic-split
      enable: true
      config:
        rules:
          - weighted_upstreams:
            - upstream:
                scheme: https
                pass_host: node
                nodes:
                  httpbin.org:443: 1
              weight: 0
            - weight: 100
---
apiVersion: apisix.apache.org/v2
kind: ApisixRoute
metadata:
  namespace: ingress-apisix
  name: canary-deployment
spec:
  ingressClassName: apisix
  http:
    - name: canary-deployment
      match:
        paths:
          - /headers
      plugin_config_name: traffic-split-plugin-config
      upstreams:
        - name: mockapi7-external-domain

Apply the configuration to your cluster:

kubectl apply -f canary-deployment.yaml

Now, all requests will be directed to mock.api7.ai:

resp=$(seq 100 | xargs -I{} curl "http://127.0.0.1:9080/headers" -sL) && \
  count_httpbin=$(echo "$resp" | grep "httpbin.org" | wc -l) && \
  count_mockapi7=$(echo "$resp" | grep "mock.api7.ai" | wc -l) && \
  echo httpbin.org: $count_httpbin, mock.api7.ai: $count_mockapi7

This is evident from the response:

httpbin.org: 0, mock.api7.ai: 100

If the new release has issues, rollback to the old service by updating the configuration until the issue is resolved.

Configure Advanced Canary Deployment

You can also deploy your releases in a more granular fashion using the traffic-split plugin. The example below shows how to route requests based on a header.

This is useful when you want to give your clients more control over which release they access. Clients can also easily fall back to the old release by just modifying/removing the header.

Configure this in a route as shown below:

Admin API
ADC
Ingress Controller

curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d '{
    "uri": "/headers",
    "id": "canary-deployment",
    "plugins": {
      "traffic-split": {
        "rules": [
          {
            "match": [
              {
                "vars": [
                  ["http_release","==","new"]
                ]
              }
            ],
            "weighted_upstreams": [
              {
                "upstream": {
                  "type": "roundrobin",
                  "scheme": "https",
                  "pass_host": "node",
                  "nodes": {
                    "mock.api7.ai:443":1
                  }
                }
              }
            ]
          }
        ]
      }
    },
    "upstream": {
      "type": "roundrobin",
      "scheme": "https",
      "pass_host": "node",
      "nodes": {
        "httpbin.org:443":1
      }
    }
  }'

apisix.yaml
routes:
  - uris:
      - /headers
    name: canary-deployment
    plugins:
      traffic-split:
        rules:
          - match:
              - vars:
                  - - http_release
                    - ==
                    - new
            weighted_upstreams:
              - weight: 1
                upstream:
                  type: roundrobin
                  pass_host: node
                  nodes:
                    mock.api7.ai:443: 1
                  scheme: https
    upstream:
      type: roundrobin
      pass_host: node
      nodes:
        httpbin.org:443: 1
      scheme: https

Synchronize the configuration to APISIX:

adc sync -f apisix.yaml

Gateway API
APISIX CRD

canary-deployment.yaml
apiVersion: apisix.apache.org/v1alpha1
kind: PluginConfig
metadata:
  namespace: ingress-apisix
  name: traffic-split-plugin-config
spec:
  plugins:
    - name: traffic-split
      config:
        rules:
          - match:
            - vars:
                - ["http_release","==","new"]
            weighted_upstreams:
            - upstream:
                scheme: https
                pass_host: node
                nodes:
                  mock.api7.ai:443: 1
---
apiVersion: v1
kind: Service
metadata:
  namespace: ingress-apisix
  name: httpbin-external-domain
spec:
  type: ExternalName
  externalName: httpbin.org
---
apiVersion: apisix.apache.org/v1alpha1
kind: BackendTrafficPolicy
metadata:
  namespace: ingress-apisix
  name: passhost-node
spec:
  targetRefs:
  - name: httpbin-external-domain
    kind: Service
    group: ""
  passHost: node
  scheme: https
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  namespace: ingress-apisix
  name: canary-deployment
spec:
  parentRefs:
  - name: apisix
  rules:
  - matches:
    - path:
        type: Exact
        value: /headers
    filters:
    - type: ExtensionRef
      extensionRef:
        group: apisix.apache.org
        kind: PluginConfig
        name: traffic-split-plugin-config
    backendRefs:
    - name: httpbin-external-domain
      port: 443
      weight: 1

canary-deployment.yaml
apiVersion: apisix.apache.org/v2
kind: ApisixUpstream
metadata:
  namespace: ingress-apisix
  name: httpbin-external-domain
spec:
  ingressClassName: apisix
  scheme: https
  passHost: node
  externalNodes:
  - type: Domain
    name: httpbin.org
    weight: 1
    port: 443
---
apiVersion: apisix.apache.org/v2
kind: ApisixPluginConfig
metadata:
  namespace: ingress-apisix
  name: traffic-split-plugin-config
spec:
  ingressClassName: apisix
  plugins:
    - name: traffic-split
      enable: true
      config:
        rules:
          - match:
            - vars:
                - ["http_release","==","new"]
            weighted_upstreams:
            - upstream:
                scheme: https
                pass_host: node
                nodes:
                  mock.api7.ai:443: 1
---
apiVersion: apisix.apache.org/v2
kind: ApisixRoute
metadata:
  namespace: ingress-apisix
  name: canary-deployment
spec:
  ingressClassName: apisix
  http:
    - name: canary-deployment
      match:
        paths:
          - /headers
      plugin_config_name: traffic-split-plugin-config
      upstreams:
        - name: httpbin-external-domain

Apply the configuration to your cluster:

kubectl apply -f canary-deployment.yaml

Now, when you send a request with the release header:

curl "http://127.0.0.1:9080/headers" -H 'release: new'

You will see a response back from mock.api7.ai:

{
  "headers": {
    "accept": "*/*",
    "host": "mock.api7.ai",
    ...
  }
}

If you do not specify the release header:

curl "http://127.0.0.1:9080/headers"

APISIX will fall back to the httpbin.org upstream:

{
  "headers": {
    "Accept": "*/*",
    "Host": "httpbin.org",
    ...
  }
}

Refer to the traffic-split plugin documentation to learn more.

Configure Canary Deployment​

Configure Advanced Canary Deployment​

Configure Canary Deployment

Configure Advanced Canary Deployment