Configure Canary Traffic Shifting

Canary traffic shifting enables testing new upstreams safely by gradually routing a small portion of traffic to the new upstream while keeping most traffic to the stable default upstream. This incremental approach mitigates risk by containing potential issues to a limited number of users, allowing you to identify and resolve problems without impacting your broader user base.

note

Canary traffic shifting differs from a canary release as the API/service version is unchanged. Canary release refers to the simultaneous operation and availability of two versions of the same API/service.

Prerequisites

1. Install API7 Enterprise.
2. Create a route to /headers.

Shift Traffic by Weight

In this example, you will direct 10% of the traffic to a new upstream. The remaining 90% will continue to be forwarded to the default upstream.

After testing the new upstream, consider rewriting the default upstream configuration with the new values and deleting the plugin to discontinue the canary process.

Dashboard

1. Select Published Services of your gateway group from the side navigation bar, then click the service you want to modify, for example, httpbin with version 1.0.0.
2. Under the published service, select Upstreams from the side navigation bar.
3. In the Connection Configuration module, click Edit, choose Use Node Host, and click Save.

   Note: Since mock.api7.ai enforces HTTPS access, the upstream needs to be configured to use port 443 for the HTTPS endpoint. The pass_host parameter must be changed to nodes to ensure a successful handshake with the upstream. Adjust accordingly per your use case.

4. Click Add Upstream.
5. In the dialog box, do the following:
   • In the Upstream Name field, enter new upstream.
   • Click Add Node to adjust the host of the node to point to the new backend. For example, use mock.api7.ai as the host and 443 as the port.
   • Click Add.
6. Click View ID at the upstream header (under the Actions button) and copy for use.
7. Under the published service, select Plugins from the side navigation bar.
8. Click Add Plugin.
9. Search for the traffic-split plugin, then click Add.
10. In the dialog box, do the following:

• Add the following configuration to the JSON Editor:

  {
    "rules": [
      {
        "weighted_upstreams": [
          {
            "upstream_id": "new_upstream_id",    // Use upstream id, not upstream name
            "weight": 1
          },
          {
            "weight": 9
          }
        ]
      }
    ]
  }

• Click Add.

Validate

Validate the canary rules by sending 10 requests:

for i in {1..10}; do "curl 127.0.0.1:9080/headers";  done

9 requests will be sent to the default upstream address, httpbin.org, and you will receive the following response:

{
  "headers": {
    "Accept": "*/*",
    "Host": "httpbin.org",
    "User-Agent": "curl/7.74.0",
    "X-Amzn-Trace-Id": "Root=1-6650ab7e-32c90eba787abbeb4e3dbb0c",
    "X-Forwarded-Host": "127.0.0.1"
  }
}

A single request will be sent to the new upstream address, mock.api7.ai:

{
  "headers": {
    "accept": "*/*",
    "accept-encoding": "gzip, br",
    "cf-connecting-ip": "159.89.160.194",
    "cf-ipcountry": "IN",
    "cf-ray": "888e28733f9604aa",
    "cf-visitor": "{\"scheme\":\"https\"}",
    "connection": "Keep-Alive",
    "content-type": "application/json",
    "host": "mock.api7.ai",
    "user-agent": "curl/7.74.0",
    "x-application-owner": "API7.ai",
    "x-forwarded-for": "127.0.0.1",
    "x-forwarded-host": "127.0.0.1",
    "x-forwarded-port": "9080",
    "x-forwarded-proto": "https",
    "x-real-ip": "159.89.160.194",
    "X-Application-Owner": "API7.ai",
    "Content-Type": "application/json"
  }
}

Below is an interactive demo for this use case. Click and follow the steps in this demo, you will better understand how to use it in API7 Enterprise.

ADC

Update your ADC configuration file (adc.yaml) to include the new upstream. The complete configuration is given below:

adc.yaml

services:
  - name: httpbin
    upstream:
      name: Test Group
      scheme: https
      nodes:
        - host: httpbin.org
          port: 443
          weight: 100
        upstreams:
          - name: newupstream
            nodes:
              - host: mock.api7.ai
                port: 443
                weight: 100
            scheme: https
    routes:
      - uris:
          - /headers
        name: getting-started-headers
        methods:
          - GET
        plugins:
          traffic-split:
            rules:
              - weighted_upstreams:
                  - upstream_id: new_upstream_id  # Use upstream id, not upstream name
                    weight: 1
                  - weight: 9

Synchronize the configuration to API7 Enterprise:

adc sync -f adc.yaml

Validate

Validate the canary rules by sending 10 requests:

for i in {1..10}; do curl "127.0.0.1:9080/headers";  done

9 requests will be sent to httpbin.org and you will receive the following response:

{
 "headers": {
   "Accept": "*/*",
   "Host": "httpbin.org",
   "User-Agent": "curl/7.74.0",
   "X-Amzn-Trace-Id": "Root=1-6650ab7e-32c90eba787abbeb4e3dbb0c",
   "X-Forwarded-Host": "127.0.0.1"
 }
}

A single request will be sent to mock.api7.ai:

{
 "headers": {
   "accept": "*/*",
   "accept-encoding": "gzip, br",
   "cf-connecting-ip": "159.89.160.194",
   "cf-ipcountry": "IN",
   "cf-ray": "888e28733f9604aa",
   "cf-visitor": "{\"scheme\":\"https\"}",
   "connection": "Keep-Alive",
   "content-type": "application/json",
   "host": "mock.api7.ai",
   "user-agent": "curl/7.74.0",
   "x-application-owner": "API7.ai",
   "x-forwarded-for": "127.0.0.1",
   "x-forwarded-host": "127.0.0.1",
   "x-forwarded-port": "9080",
   "x-forwarded-proto": "https",
   "x-real-ip": "159.89.160.194",
   "X-Application-Owner": "API7.ai",
   "Content-Type": "application/json"
 }
}

Now, update the weights to 50:50 to allow half of the traffic to be routed to the new upstream:

adc.yaml

services:
  - name: httpbin
    upstream:
      name: Test Group
      scheme: https
      nodes:
        - host: httpbin.org
          port: 443
          weight: 100
    plugins:
      api7-traffic-split:
        rules:
          - canary_upstreams:
              - upstream_name: newupstream
                weight: 50
              - weight: 50
        upstreams:
          - name: newupstream
            nodes:
              - host: mock.api7.ai
                port: 443
                weight: 100
            scheme: https
    routes:
      - uris:
          - /headers
        name: getting-started-headers
        methods:
          - GET

Send more requests to test the new upstream. Finally, update the default upstream to finish the canary traffic shifting:

adc.yaml

services:
  - name: httpbin
    upstream:
      name: Test Group
      scheme: https
      nodes:
        - host: mock.api7.ai
          port: 443
          weight: 100
    routes:
      - uris:
          - /headers
        name: getting-started-headers
        methods:
          - GET

Ingress Controller

Note that API7 Ingress Controller does not support the use of upstream_id in the traffic-split plugin.

Gateway API

Create a Kubernetes manifest file for a route (that also creates a service), where traffic-split is enabled for the services for a 10%/90% traffic split:

traffic-split.yaml

apiVersion: apisix.apache.org/v1alpha1
kind: PluginConfig
metadata:
  name: traffic-split-plugin-config
spec:
  plugins:
    - name: traffic-split
      config:
        rules:
          - weighted_upstreams:
            - upstream:
                scheme: https
                pass_host: node
                nodes:
                  mock.api7.ai: 1
              weight: 1
            - weight: 9
---
apiVersion: v1
kind: Service
metadata:
  name: httpbin-external-domain
spec:
  type: ExternalName
  externalName: httpbin.org
---
apiVersion: apisix.apache.org/v1alpha1
kind: BackendTrafficPolicy
metadata:
  name: passhost-node
spec:
  targetRefs:
  - name: httpbin-external-domain
    kind: Service
    group: ""
  passHost: node
  scheme: https
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: getting-started-ip
spec:
  parentRefs:
  - name: apisix
  rules:
  - matches:
    - path:
        type: Exact
        value: /headers
    filters:
    - type: ExtensionRef
      extensionRef:
        group: apisix.apache.org
        kind: PluginConfig
        name: traffic-split-plugin-config
    backendRefs:
    - name: httpbin-external-domain
      port: 443
      weight: 1

APISIX CRD

Create a Kubernetes manifest file for a route (that also creates a service), where traffic-split is enabled for the services for a 10%/90% traffic split:

traffic-split.yaml

apiVersion: apisix.apache.org/v2
kind: ApisixUpstream
metadata:
  name: httpbin-external-domain
spec:
  scheme: https
  passHost: node
  externalNodes:
  - type: Domain
    name: httpbin.org
    weight: 1
    port: 443
---
apiVersion: apisix.apache.org/v2
kind: ApisixPluginConfig
metadata:
  name: traffic-split-plugin-config
spec:
  ingressClassName: apisix
  plugins:
    - name: traffic-split
      enable: true
      config:
        rules:
          - weighted_upstreams:
            - upstream:
                scheme: https
                pass_host: node
                nodes:
                  mock.api7.ai: 1
              weight: 1
            - weight: 9
---
apiVersion: apisix.apache.org/v2
kind: ApisixRoute
metadata:
  name: getting-started-headers
spec:
  ingressClassName: apisix
  http:
    - name: getting-started-headers
      match:
        paths:
          - /headers
      plugin_config_name: traffic-split-plugin-config
      upstreams:
        - name: httpbin-external-domain

Alternatively, you could also assign weight to backend services without using the plugin to achieve the same:

traffic-split.yaml

apiVersion: apisix.apache.org/v2
kind: ApisixRoute
metadata:
  name: getting-started-headers
spec:
  ingressClassName: apisix
  http:
    - name: getting-started-headers
      match:
        paths:
          - /headers
      backends:
        - serviceName: httpbin
          servicePort: 80
          weight: 90
        - serviceName: new-upstream-svc-name
          servicePort: 80
          weight: 10

tip

Alternatively, you could also assign weight to upstream services without using the traffic-split plugin.

Apply the configuration to your cluster:

kubectl apply -f traffic-split.yaml

Note that if you are using the Gateway API, the plugin will appear at the service level. If you are using APISIX CRDs, the plugin will appear at the route level.

Shift Traffic by Condition: Request Header

In this example, you will direct requests with the header version = test to the new upstream, while the remaining traffic will continue to the default upstream. The canary rule applies to all routes in a service and cannot be applied to individual routes.

Dashboard

1. Select Published Services of your gateway group from the side navigation bar, then click the service you want to modify, for example, httpbin with version 1.0.0.
2. Under the service, select Upstreams from the side navigation bar.
3. Click Add Upstream.
4. In the dialog box, do the following:
   • In the Upstream Name field, enter new upstream.
   • Click Add Node to adjust the host of the node to point to the new backend. For example, use mock.api7.ai as the host and 443 as the port.
   • Click Add.
5. Click View ID at the upstream header (under the Actions button) and copy for use.
6. Under the published service, select Plugins from the side navigation bar.
7. Click Add Plugin.
8. Search for the traffic-split plugin, then click Add.
9. In the dialog box, do the following:

• Add the following configuration to the JSON Editor:

  {
    "rules": [
      {
        "match": [
          {
            "vars": [
              [
                "version",
                "==",
                "test"
              ]
            ]
          }
        ],
        "weighted_upstreams": [
          {
            "upstream_id": "new_upstream_id",    // Use upstream id, not upstream name
            "weight": 1
          },
          {
            "weight": 9
          }
        ]
      }
    ]
  }

• Click Add.

Validate

Validate the canary rules by sending requests:

• Send a request with the version:test header:

  curl 127.0.0.1:9080/headers -H "version:test"

  You shall receive the following response from the new upstream:

  {
    "headers": {
      "accept": "*/*",
      "accept-encoding": "gzip, br",
      "cf-connecting-ip": "159.89.160.194",
      "cf-ipcountry": "IN",
      "cf-ray": "888e28733f9604aa",
      "cf-visitor": "{\"scheme\":\"https\"}",
      "connection": "Keep-Alive",
      "content-type": "application/json",
      "host": "mock.api7.ai",
      "user-agent": "curl/7.74.0",
      "x-application-owner": "API7.ai",
      "x-forwarded-for": "127.0.0.1",
      "x-forwarded-host": "127.0.0.1",
      "x-forwarded-port": "9080",
      "x-forwarded-proto": "https",
      "x-real-ip": "159.89.160.194",
      "X-Application-Owner": "API7.ai",
      "Content-Type": "application/json"
    }
  }

• Send a request with the wrong header:

  curl 127.0.0.1:9080/headers -H "version:new"

  You shall receive the following response from the default upstream:

  {
    "headers": {
      "Accept": "*/*",
      "Host": "httpbin.org",
      "User-Agent": "curl/7.74.0",
      "X-Amzn-Trace-Id": "Root=1-6650ab7e-32c90eba787abbeb4e3dbb0c",
      "X-Forwarded-Host": "127.0.0.1"
    }
  }

• Send a request with no header:

  curl 127.0.0.1:9080/headers

• You shall receive the following response from the default upstream:

  {
    "headers": {
      "Accept": "*/*",
      "Host": "httpbin.org",
      "User-Agent": "curl/7.74.0",
      "X-Amzn-Trace-Id": "Root=1-6650ab7e-32c90eba787abbeb4e3dbb0c",
      "X-Forwarded-Host": "127.0.0.1"
    }
  }

11. Make more requests to test the new upstream until it meets your expectations.

ADC

Update your ADC configuration file (adc.yaml) to include the new upstream. The complete configuration is given below:

adc.yaml

services:
  - name: httpbin
    upstream:
      name: default
      scheme: https
      nodes:
        - host: httpbin.org
          port: 443
          weight: 100
      upstreams:
          - name: newupstream
            nodes:
              - host: mock.api7.ai
                port: 443
                weight: 100
            scheme: https
    routes:
      - uris:
          - /headers
        name: getting-started-headers
        methods:
          - GET
        plugins:
          traffic-split:
            rules:
              - match:
                  - vars:
                      - - version
                        - "=="
                        - test
                weighted_upstreams:
                  - upstream_id: new_upstream_id  # Use upstream id, not upstream name
                    weight: 1
                  - weight: 9

Synchronize the configuration to API7 Gateway:

adc sync -f adc.yaml

Validate the canary rules by sending a request with the version:test header:

curl 127.0.0.1:9080/headers -H "version:test"

You will get back a response from the new upstream:

{
 "headers": {
   "accept": "*/*",
   "accept-encoding": "gzip, br",
   "cf-connecting-ip": "159.89.160.194",
   "cf-ipcountry": "IN",
   "cf-ray": "888e28733f9604aa",
   "cf-visitor": "{\"scheme\":\"https\"}",
   "connection": "Keep-Alive",
   "content-type": "application/json",
   "host": "mock.api7.ai",
   "user-agent": "curl/7.74.0",
   "x-application-owner": "API7.ai",
   "x-forwarded-for": "127.0.0.1",
   "x-forwarded-host": "127.0.0.1",
   "x-forwarded-port": "9080",
   "x-forwarded-proto": "https",
   "x-real-ip": "159.89.160.194",
   "X-Application-Owner": "API7.ai",
   "Content-Type": "application/json"
 }
}

Send a request with the wrong header:

curl 127.0.0.1:9080/headers -H "version:new"

You shall receive the following response from the default upstream:

{
 "headers": {
   "Accept": "*/*",
   "Host": "httpbin.org",
   "User-Agent": "curl/7.74.0",
   "X-Amzn-Trace-Id": "Root=1-6650ab7e-32c90eba787abbeb4e3dbb0c",
   "X-Forwarded-Host": "127.0.0.1"
 }
}

Send a request with no header:

curl 127.0.0.1:9080/headers

You shall receive the following response from the default upstream:

{
 "headers": {
   "Accept": "*/*",
   "Host": "httpbin.org",
   "User-Agent": "curl/7.74.0",
   "X-Amzn-Trace-Id": "Root=1-6650ab7e-32c90eba787abbeb4e3dbb0c",
   "X-Forwarded-Host": "127.0.0.1"
 }
}

Update the default upstream with the new upstream values to finish the canary traffic shifting:

adc.yaml

services:
  - name: httpbin
    upstream:
      name: Test Group
      scheme: https
      nodes:
        - host: mock.api7.ai
          port: 443
          weight: 100
    routes:
      - uris:
          - /headers
        name: getting-started-headers
        methods:
          - GET

Ingress Controller

You will learn how to shift traffic by weight and conditionally with the traffic-split plugin.

Gateway API

Create a Kubernetes manifest file for a route (that also creates a service), where traffic-split is enabled for the service:

traffic-split.yaml

apiVersion: apisix.apache.org/v1alpha1
kind: PluginConfig
metadata:
  name: traffic-split-plugin-config
spec:
  plugins:
    - name: traffic-split
      config:
        rules:
          - match:
              - vars:
                - - version
                  - "=="
                  - test
            weighted_upstreams:
              - upstream:
                  scheme: https
                  pass_host: node
                  nodes:
                    mock.api7.ai: 1
                weight: 1
              - weight: 9
---
apiVersion: v1
kind: Service
metadata:
  name: httpbin-external-domain
spec:
  type: ExternalName
  externalName: httpbin.org
---
apiVersion: apisix.apache.org/v1alpha1
kind: BackendTrafficPolicy
metadata:
  name: passhost-node
spec:
  targetRefs:
  - name: httpbin-external-domain
    kind: Service
    group: ""
  passHost: node
  scheme: https
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: getting-started-ip
spec:
  parentRefs:
  - name: apisix
  rules:
  - matches:
    - path:
        type: Exact
        value: /headers
    filters:
    - type: ExtensionRef
      extensionRef:
        group: apisix.apache.org
        kind: PluginConfig
        name: traffic-split-plugin-config
    backendRefs:
    - name: httpbin-external-domain
      port: 443
      weight: 1

APISIX CRD

Create a Kubernetes manifest file for a route (that also creates a service), where traffic-split is configured for the service as such:

traffic-split.yaml

apiVersion: apisix.apache.org/v2
kind: ApisixUpstream
metadata:
  name: httpbin-external-domain
spec:
  scheme: https
  passHost: node
  externalNodes:
  - type: Domain
    name: httpbin.org
    weight: 1
    port: 443
---
apiVersion: apisix.apache.org/v2
kind: ApisixPluginConfig
metadata:
  name: traffic-split-plugin-config
spec:
  ingressClassName: apisix
  plugins:
    - name: traffic-split
      enable: true
      config:
        rules:
          - match:
            - vars:
              - - version
                - "=="
                - test
            weighted_upstreams:
              - upstream:
                  scheme: https
                  pass_host: node
                  nodes:
                    mock.api7.ai: 1
                weight: 1
              - weight: 9
---
apiVersion: apisix.apache.org/v2
kind: ApisixRoute
metadata:
  name: getting-started-headers
spec:
  ingressClassName: apisix
  http:
    - name: getting-started-headers
      match:
        paths:
          - /headers
      plugin_config_name: traffic-split-plugin-config
      upstreams:
        - name: httpbin-external-domain

Apply the configuration to your cluster:

kubectl apply -f traffic-split.yaml

Note that if you are using the Gateway API, the plugin will appear at the service level. If you are using APISIX CRDs, the plugin will appear at the route level.

Additional Resources

• Key Concept
  • Services
  • Upstreams
• Plugin Hub
  • Traffic Split