Configure Canary Traffic Shifting

Canary traffic shifting enables testing new upstreams safely by gradually routing a small portion of traffic to the new upstream while keeping most traffic to the stable default upstream. This incremental approach mitigates risk by containing potential issues to a limited number of users, allowing you to identify and resolve problems without impacting your broader user base.

note

Canary traffic shifting differs from a canary release as the API/service version is unchanged. Canary release refers to the simultaneous operation and availability of two versions of the same API/service.

Prerequisites

1. Install API7 Enterprise.
2. Have a running API on the gateway group.

Shift Traffic by Weight Percentage

In this example, you will direct 10% of the traffic to a new upstream. The remaining 90% will continue to be forwarded to the default upstream.

After testing the new upstream, consider rewriting the default upstream configuration with the new values and deleting the plugin to discontinue the canary process.

Dashboard

1. Select Published Services of your gateway group from the side navigation bar, then click the service you want to modify, for example, httpbin with version 1.0.0.
2. Under the published service, select Upstreams from the side navigation bar.
3. In the Connection Configuration module, click Edit, choose Use Node Host, and click Save.

   Note: Since mock.api7.ai enforces HTTPS access, the upstream needs to be configured to use port 443 for the HTTPS endpoint. The pass_host parameter must be changed to nodes to ensure a successful handshake with the upstream. Adjust accordingly per your use case.

4. Click Add Upstream.
5. In the dialog box, do the following:
   • In the Upstream Name field, enter new upstream.
   • Click Add Node to adjust the host of the node to point to the new backend. For example, use mock.api7.ai as the host and 443 as the port.
   • Click Add.
6. Click ViewID at the upstream header(under the Actions button) and copy for use.
7. Under the published service, select Plugins from the side navigation bar.
8. Click Add Plugin.
9. Search for the traffic-split plugin, then click Add.
10. In the dialog box, do the following:

• Add the following configuration to the JSON Editor:

  {
    "rules": [
              {
                  "weighted_upstreams": [
                      {
                          "upstream_id": new_upstream_id,    // Use upstream id, not upstream name
                          "weight": 1
                      },
                      {
                          "weight": 9
                      }
                  ]
              }
          ]
  }

• Click Add.

Validate

Validate the canary rules by sending 10 requests:

for i in {1..10}; do "curl 127.0.0.1:9080/headers";  done

9 requests will be sent to the default upstream address, httpbin.org, and you will receive the following response:

{
  "headers": {
    "Accept": "*/*",
    "Host": "httpbin.org",
    "User-Agent": "curl/7.74.0",
    "X-Amzn-Trace-Id": "Root=1-6650ab7e-32c90eba787abbeb4e3dbb0c",
    "X-Forwarded-Host": "127.0.0.1"
  }
}

A single request will be sent to the new upstream address, mock.api7.ai:

{
  "headers": {
    "accept": "*/*",
    "accept-encoding": "gzip, br",
    "cf-connecting-ip": "159.89.160.194",
    "cf-ipcountry": "IN",
    "cf-ray": "888e28733f9604aa",
    "cf-visitor": "{\"scheme\":\"https\"}",
    "connection": "Keep-Alive",
    "content-type": "application/json",
    "host": "mock.api7.ai",
    "user-agent": "curl/7.74.0",
    "x-application-owner": "API7.ai",
    "x-forwarded-for": "127.0.0.1",
    "x-forwarded-host": "127.0.0.1",
    "x-forwarded-port": "9080",
    "x-forwarded-proto": "https",
    "x-real-ip": "159.89.160.194",
    "X-Application-Owner": "API7.ai",
    "Content-Type": "application/json"
  }
}

Below is an interactive demo for this use case. Click and follow the steps in this demo, you will better understand how to use it in API7 Enterprise.

ADC

Update your ADC configuration file (adc.yaml) to include the new upstream. The complete configuration is given below:

adc.yaml

services:
  - name: httpbin
    upstream:
      name: Test Group
      scheme: https
      nodes:
        - host: httpbin.org
          port: 443
          weight: 100
    plugins:
      api7-traffic-split:
        rules:
          - canary_upstreams:
              - upstream_name: newupstream
                weight: 10
              - weight: 90
        upstreams:
          - name: newupstream
            nodes:
              - host: mock.api7.ai
                port: 443
                weight: 100
            scheme: https
    routes:
      - uris:
          - /headers
        name: getting-started-headers
        methods:
          - GET

Synchronize the configuration to API7 Enterprise:

adc sync -f adc.yaml

Validate

Validate the canary rules by sending 10 requests:

for i in {1..10}; do curl "127.0.0.1:9080/headers";  done

9 requests will be sent to httpbin.org and you will receive the following response:

{
 "headers": {
   "Accept": "*/*",
   "Host": "httpbin.org",
   "User-Agent": "curl/7.74.0",
   "X-Amzn-Trace-Id": "Root=1-6650ab7e-32c90eba787abbeb4e3dbb0c",
   "X-Forwarded-Host": "127.0.0.1"
 }
}

A single request will be sent to mock.api7.ai:

{
 "headers": {
   "accept": "*/*",
   "accept-encoding": "gzip, br",
   "cf-connecting-ip": "159.89.160.194",
   "cf-ipcountry": "IN",
   "cf-ray": "888e28733f9604aa",
   "cf-visitor": "{\"scheme\":\"https\"}",
   "connection": "Keep-Alive",
   "content-type": "application/json",
   "host": "mock.api7.ai",
   "user-agent": "curl/7.74.0",
   "x-application-owner": "API7.ai",
   "x-forwarded-for": "127.0.0.1",
   "x-forwarded-host": "127.0.0.1",
   "x-forwarded-port": "9080",
   "x-forwarded-proto": "https",
   "x-real-ip": "159.89.160.194",
   "X-Application-Owner": "API7.ai",
   "Content-Type": "application/json"
 }
}

Now, update the weights to 50:50 to allow half of the traffic to be routed to the new upstream:

adc.yaml

services:
  - name: httpbin
    upstream:
      name: Test Group
      scheme: https
      nodes:
        - host: httpbin.org
          port: 443
          weight: 100
    plugins:
      api7-traffic-split:
        rules:
          - canary_upstreams:
              - upstream_name: newupstream
                weight: 50
              - weight: 50
        upstreams:
          - name: newupstream
            nodes:
              - host: mock.api7.ai
                port: 443
                weight: 100
            scheme: https
    routes:
      - uris:
          - /headers
        name: getting-started-headers
        methods:
          - GET

Send more requests to test the new upstream. Finally, update the default upstream to finish the canary traffic shifting:

adc.yaml

services:
  - name: httpbin
    upstream:
      name: Test Group
      scheme: https
      nodes:
        - host: mock.api7.ai
          port: 443
          weight: 100
    routes:
      - uris:
          - /headers
        name: getting-started-headers
        methods:
          - GET

Ingress Controller

ApisixService custom resource is required for the api7-traffic-split plugin and is unavailable.

Shift Traffic by Condition: Request Header

In this example, you will direct requests with the header version = test to the new upstream, while the remaining traffic will continue to the default upstream. The canary rule applies to all routes in a service and cannot be applied to individual routes.

Dashboard

1. Select Published Services of your gateway group from the side navigation bar, then click the service you want to modify, for example, httpbin with version 1.0.0.
2. Under the service, select Upstreams from the side navigation bar.
3. Click Add Upstream.
4. In the dialog box, do the following:
   • In the Upstream Name field, enter new upstream.
   • Click Add Node to adjust the host of the node to point to the new backend. For example, use mock.api7.ai as the host and 443 as the port.
   • Click Add.
5. Click ViewID at the upstream header(under the Actions button) and copy for use.
6. Under the published service, select Plugins from the side navigation bar.
7. Click Add Plugin.
8. Search for the traffic-split plugin, then click Add.
9. In the dialog box, do the following:

• Add the following configuration to the JSON Editor:

  {
    "rules": [
      {
          "match": [
            {
              "vars": [
                ["version","==","test"]
              ]
            }
          ],
          "weighted_upstreams": [
              {
                  "upstream_id": new_upstream_id,    // Use upstream id, not upstream name
                  "weight": 1
              },
              {
                   "weight": 9
              }
          ]
      }
  ]
  }

• Click Add.

Validate

Validate the canary rules by sending requests:

• Send a request with the version:test header:

  curl 127.0.0.1:9080/headers -H "version:test"

  You shall receive the following response from the new upstream:

  {
    "headers": {
      "accept": "*/*",
      "accept-encoding": "gzip, br",
      "cf-connecting-ip": "159.89.160.194",
      "cf-ipcountry": "IN",
      "cf-ray": "888e28733f9604aa",
      "cf-visitor": "{\"scheme\":\"https\"}",
      "connection": "Keep-Alive",
      "content-type": "application/json",
      "host": "mock.api7.ai",
      "user-agent": "curl/7.74.0",
      "x-application-owner": "API7.ai",
      "x-forwarded-for": "127.0.0.1",
      "x-forwarded-host": "127.0.0.1",
      "x-forwarded-port": "9080",
      "x-forwarded-proto": "https",
      "x-real-ip": "159.89.160.194",
      "X-Application-Owner": "API7.ai",
      "Content-Type": "application/json"
    }
  }

• Send a request with the wrong header:

  curl 127.0.0.1:9080/headers -H "version:new"

  You shall receive the following response from the default upstream:

  {
    "headers": {
      "Accept": "*/*",
      "Host": "httpbin.org",
      "User-Agent": "curl/7.74.0",
      "X-Amzn-Trace-Id": "Root=1-6650ab7e-32c90eba787abbeb4e3dbb0c",
      "X-Forwarded-Host": "127.0.0.1"
    }
  }

• Send a request with no header:

  curl 127.0.0.1:9080/headers

• You shall receive the following response from the default upstream:

  {
    "headers": {
      "Accept": "*/*",
      "Host": "httpbin.org",
      "User-Agent": "curl/7.74.0",
      "X-Amzn-Trace-Id": "Root=1-6650ab7e-32c90eba787abbeb4e3dbb0c",
      "X-Forwarded-Host": "127.0.0.1"
    }
  }

8. Make more requests to test the new upstream, until it meets your expectations.

ADC

Update your ADC configuration file (adc.yaml) to include the new upstream. The complete configuration is given below:

adc.yaml

services:
  - name: httpbin
    upstream:
      name: default
      scheme: https
      nodes:
        - host: httpbin.org
          port: 443
          weight: 100
    plugins:
      api7-traffic-split:
        rules:
          - canary_upstreams:
              - upstream_name: newupstream
                weight: 100
            match:
              - exprs:
                  - - http_version
                    - ==
                    - test
        upstreams:
          - name: newupstream
            nodes:
              - host: mock.api7.ai
                port: 443
                weight: 100
            scheme: https
    routes:
      - uris:
          - /headers
        name: getting-started-headers
        methods:
          - GET

Synchronize the configuration to API7 Gateway:

adc sync -f adc.yaml

Validate the canary rules by sending a request with the version:test header:

curl 127.0.0.1:9080/headers  

You will get back a response from the new upstream:

{
 "headers": {
   "accept": "*/*",
   "accept-encoding": "gzip, br",
   "cf-connecting-ip": "159.89.160.194",
   "cf-ipcountry": "IN",
   "cf-ray": "888e28733f9604aa",
   "cf-visitor": "{\"scheme\":\"https\"}",
   "connection": "Keep-Alive",
   "content-type": "application/json",
   "host": "mock.api7.ai",
   "user-agent": "curl/7.74.0",
   "x-application-owner": "API7.ai",
   "x-forwarded-for": "127.0.0.1",
   "x-forwarded-host": "127.0.0.1",
   "x-forwarded-port": "9080",
   "x-forwarded-proto": "https",
   "x-real-ip": "159.89.160.194",
   "X-Application-Owner": "API7.ai",
   "Content-Type": "application/json"
 }
}

Send a request with the wrong header:

curl 127.0.0.1:9080/headers -H "version:new"

You shall receive the following response from the default upstream:

{
 "headers": {
   "Accept": "*/*",
   "Host": "httpbin.org",
   "User-Agent": "curl/7.74.0",
   "X-Amzn-Trace-Id": "Root=1-6650ab7e-32c90eba787abbeb4e3dbb0c",
   "X-Forwarded-Host": "127.0.0.1"
 }
}

Send a request with no header:

curl 127.0.0.1:9080/headers

You shall receive the following response from the default upstream:

{
 "headers": {
   "Accept": "*/*",
   "Host": "httpbin.org",
   "User-Agent": "curl/7.74.0",
   "X-Amzn-Trace-Id": "Root=1-6650ab7e-32c90eba787abbeb4e3dbb0c",
   "X-Forwarded-Host": "127.0.0.1"
 }
}

Update the default upstream with the new upstream values to finish the canary traffic shifting:

adc.yaml

services:
  - name: httpbin
    upstream:
      name: Test Group
      scheme: https
      nodes:
        - host: mock.api7.ai
          port: 443
          weight: 100
    routes:
      - uris:
          - /headers
        name: getting-started-headers
        methods:
          - GET

Ingress Controller

ApisixService custom resource is required for the api7-traffic-split plugin and is unavailable.

Additional Resources

• Key Concept
  • Services
  • Upstreams
• Plugin Hub
  • Traffic Split