Configure Canary Traffic Shifting

Canary traffic shifting enables you to safely test new upstreams with a small percentage of your traffic while keeping most traffic to the baseline upstream.

note

Canary traffic shifting differs from a canary release as the API/service version is unchanged. Canary release refers to the simultaneous operation and availability of two versions of the same API/service.

Prerequisites

1. Install API7 Enterprise.
2. Have a running API on the gateway group.

Shift Traffic by Percentage

In this example, you will direct 10% of the traffic to a canary upstream. The remaining 90% will continue to be forwarded to the baseline upstream.

note

The canary rule applies to all routes in a service and cannot be applied to individual routes.

After the new canary upstream is tested, all traffic can be routed to the canary upstream and it becomes the new baseline upstream. The older baseline upstream can then be removed.

Dashboard

1. Select Published Services of your gateway group from the side navigation bar, then click the service you want to modify, for example, httpbin API with version 1.0.0.
2. Under the published service, select Upstreams from the side navigation bar.
3. In the Canary Rules field, click Start Canary.
4. From the dialog box, do the following:

• In the Conditions field, close the switch.
• In the Weight field, enter 10.
• Click Next.

5. In the Choose or Create Canary Upstream field, choose to create a new upstream.

• Modify the name of the new upstream to newupstream.
• Adjust the host of the node to point to the new backend. For example, use mock.api7.ai as the host and 80 as the port.
• Keep the other properties the same as the baseline upstream.
• Click Next.

6. Confirm the displayed information and click Start. The canary rules will start working immediately.

7. Validate the canary rules by sending 10 requests:

   for i in {1..10}; do "curl 127.0.0.1:9080/headers";  done

   9 requests will be sent to the baseline upstream address, httpbin.org, and you will receive the following response:

   {
     "headers": {
       "Accept": "*/*",
       "Host": "httpbin.org",
       "User-Agent": "curl/7.74.0",
       "X-Amzn-Trace-Id": "Root=1-6650ab7e-32c90eba787abbeb4e3dbb0c",
       "X-Forwarded-Host": "127.0.0.1"
     }
   }

   A single request will be sent to the canary upstream address, mock.api7.ai:

   {
     "headers": {
       "accept": "*/*",
       "accept-encoding": "gzip, br",
       "cf-connecting-ip": "159.89.160.194",
       "cf-ipcountry": "IN",
       "cf-ray": "888e28733f9604aa",
       "cf-visitor": "{\"scheme\":\"https\"}",
       "connection": "Keep-Alive",
       "content-type": "application/json",
       "host": "mock.api7.ai",
       "user-agent": "curl/7.74.0",
       "x-application-owner": "API7.ai",
       "x-forwarded-for": "127.0.0.1",
       "x-forwarded-host": "127.0.0.1",
       "x-forwarded-port": "9080",
       "x-forwarded-proto": "https",
       "x-real-ip": "159.89.160.194",
       "X-Application-Owner": "API7.ai",
       "Content-Type": "application/json"
     }
   }

8. In the Canary Rules field, click Edit.

9. From the dialog box, do the following:

• Adjust the weight to 50%.
• Click Edit.

10. Make more requests to test the canary upstream, until it meets your expectation.
11. In the Canary Rules field, click Finish.
12. From the dialog box, do the following:

• In the Baseline Upstream field, choose canary upstream: newupstream.
• In the Delete Unselected Upstream field, close the switch.
• Click Finish.

ADC

Update your ADC configuration file (adc.yaml) to include the canary upstream. The complete configuration is given below:

adc.yaml

services:
  - name: httpbin API
    upstream:
      name: Test Group
      scheme: https
      nodes:
        - host: httpbin.org
          port: 443
          weight: 100
    plugins:
      api7-traffic-split:
        rules:
          - canary_upstreams:
              - upstream_name: newupstream
                weight: 10
              - weight: 90
        upstreams:
          - name: newupstream
            nodes:
              - host: mock.api7.ai
                port: 443
                weight: 100
            scheme: https
    routes:
      - uris:
          - /headers
        name: getting-started-headers
        methods:
          - GET

Synchronize the configuration to API7 Enterprise:

adc sync -f adc.yaml

Validate the canary rules by sending 10 requests:

for i in {1..10}; do curl "127.0.0.1:9080/headers";  done

9 requests will be sent to httpbin.org and you will receive the following response:

{
 "headers": {
   "Accept": "*/*",
   "Host": "httpbin.org",
   "User-Agent": "curl/7.74.0",
   "X-Amzn-Trace-Id": "Root=1-6650ab7e-32c90eba787abbeb4e3dbb0c",
   "X-Forwarded-Host": "127.0.0.1"
 }
}

A single request will be sent to mock.api7.ai:

{
 "headers": {
   "accept": "*/*",
   "accept-encoding": "gzip, br",
   "cf-connecting-ip": "159.89.160.194",
   "cf-ipcountry": "IN",
   "cf-ray": "888e28733f9604aa",
   "cf-visitor": "{\"scheme\":\"https\"}",
   "connection": "Keep-Alive",
   "content-type": "application/json",
   "host": "mock.api7.ai",
   "user-agent": "curl/7.74.0",
   "x-application-owner": "API7.ai",
   "x-forwarded-for": "127.0.0.1",
   "x-forwarded-host": "127.0.0.1",
   "x-forwarded-port": "9080",
   "x-forwarded-proto": "https",
   "x-real-ip": "159.89.160.194",
   "X-Application-Owner": "API7.ai",
   "Content-Type": "application/json"
 }
}

Now, update the weights to 50:50 to allow half of the traffic to be routed to the canary upstream:

adc.yaml

services:
  - name: httpbin API
    upstream:
      name: Test Group
      scheme: https
      nodes:
        - host: httpbin.org
          port: 443
          weight: 100
    plugins:
      api7-traffic-split:
        rules:
          - canary_upstreams:
              - upstream_name: newupstream
                weight: 50
              - weight: 50
        upstreams:
          - name: newupstream
            nodes:
              - host: mock.api7.ai
                port: 443
                weight: 100
            scheme: https
    routes:
      - uris:
          - /headers
        name: getting-started-headers
        methods:
          - GET

Send more requests to test the canary upstream. Finally, update the old upstream with the current upstream to finish the canary traffic shifting:

adc.yaml

services:
  - name: httpbin API
    upstream:
      name: Test Group
      scheme: https
      nodes:
        - host: mock.api7.ai
          port: 443
          weight: 100
    routes:
      - uris:
          - /headers
        name: getting-started-headers
        methods:
          - GET

You can also keep the old upstream as a canary upstream with 0 weight to roll back to if there are issues with the new upstream.

Ingress Controller

ApisixService custom resource is required for the api7-traffic-split plugin and not yet available.

Shift Traffic by Request Header

In this example, you will direct requests with the header version = test to the canary upstream, while the remaining traffic will continue to the baseline upstream. The canary rule applies to all routes in a service and cannot be applied to individual routes.

Dashboard

1. Select Published Services of your gateway group from the side navigation bar, then click the service you want to modify, for example, httpbin API with version 1.0.0.
2. Under the service, select Upstreams from the side navigation bar.
3. In the Canary Rules field, click Start Canary.
4. From the dialog box, do the following:

• In the Conditions field, open the switch.
• Fill in the header requirement with header evaluation version == test.
• In the Weight field, enter 100.
• Click Next.

5. In the Choose or Create Canary Upstream field, choose to create a new upstream.

• Modify the name of the new upstream to newupstream.
• Adjust the host of the node to point to the new backend. For example, use mock.api7.ai as the host and 80 as the port.
• Keep the other properties the same as the baseline upstream.
• Click Next.

6. Confirm the displayed information and click Start. The canary rules will start working immediately.

7. Validate the canary rules by sending requests:

   Send a request with the version:test header:

   curl 127.0.0.1:9080/headers -H "version:test"

   You shall receive the following response from the canary upstream:

   {
     "headers": {
       "accept": "*/*",
       "accept-encoding": "gzip, br",
       "cf-connecting-ip": "159.89.160.194",
       "cf-ipcountry": "IN",
       "cf-ray": "888e28733f9604aa",
       "cf-visitor": "{\"scheme\":\"https\"}",
       "connection": "Keep-Alive",
       "content-type": "application/json",
       "host": "mock.api7.ai",
       "user-agent": "curl/7.74.0",
       "x-application-owner": "API7.ai",
       "x-forwarded-for": "127.0.0.1",
       "x-forwarded-host": "127.0.0.1",
       "x-forwarded-port": "9080",
       "x-forwarded-proto": "https",
       "x-real-ip": "159.89.160.194",
       "X-Application-Owner": "API7.ai",
       "Content-Type": "application/json"
     }
   }

   Send a request with the wrong header:

   curl 127.0.0.1:9080/headers -H "version:new"

   You shall receive the following response from the baseline upstream:

   {
     "headers": {
       "Accept": "*/*",
       "Host": "httpbin.org",
       "User-Agent": "curl/7.74.0",
       "X-Amzn-Trace-Id": "Root=1-6650ab7e-32c90eba787abbeb4e3dbb0c",
       "X-Forwarded-Host": "127.0.0.1"
     }
   }

   Send a request with no header:

   curl 127.0.0.1:9080/headers

   You shall receive the following response from the baseline upstream:

   {
     "headers": {
       "Accept": "*/*",
       "Host": "httpbin.org",
       "User-Agent": "curl/7.74.0",
       "X-Amzn-Trace-Id": "Root=1-6650ab7e-32c90eba787abbeb4e3dbb0c",
       "X-Forwarded-Host": "127.0.0.1"
     }
   }

8. Make more requests to test the canary upstream, until it meets your expectation.

9. In the Canary Rules field, click Finish.

10. From the dialog box, do the following:

• In the Baseline Upstream field, choose canary upstream: newupstream.
• In the Delete Unselected Upstream field, close the switch.
• Click Finish.

ADC

Update your ADC configuration file (adc.yaml) to include the canary upstream. The complete configuration is given below:

adc.yaml

services:
  - name: httpbin API
    upstream:
      name: default
      scheme: https
      nodes:
        - host: httpbin.org
          port: 443
          weight: 100
    plugins:
      api7-traffic-split:
        rules:
          - canary_upstreams:
              - upstream_name: newupstream
                weight: 100
            match:
              - exprs:
                  - - http_version
                    - ==
                    - test
        upstreams:
          - name: newupstream
            nodes:
              - host: mock.api7.ai
                port: 443
                weight: 100
            scheme: https
    routes:
      - uris:
          - /headers
        name: getting-started-headers
        methods:
          - GET

Synchronize the configuration to API7 Gateway:

adc sync -f adc.yaml

Validate the canary rules by sending a request with the version:test header:

curl 127.0.0.1:9080/headers  

You will get back a response from the canary upstream:

{
 "headers": {
   "accept": "*/*",
   "accept-encoding": "gzip, br",
   "cf-connecting-ip": "159.89.160.194",
   "cf-ipcountry": "IN",
   "cf-ray": "888e28733f9604aa",
   "cf-visitor": "{\"scheme\":\"https\"}",
   "connection": "Keep-Alive",
   "content-type": "application/json",
   "host": "mock.api7.ai",
   "user-agent": "curl/7.74.0",
   "x-application-owner": "API7.ai",
   "x-forwarded-for": "127.0.0.1",
   "x-forwarded-host": "127.0.0.1",
   "x-forwarded-port": "9080",
   "x-forwarded-proto": "https",
   "x-real-ip": "159.89.160.194",
   "X-Application-Owner": "API7.ai",
   "Content-Type": "application/json"
 }
}

Send a request with the wrong header:

curl 127.0.0.1:9080/headers -H "version:new"

You shall receive the following response from the baseline upstream:

{
 "headers": {
   "Accept": "*/*",
   "Host": "httpbin.org",
   "User-Agent": "curl/7.74.0",
   "X-Amzn-Trace-Id": "Root=1-6650ab7e-32c90eba787abbeb4e3dbb0c",
   "X-Forwarded-Host": "127.0.0.1"
 }
}

Send a request with no header:

curl 127.0.0.1:9080/headers

You shall receive the following response from the baseline upstream:

{
 "headers": {
   "Accept": "*/*",
   "Host": "httpbin.org",
   "User-Agent": "curl/7.74.0",
   "X-Amzn-Trace-Id": "Root=1-6650ab7e-32c90eba787abbeb4e3dbb0c",
   "X-Forwarded-Host": "127.0.0.1"
 }
}

Update the old upstream with the current upstream to finish the canary traffic shifting:

adc.yaml

services:
  - name: httpbin API
    upstream:
      name: Test Group
      scheme: https
      nodes:
        - host: mock.api7.ai
          port: 443
          weight: 100
    routes:
      - uris:
          - /headers
        name: getting-started-headers
        methods:
          - GET

You can also keep the old upstream as a canary upstream with 0 weight to roll back to if there are issues with the new upstream.

Ingress Controller

ApisixService custom resource is required for the api7-traffic-split plugin and not yet available.

Additional Resources

• Key Concept
  • Services
  • Upstreams