Implement Data Plane Resilience
Resilience refers to a system's ability to withstand and recover from failures, disruptions, or unexpected events.
In this document, you will learn why you should consider data plane (DP) resilience pattern in API7 and how to implement it, such that when the control plane (CP) becomes unavailable, the DP instances can still operate normally. This helps you formulate a disaster recovery plan and quickly resume mission-critical functions when the control plane (CP) becomes unavailable, ensuring the high availability of your system.
Why Consider DP Resilience
DP could encounter issues connecting with CP. The following are a few potential causes:
- Poor network connectivity between DP and CP instances
- CP database crash
- CP upgrade
- Resource contention on the CP host
- CP host hardware failure
DP Resilience Pattern
API7 Enterprise supports configuring CP to periodically dump configurations to AWS S3 buckets, so that in the event of a CP outage, DP can start in the standalone mode and pull the latest gateway configurations from the storage to continue proxying requests.
Once the CP is recovered, DP should continue fetching configurations from the CP.
Implement DP Resilience
Prerequisites
- Install API7 Enterprise.
- Complete Deploy with API7 Ingress Controller on Kubernetes if you would like to deploy resources on Kubernetes.
- Complete Start a Sample Upstream Service and Create Service and Route.
Provision AWS Resources
- Create an AWS account and log in as an IAM user.
- Create two S3 buckets, one for gateway instance configurations, such as keyring and discovery; and one for gateway resource configurations, such as routes and services.
- Obtain the IAM user access key and secret access key.
- Attach the policy that allows read and write access to objects in S3 buckets to the IAM user.
Configure CP to Back Up Configurations
In the working directory where you ran the quickstart command to install API7 Enterprise, you should find a api7-ee
directory, in which there is a docker-compose.yaml
and various service configuration files.
Add the fallback_cp
configuration to the dashboard configuration file:
fallback_cp:
aws_s3:
access_key: your-aws-iam-access-key
region: ap-south-1
config_bucket: bucket-to-push-config-data
resource_bucket: bucket-to-push-resource-data
secret_key: your-aws-iam-secret-access-key
cron_spec: '@every 1m'
❶ Replace with your AWS IAM user access key.
❷ Replace with your AWS region.
❸ Replace with your S3 bucket name where instance configuration details, such as keyring and discovery, should be pushed.
❹ Replace with your S3 bucket name where APISIX resource configurations, such as routes and services, should be pushed.
❺ Replace with your AWS IAM user secret access key.
❻ Configure frequency to push to the S3 buckets to every minute.
Restart the dashboard container:
docker-compose -f api7-ee/docker-compose.yaml restart api7-ee-dashboard
API7 should start pushing CP configurations to the S3 buckets every minute.
Verify
In this section, you will be first sending a request to the route to see the expected upstream response. Then you will restart DP deployment in standalone mode, so that it starts fetching configurations from S3 buckets, and verify that the DP proxies requests as usual.
- Docker
- Kubernetes
Send a request to the route:
curl "http://127.0.0.1:9080/ip"
You should see a response similar to the following:
{
"origin": "127.0.0.1"
}
Suppose the CP has now become unavailable. Add the following configuration to the DP configuration file to start standalone mode:
docker exec <api7-ee-gateway-container-name> /bin/sh -c "echo '
nginx_config:
error_log_level: warn
deployment:
role: data_plane
role_data_plane:
config_provider: yaml
fallback_cp:
aws_s3:
access_key: your-aws-iam-access-key
secret_key: your-aws-iam-secret-access-key
bucket: bucket-to-push-config-data
region: ap-south-1
' > /usr/local/apisix/conf/config.yaml"
Reload the container for configuration changes to take effect:
docker exec <api7-ee-gateway-container-name> apisix reload
The DP instances should start in standalone mode and fetch configurations from the S3 buckets.
Send a request to the route:
curl "http://127.0.0.1:9080/ip"
You should see a response similar to the following, verifying the DP is routing the request from the configurations in S3:
{
"origin": "127.0.0.1"
}
When the CP has recovered, remove the earlier change to the deployment mode:
docker exec <api7-ee-gateway-container-name> /bin/sh -c "echo '
nginx_config:
error_log_level: warn
' > /usr/local/apisix/conf/config.yaml"
Reload the container for configuration changes to take effect:
docker exec <api7-ee-gateway-container-name> apisix reload
DP instances should now start synchronizing configurations from the CP.
Expose the service port to your local machine by port forwarding:
kubectl port-forward svc/api7ee3-apisix-gateway 9080:80 &
Send a request to the route:
curl "http://127.0.0.1:9080/ip"
You should see a response similar to the following:
{
"origin": "127.0.0.1"
}
Suppose the CP has now become unavailable. Edit the DP ConfigMap:
kubectl edit cm api7-ee-3-gateway
Update the configuration to standalone mode in the ConfigMap:
apiVersion: v1
data:
config.yaml: |
...
deployment:
role: data_plane
role_data_plane:
config_provider: yaml
...
Keep a copy of your configuration before modification to switch back quickly later when CP recovers.
Save the ConfigMap and restart the resource:
kubectl rollout restart deployment api7-ee-3-gateway
The DP instances should start in standalone mode and fetch configurations from the S3 buckets.
Send a request to the route:
curl "http://127.0.0.1:9080/ip"
You should see a response similar to the following, verifying the DP is routing the request from the configurations in S3:
{
"origin": "127.0.0.1"
}
When the CP has recovered, edit the DP ConfigMap:
kubectl edit cm api7-ee-3-gateway
Update the deployment mode back to the previous configurations:
apiVersion: v1
data:
config.yaml: |
...
deployment:
role: traditional
role_traditional:
config_provider: etcd
admin:
allow_admin:
- 127.0.0.1/24
etcd:
host:
- "http://192.168.101.10:7900" # replace with your address
prefix: "/apisix"
timeout: 30
...
Save the ConfigMap and restart the resource:
kubectl rollout restart deployment api7-ee-3-gateway
DP instances should now start syncing configurations from the CP.