Version: 2.13.2304

High Availability

This post guides you on deploying API7 Enterprise Edition with a minimum version of high availability, including high availability on the Data Plane (API7 Gateway), Control Plane (API7 Dashboard), and the etcd cluster.

Get Production Packages

info

The all-in-one installation package provided for PoC is not suitable for high-availability deployments and is not applicable to this documentation.

Architecture

Build a high availability architecture with the following components (minimum version):

High availability architecture

High availability of the data plane is critical to avoid any disruption to your APIs when unexpected failures occur. Therefore, we highly recommend deploying more than two instances of the data plane to prevent a single point of failure.
High availability of the etcd cluster is also necessary as it stores important data and configuration. As per the etcd Raft protocol, a minimum of three nodes is required for the etcd cluster.
The high availability of the control plane is optional. If the control plane fails, it will not affect the data plane and your end users, and consumers can consume online APIs normally. The only impact is that you cannot update the API configuration. Therefore, having only one control plane is acceptable if you are concerned about cost.
High availability for OpenSearch, Prometheus, Grafana, AlertManager, and confd is optional, as it won't affect data planes according to the flexible architecture. We will introduce this at the end of this post.

Prerequisites

Virtual Machines

info

A high-availability architecture requires seven deployed nodes (excluding OpenSearch, Prometheus, Grafana, AlertManager, Confd, and other components.) typically.
This guide uses five nodes by deploying the etcd cluster and API7 Dashboard on the same node. However, You can choose to reduce the number of nodes based on your budget.

For each node, we use the minimum configuration requirements.

Hostname	Components	Port	Specification	Operating System	Docker Version (Docker Only)
api7-highavailability1	etcd, API7-Dashboard	9000, 2379, 2380	2C4G	CentOS 7.6 or higher	20.10.7 or higher
api7-highavailability2	etcd, API7-Dashboard	9000, 2379, 2380	2C4G	CentOS 7.6 or higher	20.10.7 or higher
api7-highavailability3	etcd	2379, 2380	2C4G	CentOS 7.6 or higher	20.10.7 or higher
api7-highavailability4	API7-Gateway	80, 443, 9091	2C4G	CentOS 7.6 or higher	20.10.7 or higher
api7-highavailability5	API7-Gateway	80, 443, 9091	2C4G	CentOS 7.6 or higher	20.10.7 or higher

OS and Docker version

Docker v20.10.7 is incompatible with 3.10.0-327 (CentOS 7.2) and below. 3.10.0-927 (CentOS 7.6) or higher is recommended.

Execute the following command to check the current kernel version number.

$ uname -a

Linux api7-highavailability1 3.10.0-1160.76.1.el7.x86_64 #1 SMP Wed Aug 10 16:21:17 UTC 2022 x86_64 x86_64 x86_64 x86_64 GNU/Linux

You can upgrade the kernel using:

yum update -y kernel

Network

Configure SELinux and firewall services (firewalld or iptables) on each node to allow access to the ports.

API7 Gateway:
- 80: For HTTP traffic.
- 443: For HTTPS traffic.
- 9091: Connect to Prometheus for monitoring.
API7 Dashboard:
- 9000: Access to API7 Enterprise Dashboard.
etcd:
- 2379: Store and provide data and configurations.
- 2380: For peer communication inside a cluster.

Load Balancer

We recommend using a load balancer to distribute traffic to the API7 Gateway nodes and API7 Dashboard nodes. You can use any load balancer that supports TCP and HTTP/HTTPS.

Export IP

To reuse the IP address, you can export the IP address of each node as an environment variable.

# Use VM's Private IP for security.

export ha_ip1=${VM_IP1}
export ha_ip2=${VM_IP2}
export ha_ip3=${VM_IP3}

Deploy via Docker

Please make sure each VM has Docker installed. If not, please refer to Install Docker.

etcd

Step1: Login api7-highavailability1, api7-highavailability2, api7-highavailability3,create etcd_data directory.

sudo mkdir -p /usr/local/etcd/etcd_data

Step2: Start three etcd nodes

sudo docker run --restart always --name etcd \
  -u root \
  -v /usr/local/etcd/etcd_data:/etcd_data \ 
  -e ALLOW_NONE_AUTHENTICATION=yes \
  -e ETCD_NAME=api7-etcd-1 \
  -e ETCD_LISTEN_PEER_URLS=http://0.0.0.0:2380 \
  -e ETCD_LISTEN_CLIENT_URLS=http://0.0.0.0:2379 \
  -e ETCD_ADVERTISE_CLIENT_URLS=http://$ha_ip1:2379 \
  -e ETCD_INITIAL_ADVERTISE_PEER_URLS=http://$ha_ip1:2380 \
  -e ETCD_INITIAL_CLUSTER_TOKEN=etcd-cluster \
  -e ETCD_INITIAL_CLUSTER=api7-etcd-1=http://$ha_ip1:2380,api7-etcd-2=http://$ha_ip2:2380,api7-etcd-3=http://$ha_ip3:2380 \
  -e ETCD_DATA_DIR=/etcd_data \
  -e ETCD_ENABLE_V2=true \
  --network=host \
  -d bitnami/etcd:3.4.13

Step3: Verification

$ docker exec etcd bash -c "etcdctl --endpoints $ha_ip1:2379,$ha_ip2:2379,$ha_ip3:2379 endpoint health"

28.0.14:2379 is healthy: successfully committed proposal: took = 1.468755ms
28.0.16:2379 is healthy: successfully committed proposal: took = 2.154079ms
28.0.15:2379 is healthy: successfully committed proposal: took = 3.133916ms

API7 Dashboard

Step1: Copy api7-2.13.2304-cp.tar.gz to api7-highavailability1, api7-highavailability2,and execute the following command.

tar -zxvf api7-2.13.2304-cp.tar.gz

# load docker image
docker load < images/api7-dashboard-2.13.2304.tar.gz

Step2: Deploy API7 Dashboard

Configure the etcd cluster configuration in dashboard_conf/conf.yaml on api7-highavailability1, api7-highavailability2 and then start the API7 Dashboard.

conf:
  etcd:
    name: "default"
    endpoints:
      - ${ha_ip1}:2379
      - ${ha_ip2}:2379
      - ${ha_ip3}:2379
    enable_auth: false
    username: "root"      # remove etcd username if not enable etcd auth
    password: "123456"    # remove etcd password if not enable etcd auth
    mtls:
      enable: false
      key_file: ""          # Path of your self-signed client side key
      cert_file: ""         # Path of your self-signed client side cert
      ca_file: ""           # Path of your self-signed CA cert, the CA is used to sign callers' certificates
    prefix: /api7           # key prefix in etcd

Step3: Start API7 Dashboard.

docker compose up api7-dashboard -d

Step4: Verification

$ docker ps 

CONTAINER ID   IMAGE                                          COMMAND                  CREATED         STATUS        PORTS                                       NAMES
17a250b6f6d5   api7ee.azurecr.io/api7-dashboard:2.13.2304   "/usr/local/apisix-d…"   2 seconds ago   Up 1 second   0.0.0.0:9000->9000/tcp, :::9000->9000/tcp   api7-21322091-cp-api7-dashboard-1

2166063e6f7e   bitnami/etcd:3.4.13                            "/entrypoint.sh etcd"    6 hours ago     Up 6 hours                                                etcd

[xxx@api7-highavailability1 api7-2.13.2304-cp]$ docker logs -f 17a250b6f6d5
The manager-api is running successfully!

Version : 2.13.2304
GitHash : f2c8f46
Listen  : 0.0.0.0:9000
Loglevel: warn
Logfile : /usr/local/apisix-dashboard/logs/error.log

Step6: Create a new cluster according to the Guide.

API7 Gateway

Step1: Copy api7-2.13.2304-dp.tar.gz to api7-highavailability4, api7-highavailability5,and execute the following command.

tar -zxvf api7-2.13.2304-dp.tar.gz

# load docker image
docker load < images/api7-gateway-2.13.2304.tar.gz

Step2: Deploy API7 Gateway:

Configure the etcd cluster configuration in gateway_conf/config.yaml on api7-highavailability4, api7-highavailability5 and then start the API7 Gateway.

The cluster_id in the configuration file must be the same as the cluster_id created in the previous step.

etcd:
  host:
    - ${ha_ip1}:2379
    - ${ha_ip2}:2379
    - ${ha_ip3}:2379
  prefix: /api7/${cluster_id}

Step3: Start API7-Gateway

docker compose up -d

Step4: Verification

$ curl ${API7_Gateway_IP}:80/get -H "Host: test.com"
{
  "args": {},
  "headers": {
    "Accept": "*/*",
    "Host": "test.com",
    "User-Agent": "curl/7.29.0",
    "X-Amzn-Trace-Id": "Root=1-63e46ced-6c448fe974210a5f327d268e",
    "X-Forwarded-Host": "test.com"
  }, 
  "origin": "172.28.0.16, 20.39.184.151", 
  "url": "http://test.com/get"
}

Deploy via CentOS

etcd

Step1: Login api7-highavailability1, api7-highavailability2, api7-highavailability3 and run:

tar -zxvf api7-ee-2.13.2304.tar.gz

# Deploy etcd
cd etcd

sudo sh install.sh --ip ${VM_PRIVATE_IP}

Step2: Verification

systemctl status etcd

Loaded: loaded (/etc/systemd/system/etcd.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2023-03-29 15:30:02 UTC; 1h 38min ago
 Main PID: 9022 (etcd)
   CGroup: /system.slice/etcd.service
           └─9022 /usr/local/bin/etcd --config-file=/etc/etcd/etcd.conf.yaml

API7 Dashboard

Step1: Login api7-highavailability1, api7-highavailability2 and run:

cd dashboard

sudo sh install.sh --listen-ip ${VM_PUBLIC_OR_PRIVATE_IP} --etcd-ip ${ha_ip1},${ha_ip2},${ha_ip3} --prometheus-ip 0.0.0.0 --opensearch-ip 0.0.0.0

Step2: Verification

systemctl status api7-dashboard

api7-dashboard.service - api7-dashboard
   Loaded: loaded (/etc/systemd/system/api7-dashboard.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2023-03-29 15:41:56 UTC; 1h 34min ago
 Main PID: 10818 (manager-api)
   CGroup: /system.slice/api7-dashboard.service
           └─10818 /usr/local/apisix-dashboard/manager-api

API7 Gateway

Step1: Login api7-highavailability4, api7-highavailability5 and run:

The cluster_id in the configuration file must be the same as the cluster_id created in the previous step.

tar -zxvf api7-ee-2.13.2304.tar.gz

cd gateway

sudo sh install.sh --etcd-ip ${ha_ip1},${ha_ip2},${ha_ip3} --cluster-prefix ${cluster-id} --self-ip 127.0.0.1

Step2: Verification

ps -ef | grep nginx
 
root     12944     1  0 17:14 ?        00:00:00 nginx: master process /bin/openresty -p /usr/local/apisix -c /usr/local/apisix/conf/nginx.conf
nobody   12945 12944  0 17:14 ?        00:00:00 nginx: worker process
nobody   12946 12944  0 17:14 ?        00:00:00 nginx: worker process
root     12947 12944  0 17:14 ?        00:00:00 nginx: privileged agent process

Validate high availability

Before the two API7-Dashboard nodes, we need to mount a control plane load balancer, and before the two API7-Gateway nodes, we need to mount a data plane load balancer. You can use any load balancer product.

To verify the high availability of API7 Enterprise, check the following three scenarios.

Manually disable the API7-Gateway service

Docker installation

# Please replace the version number in accordance with your downloaded files.
[xxx@api7-highavailability4 ~]$ cd api7-2.13.2304-dp/

[xxx@api7-highavailability4 api7-2.13.2304-dp]$ ls

cli.sh  docker-compose.yaml  gateway_conf  gateway_logs  images

[zwx@api7-highavailability4 api7-2.13.2304-dp]$ docker compose down

[+] Running 2/2
 ⠿ Container api7-21322091-dp-api7-gateway-1  Removed                                                                                                                       10.2s
 ⠿ Network api7-21322091-dp_api7              Removed                                                                                                                        0.0s

[xxx@api7-highavailability4 api7-2.13.2304-dp]$ docker ps

CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES

RPM installation

[xxx@api7-highavailability4 ~]$ sudo apisix stop

/usr/local/openresty/luajit/bin/luajit /usr/local/apisix/apisix/cli/apisix.lua stop

[zwx@api7-highavailability4 ~]$ ps -ef | grep nginx
xxx      19573 19365  0 02:15 pts/0    00:00:00 grep --color=auto nginx

Validation

Access the data plane load balancer to see if the API7 Gateway service is working.

Manually disable the API7 Dashboard service

Docker installation

$ cd api7-2.13.2304-cp/

$ docker compose down

RPM installation

sudo systemctl stop api7-dashboard

sudo systemctl status api7-dashboard

api7-dashboard.service - api7-dashboard
   Loaded: loaded (/etc/systemd/system/api7-dashboard.service; enabled; vendor preset: disabled)
   Active: inactive (dead) since Thu 2023-03-30 02:16:29 UTC; 16s ago
  Process: 10818 ExecStart=/usr/local/apisix-dashboard/manager-api (code=exited, status=0/SUCCESS)
 Main PID: 10818 (code=exited, status=0/SUCCESS)

Validation

Access the control plane load balancer and data plane load balancer to see if the API7-Dashboard service is working.

Manually disable the etcd service

Docker installation

[xxx@api7-highavailability1 api7-2.13.2304-cp]$ docker ps

CONTAINER ID   IMAGE                                          COMMAND                  CREATED         STATUS         PORTS                                       NAMES
d3b572696f50   api7ee.azurecr.io/api7-dashboard:2.13.2304   "/usr/local/apisix-d…"   4 minutes ago   Up 4 minutes   0.0.0.0:9000->9000/tcp, :::9000->9000/tcp   api7-21322091-cp-api7-dashboard-1
2166063e6f7e   bitnami/etcd:3.4.13                            "/entrypoint.sh etcd"    24 hours ago    Up 24 hours                                                etcd

[xxx@api7-highavailability1 api7-2.13.2304-cp]$ docker stop 2166063e6f7e

2166063e6f7e

[xxx@api7-highavailability1 api7-2.13.2304-cp]$ docker ps

CONTAINER ID   IMAGE                                          COMMAND                  CREATED         STATUS         PORTS                                       NAMES
d3b572696f50   api7ee.azurecr.io/api7-dashboard:2.13.2304   "/usr/local/apisix-d…"   4 minutes ago   Up 4 minutes   0.0.0.0:9000->9000/tcp, :::9000->9000/tcp   api7-21322091-cp-api7-dashboard-1

RPM installation

sudo systemctl stop etcd

sudo systemctl status etcd

 etcd.service - ETCD
   Loaded: loaded (/etc/systemd/system/etcd.service; enabled; vendor preset: disabled)
   Active: inactive (dead) since Thu 2023-03-30 02:17:25 UTC; 5s ago
  Process: 9022 ExecStart=/usr/local/bin/etcd --config-file=/etc/etcd/etcd.conf.yaml (code=killed, signal=TERM)
 Main PID: 9022 (code=killed, signal=TERM)

Validation

Access the control plane load balancer, then read and write data.

HA of Other Components (Optional)

Prometheus, AlertManager and Confd

Typically, Prometheus and AlertManager are deployed separately from API7 Dashboard and API7 Gateway and can be connected with your existing insatnces.

Check official guide to learn its high availability, and make sure to add all nodes to the confd_conf/templates/api7.gateway.yaml.tmpl configuration:

alerting:
  alertmanagers:
    - static_configs:
        - targets:
            - alertmanager:9093
            - alertmanager1:9093

Grafana

Check Grafana official guide to learn its high availability.

Since you have multiple nodes of Grafana, the configuration of database needs to be MySQL or PostgreSQL, serving as the shared database access for all Grafana services.

# vim grafana_conf/config/grafana.ini

[database]
url = mysql://root:123456@localhost:3306/grafana

OpenSearch

Check OpenSearch official guide to learn its high availability.

Since you have multiple nodes of OpenSearch, make sure to add all nodes to the dashboard_conf/conf.yaml configuration:

  opensearch:
    hosts:
      - "http://opensearch:9200"
      - "http://opensearch1:9200"
      - "http://opensearch2:9200"

Get Production Packages​

Architecture​

Prerequisites​

Virtual Machines​

OS and Docker version​

Network​

Load Balancer​

Export IP​

Deploy via Docker​

etcd​

API7 Dashboard​

API7 Gateway​

Deploy via CentOS​

etcd​

API7 Dashboard​

API7 Gateway​

Validate high availability​

Manually disable the API7-Gateway service​

Docker installation​

RPM installation​

Validation​

Manually disable the API7 Dashboard service​

Docker installation​

RPM installation​

Validation​

Manually disable the etcd service​

Docker installation​

RPM installation​

Validation​

HA of Other Components (Optional)​

Prometheus, AlertManager and Confd​

Grafana​

OpenSearch​

Get Production Packages

Architecture

Prerequisites

Virtual Machines

OS and Docker version

Network

Load Balancer

Export IP

Deploy via Docker

etcd

API7 Dashboard

API7 Gateway

Deploy via CentOS

etcd

API7 Dashboard

API7 Gateway

Validate high availability

Manually disable the API7-Gateway service

Docker installation

RPM installation

Validation

Manually disable the API7 Dashboard service

Docker installation

RPM installation

Validation

Manually disable the etcd service

Docker installation

RPM installation

Validation

HA of Other Components (Optional)

Prometheus, AlertManager and Confd

Grafana

OpenSearch