Version: latest

Data Plane Resilience

API7 Gateway data plane nodes cache configuration in memory and can continue processing traffic during brief control plane (CP) outages. However, for extended outages, such as a database failure, control plane upgrade, or disaster recovery scenario, nodes that restart without a CP connection cannot load configuration.

Use this guide after you have already planned for scaling and high availability of traffic-serving nodes:

Scale Data Plane covers capacity growth.
Data Plane High Availability covers multi-node deployments, load balancers, and health checks.

Fallback CP solves this by periodically exporting all gateway configuration to external storage (AWS S3 or Azure Blob Storage). When the control plane is unreachable, data plane nodes fetch their configuration from this storage, enabling them to start up and continue operating independently.

How It Works

The Fallback CP architecture uses a backup gateway node that runs alongside your traffic-handling nodes but does not serve API traffic. Its sole purpose is to receive configuration from the control plane and export it to external storage on a cron schedule.

Normal operation

During normal operation, every data plane node — including the backup node — watches configuration from the control plane's etcd-compatible API on mTLS port 7943 (see Architecture for details on the pull/watch model). The backup node additionally writes the configuration it receives to external storage on a cron schedule.

Fallback operation

When the control plane is unavailable, traffic nodes fall back to loading configuration from external storage instead of from the control plane.

What Gets Exported

The backup node exports two categories of data per gateway group:

Category	Contents
Resource data	Routes, services, upstreams, consumers, credentials, SSL certificates, global rules, plugin metadata, plugin configs, stream routes, secrets, custom plugins, and protobuf definitions
Config data	Data plane configuration including keyring and service discovery settings

Each gateway group's data is stored as a separate object, keyed by the gateway group's short ID.

info

The backup node does not expose HTTP/HTTPS ports for API traffic and is not counted toward your license quota.

Prerequisites

A running API7 Gateway deployment with at least one gateway group.
An external storage backend:
- AWS S3: Two S3 buckets (one for resource data, one for config data) with appropriate IAM permissions.
- Azure Blob Storage: Two Blob containers (one for resource data, one for config data) with appropriate access credentials.

AWS S3 Permissions

The IAM policy for the backup node must include:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:PutObject"],
      "Resource": [
        "arn:aws:s3:::fallback-cp-data/*",
        "arn:aws:s3:::fallback-cp-config/*"
      ]
    }
  ]
}

Traffic nodes in fallback mode need s3:GetObject on the same buckets.

Azure Blob Storage Permissions

The identity or access key used must have Storage Blob Data Contributor role on both containers, or use a storage account access key.

Configure the Backup Node

The backup node is a gateway instance configured with mode: write under fallback_cp. It connects to the control plane like a regular data plane node but writes configuration to external storage instead of serving traffic.

Kubernetes (Helm)
Docker

Deploy the backup node as a separate Helm release or a dedicated deployment within your existing release.

AWS S3
Azure Blob Storage

Using IAM Roles for Service Accounts (IRSA):

backup-node-values.yaml
replicaCount: 1

gateway_conf:
  deployment:
    fallback_cp:
      interval: 60
      mode: write
      aws_s3:
        region: "us-east-1"
        resource_bucket: "fallback-cp-data"
        config_bucket: "fallback-cp-config"

serviceAccount:
  annotations:
    eks.amazonaws.com/role-arn: "arn:aws:iam::123456789012:role/fallback-cp-role"

❶ The interval (in seconds) between configuration exports.
❷ write mode means this node only exports to storage; it does not serve traffic.
❸ The AWS region where the S3 buckets are located.
❹ ❺ Separate buckets for resource data and configuration data.

Using IAM Access Keys:

backup-node-values.yaml
replicaCount: 1

gateway_conf:
  deployment:
    fallback_cp:
      interval: 60
      mode: write
      aws_s3:
        access_key: "${AWS_ACCESS_KEY_ID}"
        secret_key: "${AWS_SECRET_ACCESS_KEY}"
        region: "us-east-1"
        resource_bucket: "fallback-cp-data"
        config_bucket: "fallback-cp-config"

Using Workload Identity:

backup-node-values.yaml
replicaCount: 1

apisix:
  podLabels:
    azure.workload.identity/use: "true"
  extraEnvVarsSecret: "sc-azurite-secret"

nginx:
  envs:
    - name: AZURE_STORAGEBLOB_RESOURCEENDPOINT
      value: "https://youraccount.blob.core.windows.net"
    - name: AZURE_CLIENT_ID
      value: "<your-managed-identity-client-id>"

gateway_conf:
  deployment:
    fallback_cp:
      interval: 60
      mode: write
      azure_blob:
        account_name: "youraccount"
        resource_container: "fallback-cp-data"
        config_container: "fallback-cp-config"

Using Storage Account Access Key:

backup-node-values.yaml
replicaCount: 1

gateway_conf:
  deployment:
    fallback_cp:
      interval: 60
      mode: write
      azure_blob:
        account_name: "youraccount"
        account_key: "${AZURE_STORAGE_ACCOUNT_KEY}"
        resource_container: "fallback-cp-data"
        config_container: "fallback-cp-config"

AWS S3
Azure Blob Storage

config.yaml
deployment:
  role: data_plane
  role_data_plane:
    control_plane: "https://<cp-host>:7943"
  fallback_cp:
    interval: 60
    mode: write
    aws_s3:
      access_key: "${AWS_ACCESS_KEY_ID}"
      secret_key: "${AWS_SECRET_ACCESS_KEY}"
      region: "us-east-1"
      resource_bucket: "fallback-cp-data"
      config_bucket: "fallback-cp-config"

docker run -d --name api7-backup-node \
  -v $(pwd)/config.yaml:/usr/local/apisix/conf/config.yaml \
  api7/api7-ee-3-gateway:latest

config.yaml
deployment:
  role: data_plane
  role_data_plane:
    control_plane: "https://<cp-host>:7943"
  fallback_cp:
    interval: 60
    mode: write
    azure_blob:
      account_name: "youraccount"
      account_key: "${AZURE_STORAGE_ACCOUNT_KEY}"
      resource_container: "fallback-cp-data"
      config_container: "fallback-cp-config"

docker run -d --name api7-backup-node \
  -v $(pwd)/config.yaml:/usr/local/apisix/conf/config.yaml \
  api7/api7-ee-3-gateway:latest

Configure Traffic Nodes for Fallback

Traffic nodes must be configured to fall back to external storage when the control plane is unreachable. In fallback mode, the gateway switches to standalone mode and reads configuration from the same storage buckets used by the backup node.

Kubernetes (Helm)
Docker

Add the fallback_cp block to your traffic node Helm values (without mode or interval):

AWS S3
Azure Blob Storage

traffic-node-values.yaml
gateway_conf:
  deployment:
    fallback_cp:
      aws_s3:
        region: "us-east-1"
        resource_bucket: "fallback-cp-data"
        config_bucket: "fallback-cp-config"

traffic-node-values.yaml
gateway_conf:
  deployment:
    fallback_cp:
      azure_blob:
        account_name: "youraccount"
        resource_container: "fallback-cp-data"
        config_container: "fallback-cp-config"

When the CP goes down and a traffic node needs to restart in standalone fallback mode:

AWS S3
Azure Blob Storage

config.yaml
deployment:
  role: data_plane
  role_data_plane:
    config_provider: json
  fallback_cp:
    aws_s3:
      access_key: "${AWS_ACCESS_KEY_ID}"
      secret_key: "${AWS_SECRET_ACCESS_KEY}"
      region: "us-east-1"
      resource_bucket: "fallback-cp-data"
      config_bucket: "fallback-cp-config"

❶ Setting config_provider: json switches the gateway to standalone mode, where it reads configuration from external storage instead of connecting to the control plane.

config.yaml
deployment:
  role: data_plane
  role_data_plane:
    config_provider: json
  fallback_cp:
    azure_blob:
      account_name: "youraccount"
      account_key: "${AZURE_STORAGE_ACCOUNT_KEY}"
      resource_container: "fallback-cp-data"
      config_container: "fallback-cp-config"

caution

When traffic nodes are running in fallback mode, configuration changes cannot be applied until the control plane is restored. All nodes will serve the last configuration exported by the backup node.

Control Plane-Side Configuration

Alternatively, the fallback CP push can be configured on the control plane dashboard instead of using a dedicated backup gateway node. The dashboard periodically pushes configuration to external storage on a cron schedule.

Configure this in the dashboard configuration:

dashboard-config.yaml
fallback_cp:
  cron_spec: "@every 1m"
  aws_s3:
    access_key: "${AWS_ACCESS_KEY_ID}"
    secret_key: "${AWS_SECRET_ACCESS_KEY}"
    region: "us-east-1"
    resource_bucket: "fallback-cp-data"
    config_bucket: "fallback-cp-config"
  azure_blob:
    account_name: "youraccount"
    account_key: "${AZURE_STORAGE_ACCOUNT_KEY}"
    resource_container: "fallback-cp-data"
    config_container: "fallback-cp-config"

❶ A cron expression that controls how often the dashboard pushes configuration. Common values: @every 1m, @every 5m.
❷ ❸ You can configure one or both storage backends simultaneously. The dashboard pushes to all configured backends.

info

When multiple dashboard instances are running, a distributed lock ensures that only one instance performs the push at any given time.

Configuration Reference

Backup Node / Traffic Node (`config.yaml`)

Field	Type	Description
`deployment.fallback_cp.mode`	string	`write` for backup nodes. Omit for traffic nodes (fallback/read mode).
`deployment.fallback_cp.interval`	integer	Export interval in seconds (backup node only).
`deployment.fallback_cp.aws_s3.access_key`	string	AWS access key ID. Not required when using IRSA.
`deployment.fallback_cp.aws_s3.secret_key`	string	AWS secret access key. Not required when using IRSA.
`deployment.fallback_cp.aws_s3.region`	string	AWS region for the S3 buckets.
`deployment.fallback_cp.aws_s3.resource_bucket`	string	S3 bucket for resource data.
`deployment.fallback_cp.aws_s3.config_bucket`	string	S3 bucket for configuration data.
`deployment.fallback_cp.aws_s3.endpoint`	string	Custom S3 endpoint (for S3-compatible storage).
`deployment.fallback_cp.azure_blob.account_name`	string	Azure storage account name.
`deployment.fallback_cp.azure_blob.account_key`	string	Azure storage account key. Not required when using Workload Identity.
`deployment.fallback_cp.azure_blob.resource_container`	string	Blob container for resource data.
`deployment.fallback_cp.azure_blob.config_container`	string	Blob container for configuration data.
`deployment.fallback_cp.azure_blob.endpoint`	string	Custom Azure Blob endpoint. Defaults to `https://<account_name>.blob.core.windows.net`.

Dashboard (`dashboard-config.yaml`)

Field	Type	Description
`fallback_cp.cron_spec`	string	Cron expression for the push schedule (for example, `@every 1m`).
`fallback_cp.aws_s3.*`	—	Same fields as the gateway-side `aws_s3` configuration.
`fallback_cp.azure_blob.*`	—	Same fields as the gateway-side `azure_blob` configuration.

Next Steps

Deploy for High Availability: Set up HA for both the control plane and data plane.
Data Plane High Availability: Learn about health checks and multi-node DP architecture.
Scale Data Plane: Add or remove traffic-serving nodes as traffic changes.
Production Best Practices: Harden your deployment for production.

How It Works​

Normal operation​

Fallback operation​

What Gets Exported​

Prerequisites​

AWS S3 Permissions​

Azure Blob Storage Permissions​

Configure the Backup Node​

Configure Traffic Nodes for Fallback​

Control Plane-Side Configuration​

Configuration Reference​

Backup Node / Traffic Node (config.yaml)​

Dashboard (dashboard-config.yaml)​

Next Steps​