Skip to main content
API7.ai Logo
Version: 3.9.x

Data Plane Resilience

API7 Gateway data plane nodes cache configuration in memory and can continue processing traffic during brief control plane (CP) outages. However, for extended outages, such as a database failure, control plane upgrade, or disaster recovery scenario, nodes that restart without a CP connection cannot load configuration.

Use this guide after you have already planned for scaling and high availability of traffic-serving nodes:

Fallback CP stores gateway configuration in AWS S3, S3-compatible storage, or Azure Blob Storage. During an outage, an operator can restart or redeploy a traffic node in fallback read mode so it loads the stored snapshot instead of connecting to the control plane.

Fallback CP does not switch a running node automatically when its control-plane connection fails. A node that remains running continues with its in-memory configuration. Use fallback read mode when a node must start while the control plane is unavailable, and restore its original deployment configuration after the control plane recovers.

How It Works​

Fallback CP snapshots can be produced by either a backup gateway node or the scheduled Control Plane pusher. The backup gateway runs alongside your traffic-handling nodes but does not serve API traffic. Its sole purpose is to receive configuration from the control plane and export it to external storage on a schedule. The following diagram shows this option.

Normal Operation​

During normal operation, every data plane node, including the backup node, watches the control plane's etcd-compatible API on mTLS port 7943. See Architecture for details on the pull/watch model. The backup node also writes the configuration it receives to external storage on a schedule.

Fallback Operation​

When the control plane is unavailable, an operator redeploys or restarts a traffic node in fallback read mode. At startup, the node loads the resource and data plane configuration objects for its gateway-group short ID from external storage.

What Gets Exported​

The backup node or Control Plane pusher exports two categories of data per gateway group:

CategoryContents
Resource dataRoutes, services, upstreams, consumers, credentials, SSL certificates, global rules, plugin metadata, plugin configs, stream routes, secrets, custom plugins, and protobuf definitions
Config dataData plane configuration including keyring and service discovery settings

Each gateway group's data is stored as a separate object, keyed by the gateway group's short ID. Recovery nodes must use the same API7_GATEWAY_GROUP_SHORT_ID as the snapshot they load.

info

The chart still creates a gateway Service and publishes its configured ports for a backup release. However, the gateway recognizes write mode at startup and omits its HTTP/HTTPS and stream traffic server blocks. Keep the backup Service out of load balancers, Ingress routes, DNS records, and client configuration. The backup node is not counted toward the license quota.

Prerequisites​

  • A running API7 Gateway deployment with at least one gateway group.
  • An external storage backend:
    • AWS S3 or S3-compatible storage: Two buckets (one for resource data, one for config data) with appropriate access permissions. For another S3-compatible service, prepare the custom endpoint URL.
    • Azure Blob Storage: Two Blob containers (one for resource data, one for config data) with appropriate access credentials.

AWS S3 Permissions​

The IAM policy for a backup node or Dashboard pusher must include:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:PutObject"],
      "Resource": [
        "arn:aws:s3:::fallback-cp-data/*",
        "arn:aws:s3:::fallback-cp-config/*"
      ]
    }
  ]
}

Traffic nodes in fallback mode need s3:GetObject on the same buckets.

For S3-compatible storage, grant equivalent object read/write permissions in your storage provider.

Azure Blob Storage Permissions​

Grant Storage Blob Data Contributor on both containers to a backup gateway that uses Workload Identity. A recovery identity only needs Storage Blob Data Reader. The Control Plane pusher requires a storage account access key; an account access key authorizes access directly and is not assigned an Azure RBAC role.

Static Credentials​

The Kubernetes static-key examples expect an existing Secret named fallback-storage-credentials in every namespace that contains a backup gateway, recovery gateway, or Dashboard pusher that consumes it.

ProviderRequired Secret keys
AWS S3aws-access-key-id, aws-secret-access-key
Azure Blob Storageazure-storage-account-key

Create and populate this Secret through your approved secret-management workflow before installing or upgrading a release. Kubernetes Secrets are namespace-scoped, so create the Secret separately when gateway and control-plane releases use different namespaces. The Docker examples expect the equivalent AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, or AZURE_STORAGE_ACCOUNT_KEY variables to contain non-empty values in the host environment or secret-injection system before the container starts.

Configure the Backup Node​

The backup node is a gateway instance configured with mode: write under fallback_cp. It connects to the control plane like a regular data plane node but writes configuration to external storage instead of serving traffic.

Deploy one backup node per gateway group as a separate Helm release. Start with the values generated by the Dashboard for that gateway group. This preserves its DP Manager endpoints, mTLS Secret, gateway-group environment variable, image, and other connection settings. Add the fallback values below to that complete values file.

The chart reads fallback settings from the top-level deployment.fallback_cp value. In write mode, it keeps the DP Manager connection and renders node_listen, container ports, and gateway Service ports. It disables readiness and liveness probes and HTTP and stream access logs. At startup, the gateway reports itself as a backup node and omits the NGINX traffic server blocks. Keep the published gateway Service out of traffic paths so the release remains explicitly isolated.

Using IAM Roles for Service Accounts (IRSA):

backup-node-values.yaml
apisix:
  kind: Deployment
  replicaCount: 1

deployment:
  fallback_cp:
    interval: 60
    mode: write
    aws_s3:
      region: "us-east-1"
      resource_bucket: "fallback-cp-data"
      config_bucket: "fallback-cp-config"

serviceAccount:
  create: true
  annotations:
    eks.amazonaws.com/role-arn: "arn:aws:iam::123456789012:role/fallback-cp-role"

❶ The interval (in seconds) between configuration exports.
❷ write mode enables storage export. The chart retains port declarations, while the gateway omits its traffic server blocks.
❸ The AWS region where the S3 buckets are located.
❹ ❺ Separate buckets for resource data and configuration data.

Using IAM Access Keys:

Keep the Dashboard-generated apisix.extraEnvVars entries and append the storage credentials to that list. The complete list must retain API7_GATEWAY_GROUP_SHORT_ID and any other generated entries:

backup-node-values.yaml
apisix:
  kind: Deployment
  replicaCount: 1
  extraEnvVars:
    - name: API7_GATEWAY_GROUP_SHORT_ID
      value: "YOUR_GATEWAY_GROUP_SHORT_ID"
    - name: AWS_ACCESS_KEY_ID
      valueFrom:
        secretKeyRef:
          name: fallback-storage-credentials
          key: aws-access-key-id
    - name: AWS_SECRET_ACCESS_KEY
      valueFrom:
        secretKeyRef:
          name: fallback-storage-credentials
          key: aws-secret-access-key

deployment:
  fallback_cp:
    interval: 60
    mode: write
    aws_s3:
      access_key: '${{AWS_ACCESS_KEY_ID}}'
      secret_key: '${{AWS_SECRET_ACCESS_KEY}}'
      region: "us-east-1"
      resource_bucket: "fallback-cp-data"
      config_bucket: "fallback-cp-config"

For S3-compatible storage, use the provider's access key and secret key, and add endpoint to the aws_s3 block:

deployment:
  fallback_cp:
    aws_s3:
      endpoint: "https://s3-compatible.example.com"
      use_path_style: true

Enable use_path_style only when the storage provider requires path-style bucket URLs.

Configure the Control Plane Pusher​

As an alternative to a backup gateway, the Dashboard process can export every gateway group on a schedule. This path does not require one backup node per group, but it depends on a running Dashboard and database when each snapshot is created.

Helm​

Configure the pusher under dashboard_configuration.fallback_cp. Merge the credential entries into the complete existing dashboard.extraEnvVars list. Helm replaces lists, so a credential-only overlay can remove required entries such as a Secret-backed database DSN.

For AWS S3:

cp-values.yaml (AWS S3)
dashboard:
  extraEnvVars:
    # Keep every existing Dashboard environment variable above or below these entries.
    - name: AWS_ACCESS_KEY_ID
      valueFrom:
        secretKeyRef:
          name: fallback-storage-credentials
          key: aws-access-key-id
    - name: AWS_SECRET_ACCESS_KEY
      valueFrom:
        secretKeyRef:
          name: fallback-storage-credentials
          key: aws-secret-access-key

dashboard_configuration:
  fallback_cp:
    cron_spec: "@every 1m"
    aws_s3:
      access_key: '${AWS_ACCESS_KEY_ID}'
      secret_key: '${AWS_SECRET_ACCESS_KEY}'
      region: "us-east-1"
      resource_bucket: "fallback-cp-data"
      config_bucket: "fallback-cp-config"
      use_path_style: false

For S3-compatible storage, add custom_endpoint to the same block. Change use_path_style to true only when the provider requires path-style bucket URLs:

dashboard_configuration:
  fallback_cp:
    aws_s3:
      custom_endpoint: "https://s3-compatible.example.com"
      use_path_style: true

For Azure Blob Storage:

cp-values.yaml (Azure Blob Storage)
dashboard:
  extraEnvVars:
    # Keep every existing Dashboard environment variable above or below this entry.
    - name: AZURE_STORAGE_ACCOUNT_KEY
      valueFrom:
        secretKeyRef:
          name: fallback-storage-credentials
          key: azure-storage-account-key

dashboard_configuration:
  fallback_cp:
    cron_spec: "@every 1m"
    azure_blob:
      account_name: "youraccount"
      account_key: '${AZURE_STORAGE_ACCOUNT_KEY}'
      resource_container: "fallback-cp-data"
      config_container: "fallback-cp-config"

The Control Plane pusher requires explicit access keys. It does not use the gateway image's IAM or Workload Identity credential chain.

Docker or Packages​

Configure the same producer in the Dashboard configuration file. Populate the referenced environment variables through the process supervisor or approved secret-injection mechanism before starting every Dashboard replica:

dashboard-config.yaml
fallback_cp:
  cron_spec: "@every 1m"
  aws_s3:
    access_key: "${AWS_ACCESS_KEY_ID}"
    secret_key: "${AWS_SECRET_ACCESS_KEY}"
    region: "us-east-1"
    resource_bucket: "fallback-cp-data"
    config_bucket: "fallback-cp-config"
    use_path_style: false
  azure_blob:
    account_name: "youraccount"
    account_key: "${AZURE_STORAGE_ACCOUNT_KEY}"
    resource_container: "fallback-cp-data"
    config_container: "fallback-cp-config"

❶ A schedule expression that controls how often the Dashboard pushes configuration. Common values include @every 1m and @every 5m. ❷ ❸ Configure one or both storage backends. The Dashboard pushes to every configured backend.

For S3-compatible storage, add custom_endpoint to the Dashboard-side aws_s3 block. Change use_path_style to true only when the provider requires path-style bucket URLs:

fallback_cp:
  aws_s3:
    custom_endpoint: "https://s3-compatible.example.com"
    use_path_style: true
info

When multiple Dashboard instances are running, a distributed lock ensures that only one instance performs the push at any given time.

Recover Traffic Nodes from Fallback Storage​

Before an outage, keep the complete normal deployment values or generated Docker command for every gateway group. Confirm that the selected backup method has written one object named with the gateway-group short ID to both the resource and configuration locations.

When the control plane is unavailable and a traffic node must restart, explicitly start that node in fallback read mode. The gateway loads the stored snapshot during startup and does not connect to DP Manager in this mode.

Create a small fallback values file and apply it together with the saved normal values for the release. Keep the Dashboard-generated API7_GATEWAY_GROUP_SHORT_ID environment variable and storage identity or credential configuration.

fallback-read-values.yaml
serviceAccount:
  create: true
  annotations:
    eks.amazonaws.com/role-arn: "arn:aws:iam::123456789012:role/fallback-cp-reader-role"

deployment:
  fallback_cp:
    mode: read
    aws_s3:
      region: "us-east-1"
      resource_bucket: "fallback-cp-data"
      config_bucket: "fallback-cp-config"

For S3-compatible storage, add endpoint to the aws_s3 block. Set use_path_style to true only when the provider requires path-style bucket URLs:

deployment:
  fallback_cp:
    aws_s3:
      endpoint: "https://s3-compatible.example.com"
      use_path_style: true

The examples above create provider-specific service accounts for recovery. Grant the AWS role s3:GetObject on both buckets, or grant the Azure identity Storage Blob Data Reader on both containers. If your normal values already select a service account, merge the required annotations and Azure pod label into that existing identity instead of replacing it.

If traffic nodes use access keys, append the storage credential entries to the existing apisix.extraEnvVars list in normal-values.yaml and add the corresponding ${{...}} fields to fallback-read-values.yaml. Do not define a second apisix.extraEnvVars list in the fallback file: Helm replaces lists instead of appending them, which would remove generated entries such as API7_GATEWAY_GROUP_SHORT_ID.

Apply the fallback file over the complete saved values and use the same chart version as the normal release:

helm upgrade YOUR_GATEWAY_RELEASE api7/gateway \
  --version YOUR_GATEWAY_CHART_VERSION \
  -n YOUR_NAMESPACE \
  -f normal-values.yaml \
  -f fallback-read-values.yaml \
  --wait

The chart changes role_data_plane.config_provider to json and restarts the pods. At startup, each pod uses the gateway-group short ID to fetch the resource and data plane configuration objects. Do not remove the generated short-ID environment variable from normal-values.yaml.

caution

Fallback mode serves the stored snapshot and does not receive later control-plane changes.

  • For Helm, restore the release from normal-values.yaml without the fallback overlay. Wait for the replacement pods to report Healthy, then verify current configuration and traffic.
  • For Docker, keep other HA nodes serving traffic, drain the fallback node from the load balancer, and remove the fallback container so it releases ports 9080 and 9443. Start the original pinned Dashboard-generated command, wait for the new instance to report Healthy, verify current configuration and traffic, then return the node to the load balancer.

Verify Backup and Recovery​

Test the complete workflow in a non-production gateway group before relying on fallback storage:

  1. Create a distinctive test route and wait for the next backup interval or scheduled Dashboard run.
  2. Inspect the backup gateway or Dashboard logs and confirm that the resource object and data plane configuration object were both written successfully during the same run. Do not recover from a run where either write failed.
  3. Confirm that both storage locations contain an object named with the gateway-group short ID. The resource object should contain the test route; the configuration object should contain the data plane configuration for the same group. Compare their last-modified times or version IDs with the successful run. Enable object versioning where the storage provider supports it, and retain the matching pair needed for recovery.
  4. Save the normal deployment values or Docker command, then make the control plane unavailable to the test node.
  5. Start that node in fallback read mode and send a request directly to the test route. Confirm that authentication, plugins, upstreams, and service discovery used by the group behave as expected.
  6. Restore the normal deployment configuration after the control plane recovers. Wait for the instance to report Healthy, apply a small configuration change, and confirm that the node receives it before returning the node to production traffic.
caution

Fallback objects contain gateway resources and data plane configuration, including credential-bearing plugin fields, private keys, and the gateway-group keyring in a form the gateway can consume. Encrypt the storage backend, restrict read and write access, audit access, and apply a retention policy appropriate for configuration backups.

Configuration Reference​

Backup Node / Traffic Node (config.yaml)​

FieldTypeDescription
deployment.fallback_cp.modestringwrite for backup nodes or read for traffic nodes that start from fallback storage. The default is read.
deployment.fallback_cp.intervalintegerExport interval in seconds (backup node only).
deployment.fallback_cp.aws_s3.access_keystringAWS access key ID. Not required when using IRSA.
deployment.fallback_cp.aws_s3.secret_keystringAWS secret access key. Not required when using IRSA.
deployment.fallback_cp.aws_s3.regionstringAWS region for the S3 buckets.
deployment.fallback_cp.aws_s3.resource_bucketstringS3 bucket for resource data.
deployment.fallback_cp.aws_s3.config_bucketstringS3 bucket for configuration data.
deployment.fallback_cp.aws_s3.endpointstringCustom S3 endpoint (for S3-compatible storage).
deployment.fallback_cp.aws_s3.use_path_stylebooleanWhether to use path-style S3 bucket URLs. The default is false.
deployment.fallback_cp.azure_blob.account_namestringAzure storage account name.
deployment.fallback_cp.azure_blob.account_keystringAzure storage account key. Not required when using Workload Identity.
deployment.fallback_cp.azure_blob.resource_containerstringBlob container for resource data.
deployment.fallback_cp.azure_blob.config_containerstringBlob container for configuration data.
deployment.fallback_cp.azure_blob.endpointstringCustom Azure Blob endpoint. Defaults to https://<account_name>.blob.core.windows.net.

Dashboard (dashboard-config.yaml)​

FieldTypeDescription
fallback_cp.cron_specstringSchedule expression for configuration pushes (for example, @every 1m).
fallback_cp.aws_s3.access_keystringAWS access key ID.
fallback_cp.aws_s3.secret_keystringAWS secret access key.
fallback_cp.aws_s3.regionstringAWS region for the S3 buckets.
fallback_cp.aws_s3.resource_bucketstringS3 bucket for resource data.
fallback_cp.aws_s3.config_bucketstringS3 bucket for configuration data.
fallback_cp.aws_s3.custom_endpointstringCustom S3 endpoint for S3-compatible storage. In the API7 control plane Helm chart, configure this as dashboard_configuration.fallback_cp.aws_s3.custom_endpoint.
fallback_cp.aws_s3.use_path_stylebooleanWhether to use path-style bucket URLs. The default is false; enable it only when the storage provider requires path-style URLs. In the API7 control plane Helm chart, configure this as dashboard_configuration.fallback_cp.aws_s3.use_path_style.
fallback_cp.azure_blob.*—Same fields as the gateway-side azure_blob configuration.

Next Steps​