# Backup and Recovery

The AISIX Enterprise control plane is recoverable only when its PostgreSQL data and the secrets and configuration that interpret that data are preserved together. A database backup without the matching master keys leaves encrypted credentials unreadable. Configuration files without the matching database cannot reconstruct organizations, resources, or the certificate authority trusted by existing gateways.

Treat the database, encryption keys, deployment configuration, and software version as one recovery set. Keep a matching copy of each asset, protect the set as a secret, and rehearse restoration before relying on it for an upgrade rollback or disaster recovery.

## Choose the Recovery Scope[​](#choose-the-recovery-scope "Direct link to Choose the Recovery Scope")

| Goal                                                          | Required protection                                                                                                                                                                                                                |
| ------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Recover the complete AISIX Enterprise control plane           | Back up the complete PostgreSQL state together with the matching secrets, deployment configuration, and software version.                                                                                                          |
| Roll back an upgrade                                          | Take a recovery set immediately before the upgrade. Restore the database before starting the previous control-plane version.                                                                                                       |
| Move or reproduce one organization                            | Use [Organization Backup and Restore](https://docs.api7.ai/ai-gateway/cloud/organization-backup-and-restore.md). A bundle restores only onto a freshly installed control plane, and does not replace a complete deployment backup. |
| Keep gateways serving during a temporary control-plane outage | Configure [Offline Resilience](https://docs.api7.ai/ai-gateway/cloud/offline-resilience.md). A gateway snapshot cache is a continuity mechanism, not a control-plane backup.                                                       |

## Protect the Recovery Set[​](#protect-the-recovery-set "Direct link to Protect the Recovery Set")

Build the recovery set from the assets below, then store and retain it outside the deployment's failure domain.

### PostgreSQL State[​](#postgresql-state "Direct link to PostgreSQL State")

The control-plane database contains organizations, environments, users, models, policies, encrypted credentials, projected gateway configuration, audit and usage data, and the root certificate authority and gateway-certificate ledgers. Back up the entire database rather than selecting individual application tables.

Use a PostgreSQL-aware backup or a storage snapshot coordinated with PostgreSQL. Do not copy a live database volume as ordinary files. The backup must preserve both the `public` and `auth` schemas and the Kine table created by `dp-manager`.

For an external database, include the connection role, its ownership of the database, and any schema grant your DBA added in the database recovery plan. A per-database dump does not preserve cluster-global roles, so recreate the connection role as described in [Prepare an External Database](https://docs.api7.ai/ai-gateway/on-premises/external-database.md) before restoring, and restore as that role so that it keeps owning the control-plane tables.

A database last run by a supported control plane from 1.6.0 through 1.7.0 can still carry grants to `cp_api_app`, a role those releases used and the current control plane leaves in place. A dump of such a database contains statements that name that role. On a cluster without `cp_api_app`, `pg_restore` reports an error for each of them and restores everything else; restoring with `pg_restore --no-privileges` avoids the errors. `cp-api` reapplies the privileges on its own functions at every start. To remove the role from the source cluster, see [Remove the `cp_api_app` Role](https://docs.api7.ai/ai-gateway/on-premises/upgrade.md#remove-the-cp_api_app-role).

A recovery set taken from a control plane running a release from 1.6.0 through 1.7.0 restores with that release's software, which still uses `cp_api_app` and needs the connection role to have `CREATEROLE` and `BYPASSRLS`. Follow the documentation for that release when restoring it.

See [Prepare an External Database](https://docs.api7.ai/ai-gateway/on-premises/external-database.md) for the role requirements. AISIX does not provide a database-backup command or replace the backup tooling and retention policy operated by your database platform.

For how long the control plane keeps usage, audit, and other growing data, and how to size and maintain the database, see [Database Operations](https://docs.api7.ai/ai-gateway/on-premises/database-operations.md).

### Encryption and Authentication Secrets[​](#encryption-and-authentication-secrets "Direct link to Encryption and Authentication Secrets")

The active master key encrypts stored provider and integration credentials and the root certificate authority's private key. Preserve the active key, its key ID, and every retired key that may still wrap a stored value. Losing required master-key material is not recoverable from the database.

Also preserve the database credentials and, when email is configured, the SMTP password. Keep these values in a secret manager or encrypted backup with access controls at least as strict as the database backup.

### Deployment Configuration[​](#deployment-configuration "Direct link to Deployment Configuration")

Preserve the configuration that locates and exposes the restored services. This includes the public control-plane and `dp-manager` URLs, database connection, private CA mounts, image overrides, and network-access settings. Also preserve any extra environment variables or Kubernetes Secrets supplied outside the default package or chart.

Restoring a database with configuration from another deployment can produce authentication failures, unreadable ciphertext, or a certificate authority that does not match the gateway certificate records.

### Release Identity[​](#release-identity "Direct link to Release Identity")

Record the control-plane package or chart version and the effective image references at the time of backup. Restore that version first. Starting a newer version can migrate the restored database before you have verified the recovery, while starting an older version against a newer schema is unsupported.

Keep version-pinned packages or a tested way to retrieve them, especially for air-gapped deployments. Record intentional image overrides separately because they can outlive normal package or chart upgrades.

### Storage, Retention, and Access[​](#storage-retention-and-access "Direct link to Storage, Retention, and Access")

A recovery set is a logical match between the database, secrets, configuration, and software version. The artifacts do not have to share one storage location. Separating encryption keys from encrypted database backups can reduce the impact of one storage compromise, provided the recovery inventory identifies the matching copies.

Keep at least one recovery copy outside the deployment's host, cluster, and storage failure domain. Define the recovery point objective, backup frequency, retention period, and deletion controls for each copy. Encrypt backups in transit and at rest, restrict access, and record checksums or another integrity signal with the recovery-set manifest.

Periodically verify that retained package or chart versions and every external secret source are still available. A database backup that outlives its master keys or matching software is not recoverable.

## Back Up a Docker Compose Deployment[​](#back-up-a-docker-compose-deployment "Direct link to Back Up a Docker Compose Deployment")

The packaged deployment stores PostgreSQL data in the `aisix-self-hosted_pgdata` volume and keeps deployment secrets and settings in `./aisix-self-hosted/.env`. The data volume survives `./run.sh down` and package extraction, but `.env` is not included in later package archives and cannot be reconstructed from the volume.

Preserve these assets together:

| Asset                                                                        | Why it is required                                                                                                        |
| ---------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------- |
| Complete PostgreSQL backup or consistent `aisix-self-hosted_pgdata` snapshot | Restores all control-plane and certificate state.                                                                         |
| `.env`                                                                       | Preserves the database password, master key and key ID, SMTP settings, URLs, release pin, and image or runtime overrides. |
| Matching package or version-pinned package location                          | Restores the scripts, Compose definition, and default image set for the database schema you backed up.                    |
| External files and local deployment customization                            | Restores private CA certificates and other operator-managed inputs that are not stored in PostgreSQL.                     |

For a cold volume snapshot, stop the stack so PostgreSQL is not writing, snapshot the complete volume, and then restart the same installation. For an online backup, use PostgreSQL-aware tooling that provides a consistent restore point. Verify the backup can be read before returning the deployment to normal change activity.

Do not treat the package archive as a secret backup. Extracting another package preserves an existing `.env`; it does not recreate a lost one.

## Back Up a Helm Deployment[​](#back-up-a-helm-deployment "Direct link to Back Up a Helm Deployment")

For Helm installations, preserve the same recovery set through the systems that own each asset:

| Asset                                                | Where to protect it                                                                                                                                              |
| ---------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Built-in PostgreSQL                                  | Back up the PostgreSQL PVC with a database-aware procedure or a coordinated volume snapshot.                                                                     |
| External PostgreSQL                                  | Use the database service's backup, point-in-time recovery, and role-management procedures.                                                                       |
| Master key and SMTP password                         | Preserve the Secret rendered by the chart, any existing Secret named by `api.smtp.existingSecret`, and the protected source from which its values were supplied. |
| Database and optional serving-role credentials       | Preserve the configured existing Secrets or the protected values source used to render them.                                                                     |
| Effective Helm values and release history            | Retain the values source, chart version, namespace, release name, image tags, and intentional overrides.                                                         |
| Additional mounted Secrets, ConfigMaps, and CA files | Back them up through their owning secret or configuration system.                                                                                                |

The chart marks its primary application Secret with `helm.sh/resource-policy: keep`, so a normal Helm uninstall does not remove it. That annotation does not protect the Secret from namespace or cluster loss. Back up its source or an encrypted copy independently.

Treat exported Helm values as sensitive because they may contain database passwords and application secrets. When values reference existing Secrets, backing up the values alone does not preserve the Secret contents.

## Take a Change-Scoped Backup[​](#take-a-change-scoped-backup "Direct link to Take a Change-Scoped Backup")

Before an upgrade or another schema-affecting change:

1. Identify the current control-plane version and effective image overrides.
2. Capture a consistent PostgreSQL backup.
3. Capture the matching Compose `.env` or Helm values and secret sources.
4. Record the backup time, database identity, deployment name, software version, storage location, and integrity checksum as one recovery-set manifest.
5. Verify that the backup artifacts are readable and that the required credentials and version-pinned software remain available.

Do not make unrelated configuration or key-rotation changes between these captures. If the database or its encryption-key set changes, take a new recovery set.

## Restore the Control Plane[​](#restore-the-control-plane "Direct link to Restore the Control Plane")

Restore into an isolated target first when possible. Prevent the recovered control plane from sending notifications, reaching production providers, or accepting production gateway connections until you have verified the result.

1. Select one complete recovery set and obtain its matching package or chart version.
2. Restore the deployment configuration and secret sources, including all master keys needed by the database.
3. Restore PostgreSQL while the control-plane services are stopped. For an external database, recreate the documented connection role and privileges first. If the backup includes grants that reference `cp_api_app`, see [PostgreSQL State](#postgresql-state).
4. Start the matching control-plane version through its normal Compose or Helm deployment path.
5. Review startup logs and confirm that `cp-api` and `dp-manager` become healthy without database, master-key, migration, or `ca_root` errors.
6. Sign in with an account from the restored database and verify organizations, environments, resources, and gateway registrations. A restored database already has users, so `cp-api` creates no first administrator, whatever the first-administrator settings say.
7. Confirm that gateways can authenticate to `dp-manager` and that [resource projection](https://docs.api7.ai/ai-gateway/cloud/resource-projection.md) reaches the expected revision.
8. When a safe test provider is reachable, send a controlled request through a non-production gateway and confirm the response and configured telemetry.
9. Re-enable external integrations and production gateway connectivity only after these checks pass.

Restore the database before starting a rollback version. The `AISIX_ALLOW_DOWNGRADE=1` package override acknowledges an intentional rollback; it does not make an incompatible or already-migrated database safe. Follow [Upgrade AISIX Enterprise](https://docs.api7.ai/ai-gateway/on-premises/upgrade.md#roll-back) for the installation-specific rollback procedure.

If the database has gateway certificate rows but no `ca_root`, prefer restoring a complete database backup. The fresh-bootstrap override creates a new authority and forces every gateway to re-enroll; it does not recover the missing key. See [Recover a Missing CA Root](https://docs.api7.ai/ai-gateway/reference/on-premises-configuration.md#recover-a-missing-ca-root).

### Roll Back a Helm Control Plane Upgrade[​](#roll-back-a-helm-control-plane-upgrade "Direct link to Roll Back a Helm Control Plane Upgrade")

Use this procedure after rolling back the gateways as described in [Upgrade AISIX Enterprise](https://docs.api7.ai/ai-gateway/on-premises/upgrade.md#roll-back). Set the control-plane release, namespace, and rollback timeout:

```
export CP_NAMESPACE="aisix"

export CP_RELEASE="aisix-cp"

export CP_UPGRADE_TIMEOUT="35m"
```

List the release history and identify the successful revision from before the upgrade:

```
helm history "$CP_RELEASE" --namespace "$CP_NAMESPACE"
```

Stop the control-plane API and data-plane manager writers:

```
kubectl scale deployment \

  --namespace "$CP_NAMESPACE" \

  --selector "app.kubernetes.io/instance=$CP_RELEASE,app.kubernetes.io/component in (api,dpm)" \

  --replicas 0
```

The scale command returns before the pods finish terminating. Wait until they are deleted:

```
kubectl wait pod \

  --namespace "$CP_NAMESPACE" \

  --selector "app.kubernetes.io/instance=$CP_RELEASE,app.kubernetes.io/component in (api,dpm)" \

  --for=delete \

  --timeout="$CP_UPGRADE_TIMEOUT"
```

Complete steps 1–3 of [Restore the Control Plane](#restore-the-control-plane), keeping the writers stopped after restoring PostgreSQL. Then roll the workloads back to the pre-upgrade chart revision:

```
helm rollback "$CP_RELEASE" <pre-upgrade-revision> \

  --namespace "$CP_NAMESPACE" \

  --wait \

  --timeout "$CP_UPGRADE_TIMEOUT"
```

Do not let workloads from the newer release restart between the database restore and the Helm rollback. Complete steps 5–9 of [Restore the Control Plane](#restore-the-control-plane) against the restored release.

## Rehearse Recovery[​](#rehearse-recovery "Direct link to Rehearse Recovery")

A backup is not proven until it has been restored. Rehearse in an isolated environment on a schedule that matches the deployment's risk and change rate.

During a rehearsal:

* restore the database, secrets, configuration, and matching software from the recorded recovery set;
* verify that encrypted resources load without master-key errors;
* confirm that the certificate authority and gateway-certificate records are present;
* sign in and inspect representative organization and environment resources;
* connect a non-production gateway and verify resource projection;
* send a controlled request and verify the response and configured telemetry;
* record restore time, required manual actions, and any missing external dependency.

After the rehearsal, correct the backup process rather than documenting an unrepeatable workaround. Take a new recovery set whenever the process or required asset list changes.

## What Does Not Replace a Deployment Backup[​](#what-does-not-replace-a-deployment-backup "Direct link to What Does Not Replace a Deployment Backup")

* A gateway snapshot cache preserves the last accepted gateway configuration for temporary control-plane loss. It cannot reconstruct the control plane.
* A redacted organization bundle intentionally removes live credentials and certificate-authority state.
* A full organization bundle covers one organization and restores only onto a freshly installed control plane, in place of that installation's first organization and administrator. It is portable but not a complete control-plane backup.
* A package archive contains software and deployment templates, not the generated `.env` or PostgreSQL data.
* Helm release metadata does not replace the database or externally managed Secret contents.
