Upgrade AISIX Enterprise

AISIX Enterprise is upgraded in two parts: the control plane and the AISIX gateways that serve traffic. They are upgraded independently in a fixed order, and the releases they run may differ while you work through the fleet. The supported paths and exact procedure depend on the installation method; see the release notes for changes in each release.

Before changing the control plane, prepare its database, secrets, deployment configuration, and matching software as one recovery set. See Backup and Recovery. For a deployment on OKD, complete the checks in OKD before running the upgrade.

AISIX Enterprise Supported Upgrade Paths

The AISIX Enterprise supported upgrade floor is 0.12.0. An AISIX Enterprise deployment running 0.12.0 or later upgrades directly to any newer release, including the current one.

Releases may be skipped. Upgrading 0.12.0 straight to the current release is supported and does not require stepping through the intermediate releases. The floor is what makes the jump supported, not the distance: any source at or above 0.12.0 reaches any newer target in one step.

The floor moves only when a release announces that it moves. It is not raised silently as new releases ship, so a deployment that is supported today stays supported until an announcement says otherwise. Check the release notes of the version you are upgrading to before planning a jump from an old deployment.

A deployment below 0.12.0 upgrades in two steps: first to a release between 0.12.0 and the version you are on your way to, letting it start and finish its database migrations, then to your target.

Compatibility exceptions are per feature, not per version: what can differ during an upgrade is one configuration setting, never the release pairing as a whole. Two things can differ, and they are not the same. A setting a gateway release cannot load would cost that gateway the whole resource — between 0.12.0 and the current release there is no such setting, so every configuration the control plane can save loads as a whole resource on every gateway from the floor up. A setting a gateway release does not recognize is the ordinary case during an upgrade: the resource loads and serves normally, and that one setting does nothing on that gateway until it is upgraded.

AISIX Enterprise Upgrade Order and the Mixed-Version Window

Upgrade the control plane first, then the gateways. The control plane understands every gateway release from the floor up; a gateway does not necessarily understand a control plane newer than itself.

Between the two steps the deployment runs mixed versions: a control plane on the new release and gateways still on the old one. That window is bounded by the floor, not by a clock. It may last as long as you need it to — a staged fleet rollout, a canary environment held back for a week, a gateway that is upgraded on its own maintenance schedule — provided every gateway stays at or above 0.12.0. Gateways serve traffic normally throughout.

Gateway versions are reported per instance in the environment's Data planes view, so you can see which instances are still behind. The control-plane version is shown at the bottom of the dashboard's left navigation and by GET /api/config/public; see Verify the Installation.

Saving a setting that some gateways in scope do not recognize is allowed. The response names the field, the first supported gateway release that recognizes it, and how many gateways will ignore it. If no supported release recognizes the field yet, the response says so. The dashboard displays the same warning after the save. Read it before you rely on the setting: a restrictive setting that is ignored is not enforced on those gateways.

After configuration reaches the gateways, the dashboard also shows what each gateway reports. The environment overview distinguishes ignored fields from whole-resource rejections, and Data planes marks affected nodes as Limitations or Incompatible. These runtime signals can reveal a gateway that registered or rolled back after the original save. See Resource Projection for the verification workflow.

On the gateway side, aisix_config_partially_compatible_resources counts resources carrying fields the gateway's own version does not recognize. It can be non-zero during the window and should return to zero once the gateways are upgraded. See Metrics Reference for the configuration-health metrics.

Read the Upgrade Notes for Every Release You Skip

Skipping a release skips its release notes, not its behavior. When you upgrade across several releases at once, every intermediate release's Upgrade Notes apply to you, and they apply in order — read them oldest first, from the release after the one you are on through the release you are upgrading to.

Upgrade Notes are the pre-upgrade actions: a configuration key whose meaning changed, a Helm value that starts being honored, an alert expression to review. A note in a release you jumped over is not superseded by the notes of a later release.

Before upgrading to 1.0.0 from an earlier release, also work through Review Existing Guardrails When Upgrading to 1.0.0, which is a pre-upgrade checklist rather than a note: it asks you to identify and disable affected semantic guardrails before the control plane starts under the new version.

Upgrade from 1.5.x or Earlier: Sign-In, Access, and Dashboard Changes

Releases after 1.5.x move sign-in handling into cp-api, serve the dashboard from cp-api instead of a separate dashboard service, and run the control plane in the on-premises deployment mode. Review these changes before upgrading a control plane from 1.5.x or earlier.

danger

If the control plane you are upgrading holds more than one organization, the dashboard remains unavailable after the upgrade until an owner keeps one organization and permanently deletes all the others. Read Consolidate Several Organizations into One and back up the database before you start.

Existing Sessions End

Sessions from the release you upgrade from are not carried over, so every user signs in again once after the upgrade. Accounts and passwords are unchanged: the same email address and password work. Rolling back to that release also signs everyone out.

Google and GitHub Sign-In Are Removed

The dashboard offers email and password sign-in only. A user who only ever signed in with Google or GitHub has no password and cannot sign in after the upgrade. There is no self-service way for such a user to recover access yet: the password reset email is sent only to accounts that already have a password. Identify these users before upgrading.

The social sign-in settings no longer have any effect: api.oauthEnabled and secrets.oauth.* in the Helm chart, and the AISIX_CLOUD_OAUTH_ENABLED, AISIX_GOOGLE_OAUTH_*, and AISIX_GITHUB_OAUTH_* variables.

Removed Settings

These settings are no longer read. A values file or .env that still sets them installs and starts normally, and you can remove them:

Helm value	Variable	Note
secrets.betterAuthSecret	BETTER_AUTH_SECRET, BETTER_AUTH_URL	Sessions are no longer signed with a shared secret. Keep the value while a rollback to the release you upgraded from is still possible, because 1.5.x and earlier require it.
api.jwtAudience	AISIX_CLOUD_JWT_AUDIENCE	The dashboard no longer uses session JWTs.
—	AISIX_AUTH_REQUIRE_EMAIL_VERIFICATION	Removed.
ui.image.*, ui.replicaCount, ui.resources, and the other ui workload values	AISIX_UI_IMAGE	There is no dashboard image or workload.
—	AISIX_CLOUD_DASHBOARD_URL	cp-api serves the dashboard files from its own image.

The Dashboard Is Served by cp-api

There is no aisix-cp-ui image, dashboard pod, or dashboard container. The dashboard files ship in the cp-api image, so remove aisix-cp-ui from image mirroring and allow-lists.

• Helm. helm upgrade removes the dashboard Deployment. The <release>-ui Service stays, with the same name, type, port, and nodePort, and now forwards to cp-api, so a reverse proxy or NodePort that still points at it keeps working. New configurations should point at the cp-api Service.
• Docker Compose. The ui service is removed from the Compose file, and run.sh removes the old ui container during the upgrade.
• Trusted origins. Additional sign-in origins are now api.trustedOrigins. An AISIX_TRUSTED_ORIGINS entry in ui.extraEnvVars, where the 1.5.0 installation instructions placed it, is still passed to cp-api; move it to api.trustedOrigins when convenient.
• Language. ui.defaultLocale in Helm and AISIX_DASHBOARD_LOCALE in Docker Compose keep setting the dashboard language, and now set the email language too. If you set AISIX_DASHBOARD_LOCALE directly on the 1.5.x dashboard container instead, set ui.defaultLocale or AISIX_CLOUD_DEFAULT_LOCALE on cp-api.
• Private database CA. The dashboard no longer connects to the database, so the 1.5.x ui.extraVolumes, ui.extraVolumeMounts, and NODE_EXTRA_CA_CERTS settings for a private PostgreSQL CA no longer apply. Give the CA to cp-api and dp-manager with externalDatabase.sslRootCert instead; see Trust a Private Database CA.

Dashboard Links Change Form

Pages that carried an ID in the path now take it as a query parameter, such as /environments/<page>?env=<id> instead of /environments/<id>/<page>. Team, MCP server tool, and invitation pages change the same way. cp-api redirects the old links, including invitation links sent before the upgrade, to the new form.

Email Is Available

The control plane can now send invitation and password reset emails through an SMTP relay. Email stays off until you configure one; see Configure Email (SMTP). Without it, the invitation card no longer reports that an email was sent: it shows the link for you to pass on.

Sign-Up Is by Invitation Only

Nobody can sign up on their own any more. A person who does not have an account yet needs an invitation from an administrator, and registers through the link in it; see Invite a Member. Existing accounts and their passwords are unchanged. A client other than the dashboard that signed invitees up with only their email address must now send the invitation's token as invitationToken.

Organization Creation Is Closed

Nobody can create a new organization, from the dashboard or the API. An owner can now delete an organization, except the deployment's last one. A control plane that holds more than one organization has to be reduced to one before the dashboard can be used again; see Consolidate Several Organizations into One.

Consolidate Several Organizations into One

An on-premises control plane serves a single organization, but 1.5.x and earlier let anyone create organizations, so an upgraded control plane can hold several. While several organizations remain, users cannot access the dashboard: it sends every page to /consolidate, and dashboard requests return 403 with the error code ORG_CONSOLIDATION_REQUIRED. Admin tokens keep working, and gateway traffic is not affected.

danger

Consolidation permanently deletes every organization except the one you keep, including organizations you are not a member of, and everything in them: their environments with their models, caller API keys, and other resources, their members' memberships, and their usage and audit records. The certificates of every gateway connected to their environments are revoked, so those gateways are cut off from the control plane. There is no undo. If anyone needs the data in those organizations, back up the database before you consolidate.

Any user who owns at least one organization can consolidate, signed in with a password:

1. Sign in with your password. The dashboard opens the Keep one organization page.
2. Choose the organization to keep. Only organizations you own are offered.
3. Review the organizations that will be deleted. The page lists every other organization on the control plane by name and identifier.
4. Type the identifier of the organization you keep, then select Delete. The button names how many organizations are deleted.

Each deleted organization is removed the same way as when an owner deletes an organization: access to it ends at once, and its stored data, such as usage and audit records, is removed in the background afterwards.

When only one organization remains, the block lifts and the dashboard opens the organization you kept. The consolidation is recorded in that organization's audit log with the resource type organization and the action consolidate.

A user who owns no organization sees a page saying that an organization owner must sign in and complete the consolidation.

Accounts with a password are kept. An account without a password that belonged only to deleted organizations, such as a member created directly or provisioned through SCIM, is deleted with them. A person with a password who belonged only to deleted organizations can still sign in, but belongs to no organization and sees a message to contact an administrator until they are invited into the remaining one. Nobody is added to the kept organization automatically.

If the consolidation stops part-way, the organizations it already deleted stay deleted and the dashboard remains unavailable. Run it again to delete the rest.

Billing Disappears

The billing pages, the organization plan, and the billing permission in the role catalog are gone.

Administrators Carry Over

An upgraded control plane keeps its existing accounts, organizations, owners, and admins, and they sign in as before. Because the database already has users, cp-api creates no first administrator, even though a package .env or the chart may now carry first-administrator settings. Existing owners and admins invite new people, and can enable sign-in or reset passwords for members.

Outbound Connections to Private Addresses Are Allowed

The playground, budget notifications, the MCP spec_url fetch, and the guardrail connection test now reach private, internal, and loopback addresses by default. To keep refusing them, set the corresponding value to false; see Private Network Access.

A Helm release upgraded from chart 1.5.0 with helm upgrade --reuse-values still carries that chart's false defaults for api.playgroundAllowPrivateIPs and api.notifyAllowPrivateURLs as explicit values. Those two keep refusing private addresses until you clear them, for example by upgrading with your own values file as described in Upgrade with Helm.

Organization Restore Needs a Fresh Installation

The unauthenticated import into an empty database is gone, and so is the import into a deployment that is already in use. A bundle now restores only onto a freshly installed control plane, by its first administrator after the initial password change. A redacted restore also no longer sets the password aisix-repro on restored accounts or makes aisix-repro-<api_key_id> authenticate: enable sign-in for the accounts and rotate the caller API keys instead. See Organization Backup and Restore.

Upgrade a Package Installation

The Docker Compose packages — the online quickstart package and the offline package — are upgraded the same way: extract the new package over the existing installation directory and re-run run.sh.

Create a complete recovery set first. It must include the PostgreSQL backup, matching master keys and secret sources, deployment configuration, and software version. cp-api migrates the schema on its first start under the new version, and rollback depends on restoring this complete set.

An online installation upgrades by re-running the quickstart from the directory that contains the installation — it fetches the current package, extracts it over aisix-self-hosted, and runs run.sh:

curl -sL "https://run.api7.ai/aisix-self-hosted/quickstart" | bash

An air-gapped installation needs the offline package carried in, and the package is published per CPU architecture. Download the version-pinned package that matches the air-gapped host on a connected machine:

curl -fSL "https://run.api7.ai/aisix-self-hosted/aisix-self-hosted-offline-1.6.0-linux-amd64.tar.gz" \
  -o aisix-self-hosted-offline-1.6.0-linux-amd64.tar.gz

Replace amd64 with arm64 for an AArch64 host. The URL without an architecture suffix, aisix-self-hosted-offline-1.6.0.tar.gz, keeps working and is the linux/amd64 package. Upgrade a host with the package for its own architecture: run.sh refuses a mismatched package before loading any image, and prints the URL of the right one.

Extract it over the installation directory from the directory that contains it, not from inside it:

cd /path/that/contains/aisix-self-hosted
tar -xzf aisix-self-hosted-offline-1.6.0-linux-amd64.tar.gz
cd aisix-self-hosted
./run.sh

Running tar from inside aisix-self-hosted unpacks a nested copy instead, which leaves the original installation untouched.

The package contains no .env, so this replaces the Compose file and scripts while leaving your secrets and the data volume in place. The one value in .env that belongs to the package is AISIX_VERSION, which pins every image — including the gateway image the dashboard hands out in generated install commands. run.sh moves it to match the package and prints what it did:

==> Upgrading AISIX 1.0.0 -> 1.1.0.
    AISIX_VERSION in .env now matches this package; your secrets are untouched.

run.sh prints one of two lines here. The one above is the normal case; a .env that carried no AISIX_VERSION at all gets ==> Added AISIX_VERSION=<version> to .env (it had none). instead, and that upgrade proceeds too. If neither line appears, nothing was upgraded. Confirm what is actually running:

COMPOSE_PROJECT_NAME=aisix-self-hosted docker compose images

Individual image overrides — AISIX_API_IMAGE, AISIX_DPM_IMAGE, and AISIX_CLOUD_DP_IMAGE — are never rewritten by run.sh and continue to override AISIX_VERSION. Remove an override before upgrading if you want the component to follow the package. See AISIX Enterprise Configuration.

An offline package also carries its images. Extracting the new package adds the new version's image archives to images/ but leaves the previous versions' archives there too, and run.sh runs docker load on every archive in images/ each time it starts the stack. Once the upgrade has started successfully, delete the previous versions' archives from images/ so that later starts load only the current ones. Keep them until then: they are what a rollback on an air-gapped host would load.

Extracting the new package into a different directory is the one variation to avoid. The Compose project name belongs to the package, so the new directory would adopt the same data volume while sitting next to a brand-new .env. run.sh stops rather than starting: it refuses as soon as it finds containers for this stack running from another directory, whatever the new .env holds. Go back to the original directory and upgrade there — or, if you mean to move the installation, run ./run.sh down in the original directory first and copy its .env across before running the new one.

Upgrade with Helm

The control-plane chart and gateway chart ship with the same version and appVersion in every release. Upgrade the control plane first, verify it while the existing gateways continue to serve traffic, and then upgrade each gateway release.

Install Helm 3.15 or a later Helm 3 release, cURL, and jq. Plan a control-plane maintenance window, and create a complete recovery set whether the database is bundled or external. Verify that the database backup in the set can be restored before continuing.

These commands use aisix-cp as the control-plane release name and aisix as its namespace, matching Install AISIX Enterprise. If you chose a different release name or namespace during installation, use those values instead. Run helm list -A to find every release and its namespace: the control plane's CHART column is aisix-cp-<version>, and a gateway's is aisix-<version>. A wrong release name or namespace makes helm upgrade fail with "<name>" has no deployed releases.

Set the control-plane release, namespace, target version, and timeout:

export CP_NAMESPACE="aisix"
export CP_RELEASE="aisix-cp"
export TARGET_VERSION="1.6.0"
export CP_UPGRADE_TIMEOUT="35m"

The timeout exceeds the chart's default control-plane startup budget. Increase it if database migrations can take longer.

Record the installed control-plane chart and application versions before making changes:

helm list --namespace "$CP_NAMESPACE"
kubectl get pods --namespace "$CP_NAMESPACE"

Refresh the chart repository and confirm that the target chart is available:

helm repo add api7 https://charts.api7.ai
helm repo update
helm show chart api7/aisix-cp --version "$TARGET_VERSION"

Use your own values file rather than --reuse-values. The latter carries resolved defaults from the source chart into the target release and can prevent new defaults, including probe budgets, from taking effect.

Point the examples at the values file maintained for the release:

export CP_VALUES_FILE="your-cp-values.yaml"

If you do not have a maintained file, export only the user-supplied values with restrictive permissions:

export CP_VALUES_FILE="cp-values.yaml"

umask 077
touch "$CP_VALUES_FILE"
chmod 600 "$CP_VALUES_FILE"
helm get values "$CP_RELEASE" \
  --namespace "$CP_NAMESPACE" \
  -o yaml > "$CP_VALUES_FILE"

Do not export computed defaults with helm get values --all. Carrying source-chart defaults forward can prevent a target-chart default from taking effect. Treat the values file as sensitive because it can contain credentials.

Check whether the installed release pins either control-plane image:

helm get values "$CP_RELEASE" \
  --namespace "$CP_NAMESPACE" \
  -o json \
  | jq '{api_image_tag: (.api.image.tag // ""), dpm_image_tag: (.dpm.image.tag // ""), generated_gateway_image: (.api.dpImage // "")}'

An empty tag uses the target chart's application version. If a tag pins an older image, remove it from the values file or update it to the target version. Preserve a deliberate custom image repository, and use a target-compatible tag for a custom build.

The generated_gateway_image value controls the gateway image that the dashboard places in generated installation instructions. Docker and Compose instructions use the complete value. Helm instructions use its repository as image.repository, while the installed api7/aisix chart version determines the tag. An empty value follows the control-plane release. Clear or update an older pin so new Docker or Compose installations do not start on the previous image; preserve it only when it identifies a deliberate target-compatible custom build.

Resolve the control-plane API Deployment name from the Helm release label:

export CP_API_DEPLOYMENT="$(kubectl get deployment -n "$CP_NAMESPACE" \
  -l "app.kubernetes.io/instance=$CP_RELEASE,app.kubernetes.io/component=api" \
  -o jsonpath='{.items[0].metadata.name}')"

printf 'Control-plane API: %s\n' "$CP_API_DEPLOYMENT"

Stop and verify the release name, namespace, and Kubernetes context if the value is empty.

Render the upgrade against the live cluster before changing the release. --dry-run=server permits cluster lookups and OpenAPI validation, while --hide-secret prevents Helm from printing rendered Secrets to the terminal or CI log:

helm upgrade "$CP_RELEASE" api7/aisix-cp \
  --namespace "$CP_NAMESPACE" \
  --version "$TARGET_VERSION" \
  --values "$CP_VALUES_FILE" \
  --dry-run=server \
  --hide-secret

This command does not submit the rendered objects to Kubernetes admission. Treat it as a rendering and OpenAPI check, not as validation of SCCs, quotas, or admission policies. Resolve every reported error before continuing.

Inspect every image in the target control-plane render:

helm template "$CP_RELEASE" api7/aisix-cp \
  --namespace "$CP_NAMESPACE" \
  --version "$TARGET_VERSION" \
  --values "$CP_VALUES_FILE" \
  | grep -E '^[[:space:]]+image:'

The aisix-cp-api and aisix-cp-dpm images should use the target version or a deliberate target-compatible custom tag. PostgreSQL is an independently versioned chart dependency. Keep the PostgreSQL image tag rendered by the target chart unless a separately tested database-image plan requires another tag.

Upgrade the control plane without --atomic. The new API migrates the database during startup, while an automatic Helm rollback changes only Kubernetes objects and does not restore the complete recovery set.

helm upgrade "$CP_RELEASE" api7/aisix-cp \
  --namespace "$CP_NAMESPACE" \
  --version "$TARGET_VERSION" \
  --values "$CP_VALUES_FILE" \
  --wait \
  --timeout "$CP_UPGRADE_TIMEOUT"

Watch the rollout. If the API pod does not become ready, describe the non-ready pod first and read its logs if the container started:

kubectl rollout status -n "$CP_NAMESPACE" \
  "deployment/$CP_API_DEPLOYMENT" \
  --timeout="$CP_UPGRADE_TIMEOUT"

kubectl get pods -n "$CP_NAMESPACE" \
  -l "app.kubernetes.io/instance=$CP_RELEASE,app.kubernetes.io/component=api"

kubectl describe pod -n "$CP_NAMESPACE" <non-ready-cp-api-pod>

kubectl logs -n "$CP_NAMESPACE" \
  <non-ready-cp-api-pod> \
  --all-containers \
  --tail=300

A failed database migration, an unreachable database, or a missing Secret can prevent the API from becoming ready. Do not repeatedly rerun the upgrade. If the migration cannot complete, follow Roll Back to restore the complete pre-upgrade recovery set.

The data-plane manager can briefly log missing-table errors while the API applies database migrations. It retries automatically. Investigate the errors if they continue after the API becomes ready.

Verify the control-plane version through its public endpoint. Set AISIX_CP to the existing AISIX Cloud Admin API base URL, including /api and without a trailing slash:

export AISIX_CP="YOUR_AISIX_CONTROL_PLANE_API_URL"

curl -fsS "$AISIX_CP/config/public" | jq '{cp_version, cp_commit}'

Sign in to the dashboard and confirm that organizations, environments, models, provider keys, and caller API keys are present. In each environment's Data planes view, confirm that the existing gateways remain healthy.

After the control plane passes verification, upgrade every managed gateway by following Upgrade an AISIX Gateway on Kubernetes. Use the same TARGET_VERSION for the control-plane and gateway charts. For each release, switch the Kubernetes context when it runs in another cluster and set its release name and namespace. The gateway procedure includes image and values checks, a readiness-gated rolling update, traffic verification, and rollback steps.

Keep the complete recovery set until the control plane and every gateway have operated normally through the observation period defined by your change policy.

Move Application Settings into Chart Values

Both charts declare every application setting as an explicit value and render it into a configuration file that the component reads from a ConfigMap. Secrets stay in Kubernetes Secrets and never enter that file. extraEnvVars is meant only for system-level variables, such as TZ.

If an earlier release of a chart received application settings through extraEnvVars, they keep working after the upgrade, because an environment variable overrides the configuration file. It also overrides the value that now configures the same setting, so move each one to its value and remove the environment variable:

• Control-plane chart. For example, AISIX_TRUSTED_ORIGINS becomes api.trustedOrigins and AISIX_CLOUD_MCP_SPEC_ALLOW_PRIVATE_URLS becomes api.mcpSpecAllowPrivateURLs. See Helm Values for the full list.
• Gateway chart. An AISIX_<SECTION>__<KEY> variable becomes the same key under config, such as config.observability.log_level for AISIX_OBSERVABILITY__LOG_LEVEL. The keys the chart sets from its own values, such as the listener addresses and the control-plane connection, are refused under config. Redis URLs, usernames, and passwords are read from a Secret through configSecrets or rateLimit.redis instead. See Set Any Other Gateway Configuration.

An override of a variable that the chart sets itself, rather than one that only overrides the configuration file, keeps working too, but only as a compatibility path. For a name that the 1.5.0 chart rendered, the chart keeps its own entry ahead of yours, so the container lists the name twice. Helm 4 server-side apply rejects a repeated name, so a new installation sets the value in values.yaml, through its own value or under config, and never through extraEnvVars. Move such overrides into values as well.

Removing an override of a name the chart still renders takes two helm upgrade runs. The deployed pod specification carries the name twice, and the new one carries it once, so the first Helm 3 upgrade deletes both entries and the second puts the chart's own entry back. This limitation has existed since chart 1.5.0.

The first gateway-chart upgrade that renders the configuration file replaces every gateway pod once, even when no value changed, because the pod specification changes: the gateway reads the rendered file, and the certificate bundle from the control plane is mounted as files. A release whose extraEnvVars sets the AISIX_MANAGED__CP_CERT_PEM, AISIX_MANAGED__CP_KEY_PEM, or AISIX_MANAGED__CP_CA_PEM variables keeps receiving the bundle as environment variables.

Index Builds After a Control-Plane Upgrade

A release can add indexes to the control plane's large tables, above all dpmgr_usage_events, which holds the usage records. cp-api starts serving as soon as its schema migration completes and builds those indexes afterwards in the background, so the dashboard and configuration delivery to gateways are available while the build runs. A PostgreSQL statement_timeout set on the control plane's database role does not interrupt the build.

The build takes time in proportion to the usage table. As a rough guide from a test deployment, a table with 50 to 60 million usage rows takes about 15 minutes; your disks, cache, and database load decide the actual time. On a new installation the tables are empty and the build finishes almost at once.

cp-api logs each index as it goes: building index, then periodic index build progress lines, then index built. When the last one is done it logs background index build: every index is in place. Until the indexes they read through are built, budget spend accounting, usage and data retention cleanup, and organization deletion cleanup wait. Requests are still recorded in the meantime, and budget spend catches up in one step when the build ends.

Each index is built with two passes over the table and writes write-ahead log, so the database does noticeably more I/O and CPU work during the build. In one measurement, rebuilding the largest usage index took about 23 seconds, wrote about 180 MB of write-ahead log, and kept PostgreSQL's CPU near its limit, with no measurable change in usage-write or Usage page latency. Upgrade a deployment with a large usage table in an off-peak window all the same.

Wait for background index build: every index is in place before starting another upgrade. A newer release that has to change the same table waits for the earlier build to finish before its own migration can complete.

Package Installations

./run.sh brings the stack up one layer at a time and waits for each layer itself, so it can say what is taking long. It first stops every container the new package replaces, including the previous release's dp-manager, so nothing writes to the database while the new cp-api migrates. It then starts PostgreSQL and waits for it, starts cp-api and waits for it, and finally starts dp-manager and the remaining services.

A normal start prints nothing extra. Once a layer has taken more than about 10 seconds, run.sh prints one line every 30 seconds:

• cp-api is still migrating the database (…). During a long migration it can add It is waiting for a table lock another database session holds; ./run.sh logs api names that session.
• cp-api is still starting (…), when the migration is done and the first start after an upgrade is still bringing up the rest.
• PostgreSQL is recovering from an unclean shutdown (…). See PostgreSQL Recovery After an Interrupted Stop.

Leave it running. Pressing Ctrl-C does not stop the migration, which continues inside the container, but only run.sh starts dp-manager, so run ./run.sh again once the migration is done. To follow the background index build afterwards, run ./run.sh logs api.

Table Locks Held by Another Session

Each migration step waits at most 5 seconds for a table lock, then retries, instead of queueing every read and write on the table behind a long-running transaction. While it retries, cp-api logs database migration is waiting for a table lock held by another session; retrying with the blocking session's process ID, application name, state, transaction start time, and query. A long-running transaction or a control-plane replica still running the previous release is the usual cause. End that session, or let it finish, and the migration continues.

PostgreSQL Recovery After an Interrupted Stop

./run.sh stop and ./run.sh down give each container up to 2 minutes to shut down, because PostgreSQL's shutdown checkpoint can take longer than the 10-second Docker Compose default after heavy writes. A stop made with the previous package's run.sh still uses 10 seconds.

If PostgreSQL was cut short, it replays its log on the next start before it accepts connections, which can take minutes on a database with many recent writes. Nothing is lost. run.sh waits up to 30 minutes for the recovery and prints PostgreSQL is recovering from an unclean shutdown while it runs. If the recovery outlasts that wait, run.sh names it as the cause of the failed start: wait until ./run.sh logs postgres shows database system is ready to accept connections, then run ./run.sh again.

What the Control Plane Checks During the Window

While gateways are still behind, the control plane checks each projected configuration you save against recently reporting, registered gateways whose released compatibility contract it includes.

If one of those gateways cannot load a configuration, the save fails with HTTP 422 and the error code DP_INCOMPATIBLE. The response names the gateway release, affected gateways, field path, and schema reason. Nothing is written, so the resource keeps its previous configuration. This check is triggered by the specific setting rather than the version difference itself. Upgrade the affected gateways, or leave the setting out until they are upgraded.

Right after a gateway upgrade, the previous registration of a restarted gateway can still be counted for up to five minutes. A save refused in that window naming a version you have already upgraded away from is usually that stale registration; wait and retry.

Gateways below the floor are not checked at all. There is no supported contract to check them against, so the save succeeds and the response carries a below_floor warning giving how many gateways are affected, the versions they report, and the floor. Treat it as what it is: those gateways are out of support and the control plane cannot tell you whether they will load what you just saved. Upgrade them to 0.12.0 or later.

The control plane also skips a gateway whose reported version does not map to an included compatibility contract. This includes custom or local builds and gateway releases newer than the contracts known to the control plane. Verify these gateways directly instead of treating the absence of a save-time warning as proof of compatibility.

The Upgrade Floor Gate

An upgrade from below the floor is refused rather than attempted, because a half-migrated database is much harder to recover from than a second upgrade step.

cp-api refuses to start when the database was last run by a version below the floor. The error names the version recorded in the database, the oldest version this release can upgrade from, and the override. Nothing is migrated before the check runs, so a refused start leaves the database exactly as the previous release left it. The fix is to upgrade to a supported version first.

The check reads a version record the control plane writes from this release onward, so it can only judge a database that a release carrying that record has already run. A database last run by an earlier release carries no record, cannot be judged, and is therefore not refused — on that one upgrade the package installer's .env check is the only thing that catches it, and a Helm upgrade is not gated at all. Check the version you are coming from yourself.

The offline and online packages refuse the same upgrade in run.sh, before any container starts, and tell you the two steps: get a package between 0.12.0 and the version you are installing, extract that over the installation directory and run it, let it finish its migrations, then extract your target package and run it again.

To proceed anyway, set AISIX_ALLOW_UNSUPPORTED_UPGRADE=1. The value must be exactly 1. Create a complete recovery set first — this runs migration paths that have not been exercised from that far back, and restoring the set is your only way out.

For a package installation, set it on the command:

AISIX_ALLOW_UNSUPPORTED_UPGRADE=1 ./run.sh

For Helm, set api.allowUnsupportedUpgrade:

values.yaml

api:
  allowUnsupportedUpgrade: true

Add it to the values file you upgrade with, so it is actually applied:

helm upgrade aisix-cp api7/aisix-cp \
  --namespace "$CP_NAMESPACE" \
  --version 1.6.0 \
  -f your-cp-values.yaml

The environment variable is read by cp-api and by the package's run.sh, so one name covers both package paths, and the chart value sets the same cp-api setting. Remove the override again once the upgrade has completed, so it cannot silently permit a later unsupported upgrade.

Roll Back

Roll back in the reverse order: gateways first, then the control plane.

A rollback of the control plane restores the complete pre-upgrade recovery set, not just the images. The new version has already migrated the schema, so starting the previous version against it runs old code against a database it does not know. Restore the database, matching secrets and master keys, deployment configuration, and previous software version together.

For a package installation, run.sh refuses to start when the package in the directory is older than the version recorded in .env, because that is nearly always an accident — an old tarball re-downloaded, or a latest URL fetched onto a newer install. A deliberate rollback, after restoring the complete recovery set, acknowledges it:

AISIX_ALLOW_DOWNGRADE=1 ./run.sh

run.sh prints the deployed version when it finishes. Check that line against the version you meant to roll back to.

For a Helm installation, helm rollback returns the workloads to the previous revision; restore the complete recovery set first.

AISIX Enterprise Situations That Are Not Supported

• Upgrading from a release below the floor in one step. Upgrade to a release between 0.12.0 and your target first, let it finish its migrations, then upgrade again.
• Adding a gateway older than the environment's existing configuration already requires. The mixed-version window covers gateways being upgraded forward. Introducing a gateway on an older release into an environment whose saved configuration already depends on a newer one is not supported — that gateway would not load the configuration it is given.
• Running a gateway below 0.12.0 against a current control plane. Such a gateway is outside the supported range; the control plane cannot answer for what it will load.
• Upgrading AISIX Enterprise gateways before the control plane. Upgrade the control plane before its connected gateways.