Skip to main content
Version: 1.6.0

Upgrade AISIX on OKD

Helm-managed AISIX deployments on OKD can run as an open-source gateway without a control plane or as an on-premises AISIX Cloud deployment. This procedure upgrades either deployment from 1.4 or 1.5 to this version. Each gateway deployment has a separate Helm release; an on-premises AISIX Cloud deployment also has a control-plane release.

The upgrade sequence follows that management relationship. For an open-source gateway, validate its resources with the target gateway image and upgrade the gateway release directly. For an on-premises AISIX Cloud deployment, upgrade the control plane first, verify its connection to the existing gateways, and then upgrade each managed gateway release. Read Upgrade an On-Premises AISIX Cloud Deployment for mixed-version and control-plane rollback guidance.

Both paths use AISIX charts that support the random non-root user assigned by the OKD restricted-v2 Security Context Constraint (SCC), so do not grant the AISIX service accounts the anyuid SCC. An on-premises AISIX Cloud deployment that uses bundled PostgreSQL also requires the overrides in Run OKD Preflight Checks. Before beginning either upgrade path, prepare the cluster access, release details, and recovery materials described in Prerequisites.

Prerequisites​

Prepare these requirements for every deployment:

  • Configure oc to access the OKD project that contains AISIX, and install Helm 3.15 or a later Helm 3 release, cURL, and jq. To validate an open-source gateway's resources before upgrading, also install Docker.
  • Identify the Helm release and namespace for each gateway deployment.
  • If you are upgrading from 1.4, read the 1.5.0 and 1.6.0 release notes and apply every relevant upgrade instruction in order.
  • If you are upgrading from 1.5, read the 1.6.0 release notes and apply every relevant upgrade instruction. For an on-premises AISIX Cloud deployment, also review Upgrade from 1.5.x: Sign-In, Access, and Dashboard Changes and complete its applicable pre-upgrade preparation.
  • For an existing multi-replica gateway deployment, confirm that the load balancer removes pods whose /readyz check fails. If you plan to increase a single-replica deployment before the upgrade, first configure shared Redis when rate-limit counters or cache entries must remain consistent across replicas. Otherwise, explicitly accept the temporary per-replica behavior. See Prepare AISIX for Production.

For an on-premises AISIX Cloud deployment, also:

  • Identify the control-plane Helm release and namespace.
  • Create a complete recovery set containing the PostgreSQL backup, matching master keys and secret sources, deployment configuration, and software version. Verify that the database backup can be restored. Database migrations run when the new control-plane API starts.
  • Plan a maintenance window for the control plane.

Prepare the Upgrade​

Set the gateway release details used by the examples:

export GATEWAY_NAMESPACE="aisix"
export GATEWAY_RELEASE="aisix"
export GATEWAY_UPGRADE_TIMEOUT="60m"

The 60-minute gateway timeout is a conservative starting allowance for the chart defaults. These defaults include two replicas, a 1,230-second termination ceiling per pod, and a five-minute startup-probe budget. It is not the expected duration.

Kubernetes can overlap pod termination, and Helm does not require old pods to finish before it considers the updated Deployment ready. Recalculate the timeout when changing replicas, update strategy, termination grace, probe budgets, or when image pulls and scheduling are slower.

Confirm that the cluster can schedule the surge pod; a longer timeout does not resolve a quota or capacity rejection. If an atomic upgrade fails, Helm can use another full timeout period for its rollback, so reserve that time in the maintenance window.

For an on-premises AISIX Cloud deployment, also set the control-plane release details:

export CP_NAMESPACE="aisix"
export CP_RELEASE="aisix-cp"
export CP_UPGRADE_TIMEOUT="35m"

The 35-minute control-plane timeout covers the chart's default 30-minute API startup-probe budget with operational margin. Increase it if database migrations can take longer.

Refresh the API7 chart repository and confirm that each applicable 1.6.0 chart is available:

helm repo add api7 https://charts.api7.ai
helm repo update

helm show chart api7/aisix --version 1.6.0

# On-premises AISIX Cloud only
helm show chart api7/aisix-cp --version 1.6.0

Capture the Current Deployment​

Record the installed chart and application versions:

helm list -n "$GATEWAY_NAMESPACE"

oc get pods -n "$GATEWAY_NAMESPACE"
# If this deployment uses OpenShift Routes
oc get routes -n "$GATEWAY_NAMESPACE"

# On-premises AISIX Cloud only
helm list -n "$CP_NAMESPACE"
oc get pods -n "$CP_NAMESPACE"
# If this deployment uses OpenShift Routes
oc get routes -n "$CP_NAMESPACE"

Resolve the Deployment names from their Helm release labels:

export GATEWAY_DEPLOYMENT="$(oc get deployment -n "$GATEWAY_NAMESPACE" \
-l "app.kubernetes.io/instance=$GATEWAY_RELEASE,app.kubernetes.io/component=gateway" \
-o jsonpath='{.items[0].metadata.name}')"

printf 'Gateway: %s\n' "$GATEWAY_DEPLOYMENT"

For an on-premises AISIX Cloud deployment, also resolve the control-plane API Deployment:

export CP_API_DEPLOYMENT="$(oc get deployment -n "$CP_NAMESPACE" \
-l "app.kubernetes.io/instance=$CP_RELEASE,app.kubernetes.io/component=api" \
-o jsonpath='{.items[0].metadata.name}')"

printf 'Control-plane API: %s\n' "$CP_API_DEPLOYMENT"

Stop and check the release name and namespace if a Deployment name you expected is empty.

Save only the user-supplied values. Set a restrictive file-creation mask first because the exported values can contain credentials. Do not use helm get values --all; carrying source-chart defaults forward can prevent a target-chart default from taking effect.

umask 077
touch gateway-values.yaml
chmod 600 gateway-values.yaml
helm get values "$GATEWAY_RELEASE" -n "$GATEWAY_NAMESPACE" -o yaml > gateway-values.yaml

# On-premises AISIX Cloud only
touch cp-values.yaml
chmod 600 cp-values.yaml
helm get values "$CP_RELEASE" -n "$CP_NAMESPACE" -o yaml > cp-values.yaml

Treat these files as sensitive. They can contain credentials when the deployment does not use existing Secrets. Store them with the upgrade records and the rest of the recovery set, when applicable, and remove local copies after the upgrade.

Check whether the saved values explicitly pin the AISIX component images:

helm get values "$GATEWAY_RELEASE" -n "$GATEWAY_NAMESPACE" -o json \
| jq '{gateway_image_tag: (.image.tag // "")}'

# On-premises AISIX Cloud only
helm get values "$CP_RELEASE" -n "$CP_NAMESPACE" -o json \
| jq '{api_image_tag: (.api.image.tag // ""), dpm_image_tag: (.dpm.image.tag // ""), generated_gateway_image: (.api.dpImage // "")}'

An empty tag uses the target chart's application version. If any tag pins an older AISIX image, remove it from the saved values or update it to the target version. Preserve any deliberate custom image repository, and use a target-compatible tag for a custom build.

For an on-premises AISIX Cloud deployment, generated_gateway_image is the image that the dashboard places in generated gateway installation instructions. An empty value follows the control-plane release. Clear or update an older pin so gateways added after the upgrade do not start on the previous image; preserve it only for a deliberate target-compatible custom build.

Run OKD Preflight Checks​

AISIX application containers do not pin runAsUser, runAsGroup, or fsGroup, so OKD can assign an allowed UID. Keep runAsNonRoot enabled and remove fixed IDs unless the cluster administrator allocated them to the project.

If the control plane uses the bundled PostgreSQL instance, add the following values to cp-values.yaml. The PostgreSQL dependency otherwise pins UID and group values that restricted-v2 can reject:

cp-values.yaml
postgresql:
primary:
podSecurityContext:
enabled: false
containerSecurityContext:
enabled: false

For production, prefer an external highly available PostgreSQL database. See External Database.

Have Helm render each applicable upgrade against the live cluster. --dry-run=server permits cluster lookups and OpenAPI validation, while --hide-secret prevents Helm from printing the rendered Secrets to the terminal or CI log:

helm upgrade "$GATEWAY_RELEASE" api7/aisix \
--namespace "$GATEWAY_NAMESPACE" \
--version 1.6.0 \
--values gateway-values.yaml \
--dry-run=server \
--hide-secret

# On-premises AISIX Cloud only
helm upgrade "$CP_RELEASE" api7/aisix-cp \
--namespace "$CP_NAMESPACE" \
--version 1.6.0 \
--values cp-values.yaml \
--dry-run=server \
--hide-secret

These commands do not submit the rendered objects to Kubernetes admission. Treat them as rendering and OpenAPI checks, not as validation of SCCs, quotas, or admission policies. The SCC assigned to a new pod is confirmed only when OKD creates that pod. Resolve every reported error before continuing.

Render the target gateway Deployment and capture the exact gateway image that Helm will apply. This preserves the chart's registry and any deliberate repository or tag override:

TARGET_GATEWAY_IMAGE="$(
helm template "$GATEWAY_RELEASE" api7/aisix \
--namespace "$GATEWAY_NAMESPACE" \
--version 1.6.0 \
--values gateway-values.yaml \
--show-only templates/deployment.yaml \
| awk '
$1 == "-" && $2 == "name:" && $3 == "aisix" { in_gateway = 1; next }
in_gateway && $1 == "image:" { gsub(/"/, "", $2); print $2; exit }
'
)"
export TARGET_GATEWAY_IMAGE

if [ -z "$TARGET_GATEWAY_IMAGE" ]; then
echo "Target gateway image was not rendered" >&2
else
printf 'Target gateway image: %s\n' "$TARGET_GATEWAY_IMAGE"
fi

Do not continue if the target gateway image was not rendered.

For an on-premises AISIX Cloud deployment, also inspect every image in the target control-plane render:

helm template "$CP_RELEASE" api7/aisix-cp \
--namespace "$CP_NAMESPACE" \
--version 1.6.0 \
--values cp-values.yaml \
| grep -E '^[[:space:]]+image:'

The aisix-cp-api and aisix-cp-dpm images should use version 1.6.0 or a deliberate target-compatible custom tag. PostgreSQL is an independently versioned chart dependency. Keep the PostgreSQL image tag rendered by the 1.6.0 chart unless a separately tested database-image plan requires another tag.

Continue with the upgrade path for your deployment:

Upgrade an On-Premises AISIX Cloud Deployment​

Upgrade the Control Plane​

Upgrade the control plane without --atomic, and continue to the managed gateways only after it passes verification. The new API migrates the database during startup, while an automatic Helm rollback changes only the Kubernetes objects and does not restore the database backup.

helm upgrade "$CP_RELEASE" api7/aisix-cp \
--namespace "$CP_NAMESPACE" \
--version 1.6.0 \
--values cp-values.yaml \
--wait \
--timeout "$CP_UPGRADE_TIMEOUT"

The data-plane manager can briefly log missing-table errors while the control-plane API applies the database migrations. It retries automatically. Investigate the errors if they continue after the control-plane API becomes ready.

Watch the rollout. If the API pod does not become ready, describe it first. Read its logs if the container started:

oc rollout status -n "$CP_NAMESPACE" \
"deployment/$CP_API_DEPLOYMENT" \
--timeout="$CP_UPGRADE_TIMEOUT"

oc get pods -n "$CP_NAMESPACE" \
-l "app.kubernetes.io/instance=$CP_RELEASE,app.kubernetes.io/component=api"

oc describe pod -n "$CP_NAMESPACE" <non-ready-cp-api-pod>

oc logs -n "$CP_NAMESPACE" \
<non-ready-cp-api-pod> \
--all-containers \
--tail=300

Verify the control-plane version through its public endpoint. Set AISIX_CP to the existing endpoint followed by /api, without a trailing slash. For a deployment that uses an OpenShift Route, use the Route origin:

export AISIX_CP="YOUR_AISIX_CLOUD_ADMIN_API_URL"

curl -fsS "$AISIX_CP/config/public" | jq '{cp_version, cp_commit}'

Sign in to the dashboard and confirm that:

  • organizations, environments, models, provider keys, and caller API keys are present;
  • the existing gateways remain healthy in the environment's Data planes view;
  • any control-plane and data-plane manager Routes still reach the same Services.

Upgrade the Managed Gateways​

Upgrade each managed gateway release after the AISIX Cloud control plane passes verification.

Before running Helm, open a second terminal and watch the gateway pods. Leave the watch running until Helm returns:

oc get pods -n "$GATEWAY_NAMESPACE" \
-l "app.kubernetes.io/instance=$GATEWAY_RELEASE,app.kubernetes.io/component=gateway" \
--watch

If a new pod stalls or restarts, press Ctrl+C to stop the watch, or open a third terminal. Capture the pod's description and logs while Helm is still waiting in the original terminal. An atomic rollback can remove the failed target pods before the Helm command returns; see A Gateway Does Not Become Ready.

Run the upgrade in the original terminal:

helm upgrade "$GATEWAY_RELEASE" api7/aisix \
--namespace "$GATEWAY_NAMESPACE" \
--version 1.6.0 \
--values gateway-values.yaml \
--atomic \
--timeout "$GATEWAY_UPGRADE_TIMEOUT"

--atomic blocks until version 1.6.0 becomes ready or Helm finishes rolling it back after a failure. If the command exits with a nonzero status, do not treat a later successful oc rollout status as proof that version 1.6.0 succeeded; it may be reporting the restored revision. Retain the diagnostics captured while Helm was running, then inspect the release history and verify the restored gateway release:

helm history "$GATEWAY_RELEASE" -n "$GATEWAY_NAMESPACE"

With the chart's default rolling-update strategy and readiness probe, readiness remains false until a new gateway has loaded usable configuration. OKD therefore keeps the pod out of the Service endpoints until it can serve traffic. If the upgrade must avoid traffic interruption, confirm that gateway-values.yaml does not select a Recreate update strategy and keeps readinessProbe.enabled set to true. A single-replica deployment can still have an interruption while its only pod is replaced.

If the Helm command succeeds, confirm that the 1.6.0 rollout is complete and every gateway pod is ready:

oc rollout status -n "$GATEWAY_NAMESPACE" \
"deployment/$GATEWAY_DEPLOYMENT" \
--timeout="$GATEWAY_UPGRADE_TIMEOUT"

oc get pods -n "$GATEWAY_NAMESPACE"

Verify the Deployment​

Confirm that every gateway reports version 1.6.0 and a healthy status in the Data planes view. Set AISIX_PROXY to the existing gateway endpoint without a trailing slash. For a deployment that uses an OpenShift Route, use the Route origin. Then list the models available to the caller:

export AISIX_PROXY="YOUR_AISIX_GATEWAY_URL"
export AISIX_API_KEY="<caller-api-key>"

curl -fsS "$AISIX_PROXY/v1/models" \
-H "Authorization: Bearer $AISIX_API_KEY"

Send a representative inference request and check the access logs, metrics, and usage reporting that the deployment depends on. Keep the saved values until the observation period defined by your change policy has passed. For an on-premises AISIX Cloud deployment, keep the complete recovery set for the same period.

Upgrade an Open-Source AISIX Gateway​

If you run an open-source AISIX gateway without a control plane, complete Prerequisites, Prepare the Upgrade, Capture the Current Deployment, and Run OKD Preflight Checks, then follow this path.

Validate Gateway Resources​

Validate the gateway's complete resources with the target gateway image before upgrading. First, identify which resource source the release uses without printing the inline resource document:

helm get values "$GATEWAY_RELEASE" -n "$GATEWAY_NAMESPACE" -o json \
| jq '{inline_resources: (((.standalone.resources // {}) | length) > 0), existing_secret: (.standalone.existingSecret // ""), existing_config_map: (.standalone.existingConfigMap // "")}'

export TARGET_RESOURCES_FILE="$(mktemp "$PWD/aisix-resources.XXXXXX")"
chmod 600 "$TARGET_RESOURCES_FILE"

Use the command for that source. For inline resources, render the target chart's managed Secret and extract the complete document:

helm template "$GATEWAY_RELEASE" api7/aisix \
--namespace "$GATEWAY_NAMESPACE" \
--version 1.6.0 \
--values gateway-values.yaml \
--show-only templates/secret.yaml \
| awk '
/^ resources\.yaml: \|$/ { in_resources = 1; next }
in_resources && /^---$/ { exit }
in_resources { sub(/^ /, ""); print }
' > "$TARGET_RESOURCES_FILE"

For an existing Secret or ConfigMap, replace the placeholder with the name reported by the preceding jq command:

# Choose one command.
oc extract -n "$GATEWAY_NAMESPACE" \
'secret/<existing-secret-name>' \
--keys=resources.yaml --to=- > "$TARGET_RESOURCES_FILE"

oc extract -n "$GATEWAY_NAMESPACE" \
'configmap/<existing-config-map-name>' \
--keys=resources.yaml --to=- > "$TARGET_RESOURCES_FILE"

Validate the extracted document with TARGET_GATEWAY_IMAGE, which was captured from the target Deployment render. Export every environment variable referenced by the resource file, and add one -e option for each variable. The example uses OPENAI_API_KEY and CALLER_API_KEY; replace them with the variable names in your resource file.

The command streams the document into the short-lived container instead of bind-mounting the host file. The gateway image can therefore read the document without weakening its host permissions or changing its SELinux label.

docker run --rm -i \
-e OPENAI_API_KEY \
-e CALLER_API_KEY \
--entrypoint /bin/sh "$TARGET_GATEWAY_IMAGE" \
-c 'umask 077; cat > /tmp/resources.yaml && exec /usr/local/bin/aisix validate --resources /tmp/resources.yaml' \
< "$TARGET_RESOURCES_FILE" && {
rm -f "$TARGET_RESOURCES_FILE" &&
unset TARGET_RESOURCES_FILE
}

Resolve every validation error before continuing. A failed validation retains the extracted file for inspection and returns a nonzero status. If the resource file does not reference environment variables, omit the -e options.

Upgrade the Gateway​

Upgrade the open-source gateway after completing the shared preparation and resource validation.

Before running Helm, open a second terminal and watch the gateway pods. Leave the watch running until Helm returns:

oc get pods -n "$GATEWAY_NAMESPACE" \
-l "app.kubernetes.io/instance=$GATEWAY_RELEASE,app.kubernetes.io/component=gateway" \
--watch

If a new pod stalls or restarts, press Ctrl+C to stop the watch, or open a third terminal. Capture the pod's description and logs while Helm is still waiting in the original terminal. An atomic rollback can remove the failed target pods before the Helm command returns; see A Gateway Does Not Become Ready.

Run the upgrade in the original terminal:

helm upgrade "$GATEWAY_RELEASE" api7/aisix \
--namespace "$GATEWAY_NAMESPACE" \
--version 1.6.0 \
--values gateway-values.yaml \
--atomic \
--timeout "$GATEWAY_UPGRADE_TIMEOUT"

--atomic blocks until version 1.6.0 becomes ready or Helm finishes rolling it back after a failure. If the command exits with a nonzero status, do not treat a later successful oc rollout status as proof that version 1.6.0 succeeded; it may be reporting the restored revision. Retain the diagnostics captured while Helm was running, then inspect the release history and verify the restored gateway release:

helm history "$GATEWAY_RELEASE" -n "$GATEWAY_NAMESPACE"

With the chart's default rolling-update strategy and readiness probe, readiness remains false until a new gateway has loaded usable configuration. OKD therefore keeps the pod out of the Service endpoints until it can serve traffic. If the upgrade must avoid traffic interruption, confirm that gateway-values.yaml does not select a Recreate update strategy and keeps readinessProbe.enabled set to true. A single-replica deployment can still have an interruption while its only pod is replaced.

If the Helm command succeeds, confirm that the 1.6.0 rollout is complete and every gateway pod is ready:

oc rollout status -n "$GATEWAY_NAMESPACE" \
"deployment/$GATEWAY_DEPLOYMENT" \
--timeout="$GATEWAY_UPGRADE_TIMEOUT"

oc get pods -n "$GATEWAY_NAMESPACE"

Verify the Gateway​

Set AISIX_PROXY to the existing gateway endpoint without a trailing slash. For a deployment that uses an OpenShift Route, use the Route origin. Then list the models available to the caller:

export AISIX_PROXY="YOUR_AISIX_GATEWAY_URL"
export AISIX_API_KEY="<caller-api-key>"

curl -fsS "$AISIX_PROXY/v1/models" \
-H "Authorization: Bearer $AISIX_API_KEY"

Send a representative inference request and check the access logs, metrics, and usage reporting that the deployment depends on. Keep the saved values until the observation period defined by your change policy has passed.

Troubleshoot the Upgrade​

If an upgraded workload does not become ready or reconnect, start with the matching symptom below.

The Web Console Reports a Missing PostgreSQL Dependency​

A control-plane release originally installed with the Helm CLI can fail to upgrade in the OKD web console. If the console reports the following error, continue with the Helm CLI procedure in Upgrade the Control Plane:

found in Chart.yaml, but missing in charts/ directory: postgresql

The published control-plane chart already includes its PostgreSQL dependency, so do not download the dependency separately or rebuild the chart.

A Pod Is Rejected by an SCC​

Set the namespace to the project that contains the rejected workload. Find the FailedCreate event, then describe the controller kind and name reported by that event. A gateway or control-plane API pod is owned by a ReplicaSet, while bundled PostgreSQL is owned by a StatefulSet:

export FAILED_NAMESPACE="<project-containing-rejected-workload>"

oc get events -n "$FAILED_NAMESPACE" --sort-by=.lastTimestamp
oc describe -n "$FAILED_NAMESPACE" <controller-kind>/<controller-name>

Remove fixed runAsUser, runAsGroup, and fsGroup values from AISIX application pods. For bundled PostgreSQL, disable the pod and container security contexts as shown in Run OKD Preflight Checks.

The Control Plane Does Not Become Ready​

Find and describe the control-plane API pod that is not ready. A database migration failure, unreachable database, or missing Secret prevents readiness. Read the pod's logs if the container started:

oc get pods -n "$CP_NAMESPACE" \
-l "app.kubernetes.io/instance=$CP_RELEASE,app.kubernetes.io/component=api"

oc describe pod -n "$CP_NAMESPACE" <non-ready-cp-api-pod>

oc logs -n "$CP_NAMESPACE" \
<non-ready-cp-api-pod> \
--all-containers \
--tail=300

Do not repeatedly rerun the upgrade. Restore the complete pre-upgrade recovery set according to the approved rollback procedure if the migration cannot complete.

A Gateway Does Not Become Ready​

After stopping the watch, run these diagnostics while Helm is still waiting, using the new target pod reported by the watch. Alternatively, run them in a third terminal. After a failed atomic upgrade returns, the failed pod may already be gone; use the captured output and events, then confirm the restored revision with the release-history command above.

oc get events -n "$GATEWAY_NAMESPACE" --sort-by=.lastTimestamp

oc get pods -n "$GATEWAY_NAMESPACE" \
-l "app.kubernetes.io/instance=$GATEWAY_RELEASE,app.kubernetes.io/component=gateway"

oc describe pod -n "$GATEWAY_NAMESPACE" <non-ready-gateway-pod>

oc logs -n "$GATEWAY_NAMESPACE" \
<non-ready-gateway-pod> \
--all-containers \
--tail=300

If a container restarted, use oc describe pod to identify its name, then read that container's previous logs without --all-containers:

oc describe pod -n "$GATEWAY_NAMESPACE" <non-ready-gateway-pod>

oc logs -n "$GATEWAY_NAMESPACE" \
<non-ready-gateway-pod> \
-c <restarted-container-name> \
--previous \
--tail=300

For an on-premises AISIX Cloud deployment, verify that the pod can reach the existing data-plane manager endpoint over mutual TLS. If the endpoint uses an OpenShift Route, confirm that the Route uses TLS passthrough. In every case, confirm that the endpoint hostname still matches the address in the gateway configuration.

For an open-source AISIX gateway, inspect the log for resource-validation errors. Confirm that the saved standalone values still identify the complete resources file and that the gateway can reach every configured upstream endpoint.

Gateway Verification Fails After the Rollout​

If the rollout becomes ready but the model-list, inference, logging, metrics, or usage checks fail, inspect the gateway logs and the Helm release history. Roll the gateway back to its pre-upgrade revision when the failure cannot be corrected safely on this version:

helm history "$GATEWAY_RELEASE" -n "$GATEWAY_NAMESPACE"

helm rollback "$GATEWAY_RELEASE" <pre-upgrade-revision> \
--namespace "$GATEWAY_NAMESPACE" \
--wait \
--timeout "$GATEWAY_UPGRADE_TIMEOUT"

Wait for the rollback to finish, then repeat Verify the Deployment or Verify the Gateway, as applicable. For an on-premises AISIX Cloud deployment, leave the verified control plane on this version. The supported mixed-version window allows the previous gateway release to remain connected while you investigate.