Skip to main content
API7.ai Logo
Version: Dev

Upgrade AISIX Enterprise on OpenShift or OKD

AISIX Enterprise upgrades in two stages: the customer-operated control plane first, followed by every connected AISIX gateway. Existing gateways continue serving traffic while the new control plane is verified, but the product upgrade is not complete until the gateway fleet also runs the target release.

This procedure preserves the OpenShift or OKD PostgreSQL security-context overrides, Route host names, and project-assigned identities throughout the upgrade. It assumes the control plane and gateways were installed with Helm. For package-based control planes, follow Upgrade a Package Installation for the control-plane stage, then continue here at Upgrade the Gateway Fleet.

Prerequisites​

Before changing either release:

  • Install Helm 3.15 or a later Helm 3 release, cURL, and jq.
  • Configure oc for every cluster and project that contains an AISIX component.
  • Read the release notes for every release after the installed version and apply required upgrade notes in order.
  • Create and test a complete control-plane recovery set.
  • Confirm that the gateway load balancer removes pods whose /readyz check fails.
  • Confirm that shared Redis is configured for multi-replica gateways when limits or cache entries must remain consistent across replicas.

Complete Version-Specific Preparation​

After creating and testing the recovery set, complete every applicable version-specific action before running Helm:

  1. Read the release notes from oldest to newest, beginning with the release after the installed version.
  2. When upgrading from a release earlier than 1.0, complete Review Existing Guardrails When Upgrading to 1.0.0 before the new control plane starts.
  3. When upgrading from 1.5 or earlier, review Sign-In, Access, and Dashboard Changes. Complete the pre-start actions, identify every organization and its users, and decide which organization will remain. Do not consolidate organizations until the target control plane starts and exposes the consolidation workflow.

Complete the recovery set before any destructive change. Preserve the earlier release's dashboard authentication secret while rollback remains possible, even though the target control plane no longer reads it.

The examples use the default releases and projects from Deploy AISIX Enterprise on OpenShift or OKD. Substitute the values used by your deployment:

export CP_NAMESPACE="aisix"
export CP_RELEASE="aisix-cp"
export GATEWAY_NAMESPACE="aisix-dp"
export GATEWAY_RELEASE="aisix-dp"
export TARGET_VERSION="<target-version>"
export CP_UPGRADE_TIMEOUT="35m"
export GATEWAY_UPGRADE_TIMEOUT="60m"

The 35-minute control-plane timeout exceeds the default startup-probe budget. The 60-minute gateway timeout is a conservative starting allowance for the default two replicas, 1,230-second termination ceiling per pod, and five-minute startup-probe budget. Recalculate it when changing replicas, update strategy, termination grace, probes, or expected image-pull and scheduling time. Confirm that the cluster can schedule the surge pod; a longer timeout does not resolve quota or capacity rejection.

Trust the Route Certificate​

The default applications domain normally uses a wildcard certificate signed by the cluster's internal ingress CA. When that CA is not already trusted on the workstation, export it for the control-plane and gateway curl checks:

oc get configmap default-ingress-cert -n openshift-config-managed \
  -o jsonpath='{.data.ca-bundle\.crt}' > ingress-ca.crt
export CURL_CA_BUNDLE="$PWD/ingress-ca.crt"

Skip this export when the Routes or default Ingress Controller use certificates that the workstation already trusts.

Capture the Current Deployment​

Record the releases, pods, and Routes before changing them:

helm list -n "$CP_NAMESPACE"
helm list -n "$GATEWAY_NAMESPACE"
oc get pods,routes -n "$CP_NAMESPACE"
oc get pods,routes -n "$GATEWAY_NAMESPACE"

Save only user-supplied Helm values. The files can contain credentials, so create them with restrictive permissions:

umask 077
touch cp-values.yaml gateway-values.yaml
chmod 600 cp-values.yaml gateway-values.yaml

helm get values "$CP_RELEASE" -n "$CP_NAMESPACE" -o yaml > cp-values.yaml
helm get values "$GATEWAY_RELEASE" -n "$GATEWAY_NAMESPACE" -o yaml > gateway-values.yaml

Do not use helm get values --all or --reuse-values. Carrying computed source-chart defaults forward can prevent new defaults from taking effect.

If the control plane uses bundled PostgreSQL, confirm that cp-values.yaml preserves both SCC overrides:

cp-values.yaml
postgresql:
  primary:
    podSecurityContext:
      enabled: false
    containerSecurityContext:
      enabled: false

Also preserve api.publicBaseURL and api.dpmgrBaseURL so the public links, Routes, and data-plane manager certificate continue using the existing host names. Remove fixed runAsUser, runAsGroup, or fsGroup values that were not allocated to the project.

Refresh the chart repository and confirm that both target charts are available:

helm repo add api7 https://charts.api7.ai
helm repo update
helm show chart api7/aisix-cp --version "$TARGET_VERSION"
helm show chart api7/aisix --version "$TARGET_VERSION"

Upgrade the Control Plane​

Resolve the control-plane API Deployment name:

export CP_API_DEPLOYMENT="$(oc get deployment -n "$CP_NAMESPACE" \
  -l "app.kubernetes.io/instance=$CP_RELEASE,app.kubernetes.io/component=api" \
  -o jsonpath='{.items[0].metadata.name}')"

Do not continue if the value is empty. Render the upgrade against the live cluster before changing the release:

helm upgrade "$CP_RELEASE" api7/aisix-cp \
  --namespace "$CP_NAMESPACE" \
  --version "$TARGET_VERSION" \
  --values cp-values.yaml \
  --dry-run=server \
  --hide-secret

The server dry run checks rendering, cluster lookups, and OpenAPI validation. It does not submit objects to SCC, quota, or other admission checks.

Inspect the target images before continuing:

helm template "$CP_RELEASE" api7/aisix-cp \
  --namespace "$CP_NAMESPACE" \
  --version "$TARGET_VERSION" \
  --values cp-values.yaml \
  | grep -E '^[[:space:]]+image:'

The control-plane API and data-plane manager images should use the target version or an intentional target-compatible custom tag. PostgreSQL is independently versioned; keep the image rendered by the target chart unless you have a separately tested database-image plan.

Upgrade the control plane without --atomic. The new API migrates the database during startup, while an automatic Helm rollback would change only Kubernetes objects and would not restore the complete recovery set:

helm upgrade "$CP_RELEASE" api7/aisix-cp \
  --namespace "$CP_NAMESPACE" \
  --version "$TARGET_VERSION" \
  --values cp-values.yaml \
  --wait \
  --timeout "$CP_UPGRADE_TIMEOUT"

Watch the API rollout and capture diagnostics before retrying a failed migration:

oc rollout status -n "$CP_NAMESPACE" \
  "deployment/$CP_API_DEPLOYMENT" \
  --timeout="$CP_UPGRADE_TIMEOUT"

oc get pods -n "$CP_NAMESPACE"

Verify the Control Plane​

Confirm that the control-plane Routes still target the same API and data-plane manager Services:

oc get routes -n "$CP_NAMESPACE"

Inspect the admitted SCC and runtime UID for every control-plane pod:

oc get pods -n "$CP_NAMESPACE" \
  -o custom-columns='NAME:.metadata.name,SCC:.metadata.annotations.openshift\.io/scc,READY:.status.containerStatuses[*].ready'

for pod in $(oc get pods -n "$CP_NAMESPACE" -o name); do
  printf '%s UID=' "$pod"
  oc exec -n "$CP_NAMESPACE" "$pod" -- id -u
done

The AISIX and bundled PostgreSQL pods should report restricted-v2 and ready containers. Each running process should use a UID from the project range.

Set AISIX_CP to the public API base URL, including /api, and verify the reported version:

export AISIX_CP="https://<control-plane-route-host>/api"
curl -fsS "$AISIX_CP/config/public" | jq '{cp_version, cp_commit}'

If the source deployment had more than one organization, sign in and complete Consolidate Several Organizations into One before normal dashboard use. This operation permanently deletes the organizations you do not retain, so confirm the selected organization and recovery set before submitting it.

Confirm that the retained organization, environments, models, provider keys, and caller API keys remain present. Existing gateways should remain healthy while they still run the previous release.

Do not begin the gateway rollout until the control plane passes these checks.

Upgrade the Gateway Fleet​

Resolve the gateway Deployment name:

export GATEWAY_DEPLOYMENT="$(oc get deployment -n "$GATEWAY_NAMESPACE" \
  -l "app.kubernetes.io/instance=$GATEWAY_RELEASE,app.kubernetes.io/component=gateway" \
  -o jsonpath='{.items[0].metadata.name}')"

Do not continue if the Deployment name is empty. Check whether gateway-values.yaml pins an older image tag and remove or update that pin before rendering the target. Preserve an intentional custom image repository with a target-compatible tag.

Render the target Deployment and capture the exact gateway image Helm will apply:

export TARGET_GATEWAY_IMAGE="$(
  helm template "$GATEWAY_RELEASE" api7/aisix \
    --namespace "$GATEWAY_NAMESPACE" \
    --version "$TARGET_VERSION" \
    --values gateway-values.yaml \
    --show-only templates/deployment.yaml \
    | awk '
      $1 == "-" && $2 == "name:" && $3 == "aisix" { in_gateway = 1; next }
      in_gateway && $1 == "image:" { gsub(/"/, "", $2); print $2; exit }
    '
)"

printf 'Target gateway image: %s\n' "$TARGET_GATEWAY_IMAGE"

Stop if the target image is empty or does not identify the intended release or target-compatible custom build.

Run a server-side dry run:

helm upgrade "$GATEWAY_RELEASE" api7/aisix \
  --namespace "$GATEWAY_NAMESPACE" \
  --version "$TARGET_VERSION" \
  --values gateway-values.yaml \
  --dry-run=server \
  --hide-secret

Before running Helm, open a second terminal and watch the gateway pods. Capture the description and logs of a failed target pod while Helm is still waiting; an atomic rollback can remove it before the command returns:

oc get pods -n "$GATEWAY_NAMESPACE" \
  -l "app.kubernetes.io/instance=$GATEWAY_RELEASE,app.kubernetes.io/component=gateway" \
  --watch

Run the upgrade in the original terminal:

helm upgrade "$GATEWAY_RELEASE" api7/aisix \
  --namespace "$GATEWAY_NAMESPACE" \
  --version "$TARGET_VERSION" \
  --values gateway-values.yaml \
  --atomic \
  --timeout "$GATEWAY_UPGRADE_TIMEOUT"

If Helm succeeds, confirm that the target rollout completes:

oc rollout status -n "$GATEWAY_NAMESPACE" \
  "deployment/$GATEWAY_DEPLOYMENT" \
  --timeout="$GATEWAY_UPGRADE_TIMEOUT"

--atomic blocks until the target release becomes ready or Helm finishes rolling it back. If it returns a nonzero status, a later successful rollout status may describe the restored revision rather than the target. Use helm history "$GATEWAY_RELEASE" -n "$GATEWAY_NAMESPACE" and the diagnostics captured while Helm was waiting.

Keep readinessProbe.enabled set to true, and do not use the Recreate strategy when the rollout must avoid traffic interruption. A single-replica deployment can still have an interruption while its only pod is replaced.

Repeat this process for every gateway release and cluster connected to the AISIX Enterprise control plane.

Verify the Complete Product Upgrade​

For every gateway, confirm that its Route still targets the same Service and that a pod retains the expected SCC and project-assigned UID:

oc get routes -n "$GATEWAY_NAMESPACE"

export GATEWAY_POD="$(oc get pod -n "$GATEWAY_NAMESPACE" \
  -l "app.kubernetes.io/instance=$GATEWAY_RELEASE,app.kubernetes.io/component=gateway" \
  -o jsonpath='{.items[0].metadata.name}')"

oc get pod -n "$GATEWAY_NAMESPACE" "$GATEWAY_POD" \
  -o jsonpath='{.metadata.annotations.openshift\.io/scc}{"\n"}'
oc exec -n "$GATEWAY_NAMESPACE" "$GATEWAY_POD" -- id -u

The first line should report restricted-v2, and the second should show the UID used by the running process. Confirm that the UID is in the gateway project's allocated range.

Confirm in the dashboard that every gateway reports the target version, a healthy status, and a current heartbeat. Send representative authenticated inference traffic through every gateway group and check access logs, metrics, upstream behavior, and any configuration features required by the deployment.

Confirm that the data-plane manager passthrough Route still uses the same host name stored in each gateway's control-plane address. A mismatch breaks the mutual TLS connection even when the Route continues targeting the correct Service.

Keep the recovery set and exported values until the control plane and every gateway have operated normally through the observation period defined by your change policy.

Roll Back​

Roll back in reverse order: gateways first, then the control plane.

For a gateway, identify the previous Helm revision and restore it:

helm history "$GATEWAY_RELEASE" -n "$GATEWAY_NAMESPACE"

helm rollback "$GATEWAY_RELEASE" <previous-revision> \
  --namespace "$GATEWAY_NAMESPACE" \
  --wait \
  --timeout "$GATEWAY_UPGRADE_TIMEOUT"

A control-plane rollback must restore the complete pre-upgrade recovery set, not only the previous Kubernetes objects. Restore the database, matching secrets and master keys, deployment configuration, and previous software version together. See Roll Back for the full recovery requirements.

OKD Web Console Upgrade Error​

This issue has been validated only in the OKD web console. A control-plane release originally installed with the Helm CLI can fail to upgrade there with the following error:

found in Chart.yaml, but missing in charts/ directory: postgresql

Continue with the Helm CLI procedure on this page. The published chart already includes its PostgreSQL dependency, so do not download the dependency separately or rebuild the chart.

Troubleshoot SCC Admission​

Find the FailedCreate event, then describe the ReplicaSet or StatefulSet named by the event:

export FAILED_NAMESPACE="<project-containing-rejected-workload>"

oc get events -n "$FAILED_NAMESPACE" --sort-by=.lastTimestamp
oc describe -n "$FAILED_NAMESPACE" <controller-kind>/<controller-name>

For AISIX application pods, remove fixed runAsUser, runAsGroup, and fsGroup values that were not allocated to the project. For bundled PostgreSQL, preserve the disabled dependency-chart security contexts from the installation. Do not grant anyuid as a workaround.

Troubleshoot a Container Start Failure​

Describe the failed pod and read its current logs when the container started:

oc describe pod -n <namespace> <pod>
oc logs -n <namespace> <pod> --all-containers --tail=300

If a container restarted, identify its name in the pod description and read its previous logs:

oc logs -n <namespace> <pod> -c <restarted-container> --previous --tail=300

If CRI-O reports exec: Operation not permitted for a customized gateway image, confirm that the gateway container retains NET_BIND_SERVICE. Keep all other capabilities dropped.