Version: Dev

Upgrade AISIX OSS on OpenShift or OKD

AISIX OSS runs without a control plane, so its declarative resources must remain valid for the target gateway image. This procedure captures the existing release, validates its resources with that image, performs the Helm rollout, and verifies the OpenShift or OKD Route and security context afterward.

Prerequisites

Before starting the upgrade:

Configure oc to access the cluster and project that contain AISIX.
Install Helm 3.15 or a later Helm 3 release, cURL, jq, and Docker.
Read the release notes for every release after the installed version. Apply any required configuration or observability changes in order.
Confirm that the load balancer removes pods whose /readyz check fails.
For a multi-replica deployment, confirm that shared Redis is configured when rate-limit counters or cache entries must remain consistent across replicas. If you plan to increase a single-replica deployment before the upgrade, configure shared Redis first or explicitly accept the temporary per-replica behavior. See Prepare for Production.

The examples use aisix-dp as the release and namespace, matching Deploy AISIX OSS on OpenShift or OKD. If your installation uses another value, run helm list -A, find the release whose CHART column is aisix-<version>, and use its release name and namespace throughout this procedure. A wrong value makes helm upgrade report that the release has no deployed releases and leaves the Deployment lookup below empty.

Set the release details used by the examples:

export GATEWAY_NAMESPACE="aisix-dp"
export GATEWAY_RELEASE="aisix-dp"
export TARGET_VERSION="<target-version>"
export GATEWAY_UPGRADE_TIMEOUT="60m"

Capture the current Route and its Service target before changing the release:

oc get routes -n "$GATEWAY_NAMESPACE"

The 60-minute timeout is a conservative starting allowance for the chart defaults. These defaults include two replicas, a 1,230-second termination ceiling per pod, and a five-minute startup-probe budget. It is not the expected duration.

Kubernetes can overlap pod termination, and Helm does not require old pods to finish before it considers the updated Deployment ready. Recalculate the timeout when changing replicas, update strategy, termination grace, probe budgets, or when image pulls and scheduling are slower.

Confirm that the cluster can schedule the surge pod; a longer timeout does not resolve a quota or capacity rejection. If an atomic upgrade fails, Helm can use another full timeout period for its rollback, so reserve that time in the maintenance window.

Refresh the API7 chart repository and confirm that the target chart is available:

helm repo add api7 https://charts.api7.ai
helm repo update
helm show chart api7/aisix --version "$TARGET_VERSION"

Trust the Route Certificate

The default applications domain normally uses a wildcard certificate signed by the cluster's internal ingress CA. When that CA is not already trusted on the workstation, export it for the post-upgrade curl checks:

oc get configmap default-ingress-cert -n openshift-config-managed \
  -o jsonpath='{.data.ca-bundle\.crt}' > ingress-ca.crt
export CURL_CA_BUNDLE="$PWD/ingress-ca.crt"

Skip this export when the Route or default Ingress Controller uses a certificate that the workstation already trusts.

Capture the Current Release

Record the installed chart and application versions:

helm list -n "$GATEWAY_NAMESPACE"
oc get pods -n "$GATEWAY_NAMESPACE"

Resolve the Deployment name from the Helm release label. This works when the chart uses a name override:

export GATEWAY_DEPLOYMENT="$(oc get deployment -n "$GATEWAY_NAMESPACE" \
  -l "app.kubernetes.io/instance=$GATEWAY_RELEASE,app.kubernetes.io/component=gateway" \
  -o jsonpath='{.items[0].metadata.name}')"

printf 'Gateway: %s\n' "$GATEWAY_DEPLOYMENT"

Stop and verify the release name and namespace if the value is empty.

Save only the user-supplied values. The file can contain credentials, so create it with restrictive permissions. Do not export all computed defaults with helm get values --all; carrying old defaults forward can prevent a new chart default from taking effect.

umask 077
touch gateway-values.yaml
chmod 600 gateway-values.yaml
helm get values "$GATEWAY_RELEASE" -n "$GATEWAY_NAMESPACE" -o yaml > gateway-values.yaml

Keep this file with the upgrade records, and remove the local copy after the observation period.

Check whether the values pin an image tag:

helm get values "$GATEWAY_RELEASE" -n "$GATEWAY_NAMESPACE" -o json \
  | jq '{gateway_image_tag: (.image.tag // "")}'

An empty tag uses the target chart's application version. If the value pins an older AISIX image, remove it from the saved values or update it to the target version. Preserve a deliberate custom repository, and use a target-compatible tag for a custom build.

Render the Target Gateway Image

Render the target Deployment and capture the exact gateway image that Helm will apply. This preserves the chart's registry and any deliberate repository or tag override:

TARGET_GATEWAY_IMAGE="$(
  helm template "$GATEWAY_RELEASE" api7/aisix \
    --namespace "$GATEWAY_NAMESPACE" \
    --version "$TARGET_VERSION" \
    --values gateway-values.yaml \
    --show-only templates/deployment.yaml \
    | awk '
      $1 == "-" && $2 == "name:" && $3 == "aisix" { in_gateway = 1; next }
      in_gateway && $1 == "image:" { gsub(/"/, "", $2); print $2; exit }
    '
)"
export TARGET_GATEWAY_IMAGE

if [ -z "$TARGET_GATEWAY_IMAGE" ]; then
  echo "Target gateway image was not rendered" >&2
else
  printf 'Target gateway image: %s\n' "$TARGET_GATEWAY_IMAGE"
fi

Do not continue if the target gateway image was not rendered.

Validate AISIX OSS Resources

Identify which resource source the release uses without printing the inline resource document:

helm get values "$GATEWAY_RELEASE" -n "$GATEWAY_NAMESPACE" -o json \
  | jq '{inline_resources: (((.standalone.resources // {}) | length) > 0), existing_secret: (.standalone.existingSecret // ""), existing_config_map: (.standalone.existingConfigMap // "")}'

export TARGET_RESOURCES_FILE="$(mktemp "$PWD/aisix-resources.XXXXXX")"
chmod 600 "$TARGET_RESOURCES_FILE"

Use the command for the reported source. For inline resources, render the target chart's managed Secret and extract the complete document:

helm template "$GATEWAY_RELEASE" api7/aisix \
  --namespace "$GATEWAY_NAMESPACE" \
  --version "$TARGET_VERSION" \
  --values gateway-values.yaml \
  --show-only templates/secret.yaml \
  | awk '
    /^  resources\.yaml: \|$/ { in_resources = 1; next }
    in_resources && /^---$/ { exit }
    in_resources { sub(/^    /, ""); print }
  ' > "$TARGET_RESOURCES_FILE"

For an existing Secret or ConfigMap, replace the placeholder with the name reported by the preceding jq command:

# Choose one command.
oc get secret -n "$GATEWAY_NAMESPACE" \
  <existing-secret-name> \
  -o jsonpath='{.data.resources\.yaml}' \
  | base64 --decode > "$TARGET_RESOURCES_FILE"

oc get configmap -n "$GATEWAY_NAMESPACE" \
  <existing-config-map-name> \
  -o jsonpath='{.data.resources\.yaml}' > "$TARGET_RESOURCES_FILE"

Validate the extracted document with TARGET_GATEWAY_IMAGE, which was captured from the target Deployment render. Export every environment variable referenced by the resource file, and add one -e option for each variable. The example uses OPENAI_API_KEY and CALLER_API_KEY; replace them with the variables in your file.

The command streams the document into the short-lived container instead of bind-mounting the host file. The gateway image can therefore read the document without weakening its host permissions or changing its SELinux label.

docker run --rm -i \
  -e OPENAI_API_KEY \
  -e CALLER_API_KEY \
  --entrypoint /bin/sh "$TARGET_GATEWAY_IMAGE" \
  -c 'umask 077; cat > /tmp/resources.yaml && exec /usr/local/bin/aisix validate --resources /tmp/resources.yaml' \
  < "$TARGET_RESOURCES_FILE" && {
    rm -f "$TARGET_RESOURCES_FILE" &&
    unset TARGET_RESOURCES_FILE
  }

Resolve every validation error before continuing. A failed validation retains the extracted file for inspection and returns a nonzero status. If the resource file does not reference environment variables, omit the -e options.

Check the Target Release

Have Helm render the upgrade against the live cluster before changing the release. --dry-run=server permits cluster lookups and OpenAPI validation, while --hide-secret prevents Helm from printing rendered Secrets to the terminal or CI log:

helm upgrade "$GATEWAY_RELEASE" api7/aisix \
  --namespace "$GATEWAY_NAMESPACE" \
  --version "$TARGET_VERSION" \
  --values gateway-values.yaml \
  --dry-run=server \
  --hide-secret

This command does not submit the rendered objects to admission. Resolve every reported error before continuing.

Upgrade the Gateway

Before running Helm, open a second terminal and watch the gateway pods. Leave the watch running until Helm returns:

oc get pods -n "$GATEWAY_NAMESPACE" \
  -l "app.kubernetes.io/instance=$GATEWAY_RELEASE,app.kubernetes.io/component=gateway" \
  --watch

If a new pod stalls or restarts, press Ctrl+C to stop the watch, or open a third terminal. Capture the pod's description and logs while Helm is still waiting in the original terminal. An atomic rollback can remove the failed target pods before the Helm command returns; see Troubleshoot a Gateway That Is Not Ready.

Run the upgrade in the original terminal:

helm upgrade "$GATEWAY_RELEASE" api7/aisix \
  --namespace "$GATEWAY_NAMESPACE" \
  --version "$TARGET_VERSION" \
  --values gateway-values.yaml \
  --atomic \
  --timeout "$GATEWAY_UPGRADE_TIMEOUT"

--atomic blocks until the target release becomes ready or Helm finishes rolling it back after a failure. If the command exits with a nonzero status, do not treat a later successful oc rollout status as proof that the target version succeeded; it may be reporting the restored revision. Retain the diagnostics captured while Helm was running, then inspect the release history and verify the restored gateway release:

helm history "$GATEWAY_RELEASE" -n "$GATEWAY_NAMESPACE"

With the default rolling-update strategy and readiness probe, a new gateway stays out of Service endpoints until it loads usable configuration. Keep readinessProbe.enabled set to true, and do not select the Recreate strategy when the upgrade must avoid traffic interruption. A single-replica deployment can still have an interruption while its only pod is replaced.

If the Helm command succeeds, confirm that the target rollout is complete and every gateway pod is ready:

oc rollout status -n "$GATEWAY_NAMESPACE" \
  "deployment/$GATEWAY_DEPLOYMENT" \
  --timeout="$GATEWAY_UPGRADE_TIMEOUT"

oc get pods -n "$GATEWAY_NAMESPACE"

Verify the Upgrade

Set AISIX_PROXY to the gateway origin without a trailing slash, then list the models available to the caller:

export AISIX_PROXY="YOUR_AISIX_GATEWAY_URL"
export AISIX_API_KEY="<caller-api-key>"

curl -fsS "$AISIX_PROXY/v1/models" \
  -H "Authorization: Bearer $AISIX_API_KEY"

Send a known-good inference request for one of the listed models. Check the access logs, metrics, and other telemetry that the deployment depends on. Keep the saved values until the gateway has operated normally through the observation period defined by your change policy.

Confirm that the Route still targets the gateway Service and that a gateway pod still uses restricted-v2 with a UID from the project range:

oc get routes -n "$GATEWAY_NAMESPACE"

export GATEWAY_POD="$(oc get pod -n "$GATEWAY_NAMESPACE" \
  -l "app.kubernetes.io/instance=$GATEWAY_RELEASE,app.kubernetes.io/component=gateway" \
  -o jsonpath='{.items[0].metadata.name}')"

oc get pod -n "$GATEWAY_NAMESPACE" "$GATEWAY_POD" \
  -o jsonpath='{.metadata.annotations.openshift\.io/scc}{"\n"}'
oc exec -n "$GATEWAY_NAMESPACE" "$GATEWAY_POD" -- id -u

The first line should report restricted-v2, and the second should show the UID used by the running process. Confirm that the UID is in the project's allocated range.

Roll Back

If Helm's atomic upgrade fails, it restores the previous release automatically. Diagnose the failure before retrying.

To roll back after a successful upgrade, identify the previous revision and restore it:

helm history "$GATEWAY_RELEASE" -n "$GATEWAY_NAMESPACE"

helm rollback "$GATEWAY_RELEASE" <previous-revision> \
  --namespace "$GATEWAY_NAMESPACE" \
  --wait \
  --timeout "$GATEWAY_UPGRADE_TIMEOUT"

Confirm that the restored pods become ready and repeat the traffic verification.

Troubleshoot a Gateway That Is Not Ready

After stopping the watch, run these diagnostics while Helm is still waiting, using the new target pod reported by the watch. Alternatively, run them in a third terminal. After a failed atomic upgrade returns, the failed pod may already be gone; use the captured output and events, then confirm the restored revision with the release-history command above.

oc get events -n "$GATEWAY_NAMESPACE" --sort-by=.lastTimestamp

oc get pods -n "$GATEWAY_NAMESPACE" \
  -l "app.kubernetes.io/instance=$GATEWAY_RELEASE,app.kubernetes.io/component=gateway"

oc describe pod -n "$GATEWAY_NAMESPACE" <non-ready-gateway-pod>

oc logs -n "$GATEWAY_NAMESPACE" \
  <non-ready-gateway-pod> \
  --all-containers \
  --tail=300

If a container restarted, use oc describe pod to identify its name, then read that container's previous logs with oc logs --previous.

For AISIX OSS resource-validation errors, confirm that the saved standalone values identify the complete resources file and that the gateway can reach every configured upstream endpoint.

If SCC admission rejects a target pod, inspect the FailedCreate event and remove fixed runAsUser, runAsGroup, or fsGroup values that were not allocated to the project. Do not grant the gateway service account anyuid.

If CRI-O reports exec: Operation not permitted for a customized image, confirm that the container security context retains NET_BIND_SERVICE. Keep all other capabilities dropped.

Prerequisites​

Trust the Route Certificate​

Capture the Current Release​

Render the Target Gateway Image​

Validate AISIX OSS Resources​

Check the Target Release​

Upgrade the Gateway​

Verify the Upgrade​

Roll Back​

Troubleshoot a Gateway That Is Not Ready​