Skip to main content
API7.ai Logo
Version: 1.8.0

Upgrade a Gateway on Kubernetes

An AISIX gateway installed with the api7/aisix Helm chart is upgraded as its own release. This procedure applies to AISIX OSS and to gateways connected to AISIX Enterprise or AISIX Cloud. Before the rollout, an open-source AISIX gateway validates its declarative resources with the target image. A connected gateway skips that validation because it receives configuration from its control plane.

For a product-specific OpenShift or OKD procedure, follow Upgrade AISIX OSS or Upgrade AISIX Enterprise. AISIX Cloud customers should coordinate the platform-specific upgrade workflow with API7.

For a connected gateway, the product determines where the upgrade procedure starts. For AISIX Enterprise, first follow Upgrade AISIX Enterprise and return here after the upgraded control plane passes verification. For AISIX Cloud, API7 operates the control plane, so its gateways start directly with this procedure.

Prerequisites​

Before starting the upgrade:

  • Configure kubectl to access the cluster and namespace that contain AISIX.
  • Install Helm 3.15 or a later Helm 3 release, cURL, and jq. For AISIX OSS, also install Docker to validate its resources.
  • Read the release notes for every release after the installed version. Apply any required configuration or observability changes in order.
  • Confirm that the load balancer removes pods whose /readyz check fails.
  • For a multi-replica deployment, confirm that shared Redis is configured when rate-limit counters or cache entries must remain consistent across replicas. If you plan to increase a single-replica deployment before the upgrade, configure shared Redis first or explicitly accept the temporary per-replica behavior. See Prepare for Production.

The examples use aisix-dp as the release and namespace, matching the gateway installation guides and dashboard-generated Helm command. If your installation uses another value, run helm list -A, find the release whose CHART column is aisix-<version>, and use its release name and namespace throughout this procedure. A wrong value makes helm upgrade report that the release has no deployed releases and leaves the Deployment lookup below empty.

Set the release details used by the examples:

export GATEWAY_NAMESPACE="aisix-dp"
export GATEWAY_RELEASE="aisix-dp"
export TARGET_VERSION="1.8.0"
export GATEWAY_UPGRADE_TIMEOUT="60m"

The 60-minute timeout is a conservative starting allowance for the chart defaults. These defaults include two replicas, a 1,230-second termination ceiling per pod, and a five-minute startup-probe budget. It is not the expected duration.

Kubernetes can overlap pod termination, and Helm does not require old pods to finish before it considers the updated Deployment ready. Recalculate the timeout when changing replicas, update strategy, termination grace, probe budgets, or when image pulls and scheduling are slower.

Confirm that the cluster can schedule the surge pod; a longer timeout does not resolve a quota or capacity rejection. If an atomic upgrade fails, Helm can use another full timeout period for its rollback, so reserve that time in the maintenance window.

Refresh the API7 chart repository and confirm that the target chart is available:

helm repo add api7 https://charts.api7.ai
helm repo update
helm show chart api7/aisix --version "$TARGET_VERSION"

Capture the Current Release​

Record the installed chart and application versions:

helm list -n "$GATEWAY_NAMESPACE"
kubectl get pods -n "$GATEWAY_NAMESPACE"

Resolve the Deployment name from the Helm release label. This works when the chart uses a name override:

export GATEWAY_DEPLOYMENT="$(kubectl get deployment -n "$GATEWAY_NAMESPACE" \
  -l "app.kubernetes.io/instance=$GATEWAY_RELEASE,app.kubernetes.io/component=gateway" \
  -o jsonpath='{.items[0].metadata.name}')"

printf 'Gateway: %s\n' "$GATEWAY_DEPLOYMENT"

Stop and verify the release name and namespace if the value is empty.

Save only the user-supplied values. The file can contain credentials, so create it with restrictive permissions. Do not export all computed defaults with helm get values --all; carrying old defaults forward can prevent a new chart default from taking effect.

umask 077
touch gateway-values.yaml
chmod 600 gateway-values.yaml
helm get values "$GATEWAY_RELEASE" -n "$GATEWAY_NAMESPACE" -o yaml > gateway-values.yaml

Keep this file with the upgrade records, and remove the local copy after the observation period.

Check whether the values pin an image tag:

helm get values "$GATEWAY_RELEASE" -n "$GATEWAY_NAMESPACE" -o json \
  | jq '{gateway_image_tag: (.image.tag // "")}'

An empty tag uses the target chart's application version. If the value pins an older AISIX image, remove it from the saved values or update it to the target version. Preserve a deliberate custom repository, and use a target-compatible tag for a custom build.

Prepare Application Settings​

Review extraEnvVars in gateway-values.yaml before rendering the target release. Only these overrides carried forward from chart 1.5.0 need the post-upgrade two-pass cleanup:

  • AISIX_CONFIG_PATH
  • AISIX_RATELIMIT__REDIS__URL, when rateLimit.backend is redis and either rateLimit.redis.url or rateLimit.redis.existingSecret is set

For a name in that list, configure or confirm the replacement chart or Secret-backed setting now, but leave the existing extraEnvVars override in place for the version upgrade. After the target release is healthy, remove the override through Remove a Chart-Owned Override. The final Deployment should contain one entry with that name.

Move every other gateway application setting to config, configSecrets, rateLimit.redis, or its dedicated chart value, then remove its extraEnvVars entry before the ordinary version upgrade. Listener, metrics-listener, control-plane connection, and certificate PEM settings follow this path. Depending on the replacement, the target Deployment can contain zero or one environment entry with the old name; neither outcome uses the two-pass cleanup.

See Set Any Other Gateway Configuration for the chart's config and configSecrets rules and the settings the chart owns.

Render the Target Gateway Image​

Render the target Deployment and capture the exact gateway image that Helm will apply. This preserves the chart's registry and any deliberate repository or tag override:

TARGET_GATEWAY_IMAGE="$(
  helm template "$GATEWAY_RELEASE" api7/aisix \
    --namespace "$GATEWAY_NAMESPACE" \
    --version "$TARGET_VERSION" \
    --values gateway-values.yaml \
    --show-only templates/deployment.yaml \
    | awk '
      $1 == "-" && $2 == "name:" && $3 == "aisix" { in_gateway = 1; next }
      in_gateway && $1 == "image:" { gsub(/"/, "", $2); print $2; exit }
    '
)"
export TARGET_GATEWAY_IMAGE

if [ -z "$TARGET_GATEWAY_IMAGE" ]; then
  echo "Target gateway image was not rendered" >&2
else
  printf 'Target gateway image: %s\n' "$TARGET_GATEWAY_IMAGE"
fi

Do not continue if the target gateway image was not rendered.

Validate AISIX OSS Resources​

This section applies only to AISIX OSS. For a gateway connected to AISIX Enterprise or AISIX Cloud, continue to Check the Target Release.

Identify which resource source the release uses without printing the inline resource document:

helm get values "$GATEWAY_RELEASE" -n "$GATEWAY_NAMESPACE" -o json \
  | jq '{inline_resources: (((.standalone.resources // {}) | length) > 0), existing_secret: (.standalone.existingSecret // ""), existing_config_map: (.standalone.existingConfigMap // "")}'

export TARGET_RESOURCES_FILE="$(mktemp "$PWD/aisix-resources.XXXXXX")"
chmod 600 "$TARGET_RESOURCES_FILE"

Use the command for the reported source. For inline resources, render the target chart's managed Secret and extract the complete document:

helm template "$GATEWAY_RELEASE" api7/aisix \
  --namespace "$GATEWAY_NAMESPACE" \
  --version "$TARGET_VERSION" \
  --values gateway-values.yaml \
  --show-only templates/secret.yaml \
  | awk '
    /^  resources\.yaml: \|$/ { in_resources = 1; next }
    in_resources && /^---$/ { exit }
    in_resources { sub(/^    /, ""); print }
  ' > "$TARGET_RESOURCES_FILE"

For an existing Secret or ConfigMap, replace the placeholder with the name reported by the preceding jq command:

# Choose one command.
kubectl get secret -n "$GATEWAY_NAMESPACE" \
  <existing-secret-name> \
  -o jsonpath='{.data.resources\.yaml}' \
  | base64 --decode > "$TARGET_RESOURCES_FILE"

kubectl get configmap -n "$GATEWAY_NAMESPACE" \
  <existing-config-map-name> \
  -o jsonpath='{.data.resources\.yaml}' > "$TARGET_RESOURCES_FILE"

Validate the extracted document with TARGET_GATEWAY_IMAGE, which was captured from the target Deployment render. Export every environment variable referenced by the resource file, and add one -e option for each variable. The example uses OPENAI_API_KEY and CALLER_API_KEY; replace them with the variables in your file.

The command streams the document into the short-lived container instead of bind-mounting the host file. The gateway image can therefore read the document without weakening its host permissions or changing its SELinux label.

docker run --rm -i \
  -e OPENAI_API_KEY \
  -e CALLER_API_KEY \
  --entrypoint /bin/sh "$TARGET_GATEWAY_IMAGE" \
  -c 'umask 077; cat > /tmp/resources.yaml && exec /usr/local/bin/aisix validate --resources /tmp/resources.yaml' \
  < "$TARGET_RESOURCES_FILE" && {
    rm -f "$TARGET_RESOURCES_FILE" &&
    unset TARGET_RESOURCES_FILE
  }

Resolve every validation error before continuing. A failed validation retains the extracted file for inspection and returns a nonzero status. If the resource file does not reference environment variables, omit the -e options.

Check the Target Release​

Have Helm render the upgrade against the live cluster before changing the release. --dry-run=server permits cluster lookups and OpenAPI validation, while --hide-secret prevents Helm from printing rendered Secrets to the terminal or CI log:

helm upgrade "$GATEWAY_RELEASE" api7/aisix \
  --namespace "$GATEWAY_NAMESPACE" \
  --version "$TARGET_VERSION" \
  --values gateway-values.yaml \
  --dry-run=server \
  --hide-secret

This command does not submit the rendered objects to admission. Resolve every reported error before continuing.

Upgrade the Gateway​

Before running Helm, open a second terminal. Select the same cluster context and export GATEWAY_NAMESPACE and GATEWAY_RELEASE with the values from the original terminal; shell variables are not shared between terminals. Repeat this setup in any additional diagnostic terminal. Watch the gateway pods until Helm returns:

export GATEWAY_NAMESPACE="aisix-dp"
export GATEWAY_RELEASE="aisix-dp"

kubectl get pods -n "$GATEWAY_NAMESPACE" \
  -l "app.kubernetes.io/instance=$GATEWAY_RELEASE,app.kubernetes.io/component=gateway" \
  --watch

If a new pod stalls or restarts, press Ctrl+C to stop the watch, or open a third terminal. Capture the pod's description and logs while Helm is still waiting in the original terminal. An atomic rollback can remove the failed target pods before the Helm command returns; see Troubleshoot a Gateway That Is Not Ready.

Run the upgrade in the original terminal. Leave a chart-owned extraEnvVars override in place. After this upgrade is verified, remove that override in Remove a Chart-Owned Override.

helm upgrade "$GATEWAY_RELEASE" api7/aisix \
  --namespace "$GATEWAY_NAMESPACE" \
  --version "$TARGET_VERSION" \
  --values gateway-values.yaml \
  --atomic \
  --timeout "$GATEWAY_UPGRADE_TIMEOUT"

--atomic blocks until the target release becomes ready or Helm finishes rolling it back after a failure. If the command exits with a nonzero status, do not treat a later successful kubectl rollout status as proof that the target version succeeded; it may be reporting the restored revision. Retain the diagnostics captured while Helm was running, then inspect the release history and verify the restored gateway release:

helm history "$GATEWAY_RELEASE" -n "$GATEWAY_NAMESPACE"

With the default rolling-update strategy and readiness probe, a new gateway stays out of Service endpoints until it loads usable configuration. Keep readinessProbe.enabled set to true, and do not select the Recreate strategy when the upgrade must avoid traffic interruption. A single-replica deployment can still have an interruption while its only pod is replaced.

If the Helm command succeeds, confirm that the target rollout is complete and every gateway pod is ready:

kubectl rollout status -n "$GATEWAY_NAMESPACE" \
  "deployment/$GATEWAY_DEPLOYMENT" \
  --timeout="$GATEWAY_UPGRADE_TIMEOUT"

kubectl get pods -n "$GATEWAY_NAMESPACE"

Verify the Upgrade​

Set AISIX_PROXY to the gateway origin without a trailing slash, then list the models available to the caller:

export AISIX_PROXY="YOUR_AISIX_GATEWAY_URL"
export AISIX_API_KEY="<caller-api-key>"

curl -fsS "$AISIX_PROXY/v1/models" \
  -H "Authorization: Bearer $AISIX_API_KEY"

Send a known-good inference request for one of the listed models. Check the access logs, metrics, and other telemetry that the deployment depends on. Keep the saved values until the gateway has operated normally through the observation period defined by your change policy.

For a gateway connected to AISIX Enterprise or AISIX Cloud, also confirm in the dashboard that every upgraded gateway reports the target version and a healthy status.

Remove a Chart-Owned Override​

Use this section only for AISIX_CONFIG_PATH, or for AISIX_RATELIMIT__REDIS__URL when rateLimit.backend is redis and either rateLimit.redis.url or rateLimit.redis.existingSecret is set. Repeat it for each of those names. Do not use it for another setting that moves into config, configSecrets, a dedicated chart value, or a mounted file; remove that override before the version upgrade.

For a name in this recovery path, leave the override in place for the version upgrade above. The chart renders its own entry ahead of yours, so the version upgrade does not delete the chart-owned variable.

Set the exact chart-owned name, then count it on the gateway Deployment:

export GATEWAY_ENV_NAME="<chart-owned-environment-variable>"

gateway_env_count() {
  kubectl get deployment "$GATEWAY_DEPLOYMENT" \
    --namespace "$GATEWAY_NAMESPACE" \
    -o json | jq --arg name "$GATEWAY_ENV_NAME" \
    '[.spec.template.spec.containers[0].env[]? | select(.name == $name)] | length'
}

export GATEWAY_ENV_COUNT="$(gateway_env_count)"
printf '%s appears %s time(s)\n' "$GATEWAY_ENV_NAME" "$GATEWAY_ENV_COUNT"

If the count is 1, the chart entry is already correct and no cleanup is needed. Continue only when the count is 2 or 0; stop on any other value.

Before either recovery path, record the latest successful revision on the target chart version whose gateway Deployment still lists the name twice. This is the restore point even when the current Deployment already has no copy. In helm history, consider only revisions whose status is deployed or superseded. Start with the newest one on the target chart version, set it as the candidate, and count the name in that revision's gateway Deployment:

helm history "$GATEWAY_RELEASE" -n "$GATEWAY_NAMESPACE"

export GATEWAY_ENV_REVISION="<candidate-target-version-revision>"

export GATEWAY_REVISION_ENV_COUNT="$(helm get manifest "$GATEWAY_RELEASE" \
  --namespace "$GATEWAY_NAMESPACE" \
  --revision "$GATEWAY_ENV_REVISION" \
  | awk -v deployment="$GATEWAY_DEPLOYMENT" -v env_name="$GATEWAY_ENV_NAME" '
    /^---$/ {
      kind = ""; resource_name = ""; in_metadata = 0
      in_containers = 0; container_index = 0
    }
    /^kind: Deployment$/ { kind = "Deployment" }
    kind == "Deployment" && /^metadata:$/ { in_metadata = 1; next }
    in_metadata && /^  name: / { resource_name = $2; in_metadata = 0 }
    kind == "Deployment" && resource_name == deployment && /^      containers:$/ {
      in_containers = 1; container_index = 0; next
    }
    in_containers && /^      [^ ]/ { in_containers = 0 }
    in_containers && /^        - name: / { container_index++; next }
    in_containers && container_index == 1 &&
      $0 ~ "^[[:space:]]*- name: " env_name "$" { count++ }
    END { print count + 0 }
  ')"

printf 'revision %s contains %s copy or copies\n' \
  "$GATEWAY_ENV_REVISION" "$GATEWAY_REVISION_ENV_COUNT"

If the revision count is not 2, repeat the candidate and count commands with the preceding successful target-version revision. Keep GATEWAY_ENV_REVISION set to the first candidate whose count is 2. Stop if no successful revision on the target chart version contains both entries. An older chart revision is not a safe restore point for this cleanup; follow Roll Back instead.

These recovery upgrades omit --atomic and stay on the verified target version. With --atomic, a failed restore would roll back only to the revision created by the destructive pass, where the variable is already missing.

When the Name Appears Twice​

Remove the override from gateway-values.yaml. Its replacement chart or Secret-backed setting should already be in place. Then run the destructive pass once. It deletes both entries and returns without waiting:

helm upgrade "$GATEWAY_RELEASE" api7/aisix \
  --namespace "$GATEWAY_NAMESPACE" \
  --version "$TARGET_VERSION" \
  --values gateway-values.yaml

Continue immediately to Restore and Verify the Chart Entry.

When the Name Is Missing​

Keep the override out of gateway-values.yaml, confirm that its replacement chart or Secret-backed setting is in place, and do not run the destructive pass above. Continue directly to Restore and Verify the Chart Entry.

Restore and Verify the Chart Entry​

Run one waiting upgrade. It restores the chart's entry and waits for the rollout:

helm upgrade "$GATEWAY_RELEASE" api7/aisix \
  --namespace "$GATEWAY_NAMESPACE" \
  --version "$TARGET_VERSION" \
  --values gateway-values.yaml \
  --wait \
  --timeout "$GATEWAY_UPGRADE_TIMEOUT"

Count the live Deployment again. Continue only when it prints 1:

export GATEWAY_ENV_COUNT="$(gateway_env_count)"
printf '%s appears %s time(s)\n' "$GATEWAY_ENV_NAME" "$GATEWAY_ENV_COUNT"

If the waiting upgrade fails, or the count is not 1, restore the recorded target-version revision:

helm rollback "$GATEWAY_RELEASE" "$GATEWAY_ENV_REVISION" \
  --namespace "$GATEWAY_NAMESPACE" \
  --wait \
  --timeout "$GATEWAY_UPGRADE_TIMEOUT"

Roll Back​

If Helm's atomic upgrade fails, it restores the previous release automatically. Diagnose the failure before retrying.

To roll back after a successful upgrade, identify the previous revision and restore it:

helm history "$GATEWAY_RELEASE" -n "$GATEWAY_NAMESPACE"

helm rollback "$GATEWAY_RELEASE" <previous-revision> \
  --namespace "$GATEWAY_NAMESPACE" \
  --wait \
  --timeout "$GATEWAY_UPGRADE_TIMEOUT"

Confirm that the restored pods become ready and repeat the traffic verification.

Troubleshoot a Gateway That Is Not Ready​

After stopping the watch, run these diagnostics while Helm is still waiting, using the new target pod reported by the watch. Alternatively, run them in a third terminal. After a failed atomic upgrade returns, the failed pod may already be gone; use the captured output and events, then confirm the restored revision with the release-history command above.

kubectl get events -n "$GATEWAY_NAMESPACE" --sort-by=.lastTimestamp

kubectl get pods -n "$GATEWAY_NAMESPACE" \
  -l "app.kubernetes.io/instance=$GATEWAY_RELEASE,app.kubernetes.io/component=gateway"

kubectl describe pod -n "$GATEWAY_NAMESPACE" <non-ready-gateway-pod>

kubectl logs -n "$GATEWAY_NAMESPACE" \
  <non-ready-gateway-pod> \
  --all-containers \
  --tail=300

If a container restarted, use kubectl describe pod to identify its name, then read that container's previous logs with kubectl logs --previous.

For AISIX OSS resource-validation errors, confirm that the saved standalone values identify the complete resources file and that the gateway can reach every configured upstream endpoint.

For a gateway connected to AISIX Enterprise or AISIX Cloud, verify that the pod can reach the existing data-plane manager endpoint over mutual TLS. If the endpoint uses a Route, confirm that the Route uses TLS passthrough. In every case, confirm that the endpoint host name still matches the address in the gateway configuration.