Skip to main content
API7.ai Logo
Version: 3.10.x

Upgrade from 3.9 LTS to 3.10 LTS

API7 Gateway 3.9 LTS can upgrade directly to 3.10 LTS. This guide documents the verified 3.9.19 to 3.10.6 route for Helm and Kubernetes with an external PostgreSQL database.

For an in-place upgrade, one target Control Plane (CP) Dashboard migrates the existing database. After the target CP is healthy, 3.9.19 Data Plane (DP) nodes continue serving traffic while 3.10.6 nodes are added, validated, and rolled into service. For a dual-cluster upgrade, the source cluster remains unchanged while an independent target cluster is built and receives traffic gradually.

Applicability​

The direct route is self-service only when the source, target, database, and deployment method match these boundaries.

AreaSupported scope
Release-line routeUpgrade an earlier 3.9.x source to 3.9.19 before following this guide.
Direct upgrade anchors3.9.19 to 3.10.6
CP in-place production upgradeExternal PostgreSQL 15.x with the Helm and Kubernetes procedure in this guide. Contact API7 Support before using the chart's built-in PostgreSQL, MySQL, Microsoft SQL Server, or another database.
Dual-cluster production upgradeSelf-service with two independent external PostgreSQL 15.x databases, the exact Helm and Kubernetes artifacts below, and an external load balancer. Contact API7 Support for another database or deployment topology.
Helm and KubernetesExact released artifacts listed below. Gateway releases use a Deployment workload (apisix.kind: Deployment).
Other deployment methodsContact API7 Support for RPM, a change of installation method, DaemonSet gateways, or chart and image combinations outside the versions above.
DP rolling upgrade after in-place CP upgradeA 3.9.19 DP can report Partially Compatible with zero compatibility errors while connected to the 3.10.6 CP. Stop if any source DP reports Incompatible or an unexpected compatibility error.
LDAP Auth with structural DN charactersAPI7 Support-assisted. Source and target DPs require different stored user_dn representations for affected usernames, so the plan must coordinate the target rewrite, traffic switch, and DP-only rollback.
Developer PortalAPI7 Support-assisted. This guide does not provide a self-service procedure for Portal frontend, identity, organization, or resource-ownership migration.

The self-service route was verified with these released artifacts:

ComponentSourceTarget
Control PlaneChart 3.9.7, images v3.9.19Chart 3.10.7, images v3.10.6
Data PlaneGateway chart 3.9.11, image 3.9.19Gateway chart 3.10.13, image 3.10.6

Helm chart versions and product versions are independent. In this route, CP chart 3.10.7 deploys API7 Gateway 3.10.6 components; it does not represent product version 3.10.7.

The database must also satisfy the target requirements in Supported Versions and Interoperability.

Review Compatibility Changes​

Use 3.9.19 as the source baseline when reviewing the release notes from 3.10.0 through 3.10.6. Complete the upgrade actions for behavior that differs in 3.10.6.

The following changes require action for this route:

  • Service model migration: Service Templates, Service Hub publishing, service versions, and service rollback are removed. Publish and verify configurations that must remain active, and archive drafts or historical versions needed for reference. Verify migrated services, routes, upstreams, OpenAPI documents, API product links, and IAM policies after migration.
  • Custom plugin migration: Uploaded custom plugins become scoped to gateway groups. Inventory the source files, current usage, roles that must list or read plugin source, and automation that calls the custom-plugin APIs. Replace /api/custom_plugins calls with /api/gateway_groups/{gateway_group_id}/custom_plugins, add gateway_group_id to GET /api/plugins, and remove gateway_groups from custom-plugin payloads. After migration, verify each required gateway group has the expected plugin assignment and confirm the first target DP loads the plugin successfully. Existing custom-plugin policy grants are migrated, but roles that could previously read plugin source without an explicit permission need gateway:GetCustomPlugin for the required gateway groups. Test target list and upload operations with the affected automation and verify access with a read-only user before shifting production traffic.
  • Gateway runtime: The gateway moves from OpenResty 1.21.4.4 to 1.29.2.4. Test custom plugins, NGINX snippets, modules, and operational scripts against the 3.10.6 image.
  • Authentication and secrets: Test every authentication flow and credential type on the target canary. Keep management writes frozen while 3.9.19 DPs remain because target-only fields and newly encrypted values might not be readable by source DPs.
  • LDAP Auth distinguished names: Before upgrading, inventory consumers whose user_dn contains a character with structural meaning in an LDAP distinguished name, such as ,, +, =, <, >, ;, ", or \. Preserve the current plain-string values in the source backup and do not rewrite them while 3.9.19 DPs serve traffic. If any are present, obtain an API7 Support plan that schedules the RFC 4514 rewrite for the transition to target DPs and restores the original values before a DP-only rollback. Schema validation and the compatibility report do not detect this mismatch.

Run the target compatibility report against every gateway group before upgrading DPs. Treat ADC lint and diff as configuration checks, not proof of runtime compatibility.

Prepare the Upgrade​

Preparation establishes the source state, target manifests, rollback authority, and go/no-go criteria before the database migration begins.

  1. Confirm that every source CP and DP reports 3.9.19 and is healthy. For Helm, confirm CP chart 3.9.7 and gateway chart 3.9.11, and record the image digest running in every pod.

  2. Save the user-supplied values and rendered manifest for every CP and gateway release. Store manifests and Secret data in an approved encrypted location.

  3. Build the target manifests from the defaults for CP chart 3.10.7 and gateway chart 3.10.13, then reapply reviewed production settings. Do not use --reuse-values.

  4. Complete the compatibility review and define acceptance criteria, abort thresholds, observation duration, rollback owner, and rollback deadline.

  5. Export and lint every gateway group with ADC. Use --gateway-group and a separate 3.9.19 filename for each export. See Backup and Restoration.

    adc dump -o "api7ee-{GATEWAY_GROUP}-3.9.19.yaml" \
      --backend api7ee \
      --server "https://{DASHBOARD_ADDR}" \
      --gateway-group "{GATEWAY_GROUP}"

    adc lint -f "api7ee-{GATEWAY_GROUP}-3.9.19.yaml"

  6. Save custom plugin source, DP certificates, target-side integration configuration, and every required source image.

  7. Rehearse the upgrade and rollback with representative public and authenticated traffic.

  8. Create a preparatory database-native backup and restore it into an isolated database. Verify the source version and representative CP resources in the restored copy.

  9. Start the management write freeze. Block writes through Dashboard, Admin API, Developer Portal, ADC, deployment automation, scheduled jobs, direct database integrations, and every other path that can modify the CP database. Repeat and lint every gateway-group ADC export after the freeze; use these final exports for validation and fallback recovery. Keep the freeze in effect until acceptance or rollback completes.

Upgrade the Control Plane​

The production topology determines whether to migrate the existing CP in place or build an independent target cluster.

In-Place Upgrade​

The self-service route uses an in-place Control Plane upgrade. Existing DPs continue proxying their last valid configuration while the source CP is stopped and the target Dashboard migrates the database.

Prepare Target Values​

Create migration and final values from CP chart 3.10.7 defaults. Both files must preserve the external PostgreSQL DSN and set postgresql.builtin to false.

The migration values start exactly one Dashboard and no other target database client:

postgresql:
  builtin: false
dashboard:
  replicaCount: 1
dp_manager:
  replicaCount: 0
developer_portal:
  replicaCount: 0
developer_portal_configuration:
  enable: false
api_usage:
  enable: false
file_server:
  enabled: false
prometheus:
  builtin: false
jaeger:
  builtin: false

Set the production database DSN under dashboard_configuration.database.dsn, dp_manager_configuration.database.dsn, and every enabled optional component. The final values restore the required target replica counts and components only after the migration replica passes validation.

Run the Migration​

  1. Keep the write freeze in effect. Scale every source CP database client to zero and wait until all 3.9.19 CP pods have stopped. Existing DPs remain available.

  2. Take the final database backup and repeat the isolated restore check.

  3. Upgrade the stopped release with the migration values:

    helm upgrade {CP_RELEASE} api7/api7ee3 \
      --namespace {CP_NAMESPACE} \
      --version 3.10.7 \
      --values api7ee-cp-3.10.6-migration-values.yaml \
      --wait \
      --timeout {TIMEOUT}

  4. Watch the migration Dashboard and inspect previous logs if it restarts. A restart, SQLSTATE error, skipped-resource warning, pod that does not become Ready, or another migration error is an abort condition. Preserve the migrated database and logs, keep writes frozen, and execute the rollback.

  5. Confirm that the Dashboard reports 3.10.6 and that migrated services, routes, consumers, credentials, API products, IAM policies, and gateway groups are present.

  6. Apply the final values and restore the required Dashboard, DP Manager, and optional-component replicas. Never run source and target CP versions against the same database.

  7. Confirm every source DP reports Healthy and Partially Compatible with zero compatibility errors before starting DP replacement.

For the in-place strategy, skip the dual-cluster section and continue with Upgrade the Data Plane.

Dual-Cluster Upgrade​

The verified dual-cluster route keeps the source cluster running while the target cluster uses an independently restored database. Follow the dual-cluster strategy with these pair-specific requirements:

  1. Keep the rollback-relevant business-write freeze in effect and leave the source CP, DPs, and database unchanged through the rollback window.
  2. Take the final 3.9.19 database backup, restore it into a new external PostgreSQL 15.x database, and verify representative resources before starting the target CP.
  3. Deploy CP chart 3.10.7 against the target database with one 3.10.6 Dashboard replica and every other target database client stopped. Treat any Dashboard restart, migration error, skipped-resource warning, readiness failure, or missing migrated data as an abort condition.
  4. Validate the target version, migrated resources, IAM policies, active license, and database health. Apply the final target values only after the migration gate passes.
  5. Generate new DP certificates and values from the target Dashboard. Deploy gateway chart 3.10.13 and image 3.10.6 as a separate target gateway release outside the production load balancer.
  6. Confirm every target DP reports Healthy and Compatible with zero errors, then run representative traffic directly through the target Service. Copied source DP records appear as LostConnection in the target Dashboard; evaluate the new target records separately. The copied records can be removed after they report Offline.
  7. Shift traffic in the stages and observation periods defined by the production plan. Keep enough source capacity for rollback and keep writes frozen until acceptance or traffic rollback completes.

After completing the traffic move, skip the in-place DP rolling section and continue with Validate the Upgrade.

Upgrade the Data Plane​

After an in-place CP upgrade, use Data Plane Rolling Upgrade for the replacement mechanics. The chart versions and compatibility gates below are authoritative for this pair.

  1. Generate new Helm values and DP certificates from the target Dashboard for each gateway group.

  2. Deploy one 3.10.6 DP as a separate canary release with gateway chart 3.10.13, apisix.kind: Deployment, one replica, and autoscaling disabled. Keep the canary Service outside the production load balancer.

  3. Wait until the canary reports Healthy and Compatible with zero errors. Send representative public and authenticated traffic directly to the canary.

  4. Record the main Deployment's desired and Ready replica counts. Suspend autoscaling at a verified safe replica count, then configure add-before-drain replacement:

    apisix:
      kind: Deployment
      replicaCount: {SAFE_REPLICA_COUNT}
    updateStrategy:
      type: RollingUpdate
      rollingUpdate:
        maxSurge: 1
        maxUnavailable: 0
    autoscaling:
      enabled: false

  5. Upgrade the main release from gateway chart 3.9.11 and image 3.9.19 to gateway chart 3.10.13 and image 3.10.6:

    helm upgrade {GATEWAY_RELEASE} api7/gateway \
      --namespace {GATEWAY_NAMESPACE} \
      --version 3.10.13 \
      --values api7-gateway-3.10.6-values.yaml \
      --wait \
      --timeout {TIMEOUT}

  6. Watch pods, traffic, and Gateway Instances throughout the rollout. Each target pod must become Ready before a source pod is drained.

  7. Confirm every main DP reports Healthy and Compatible with zero errors. Remove the canary only after the main rollout and representative traffic are accepted, then restore the production autoscaling policy.

Do not create or modify gateway resources while any 3.9.19 DP remains connected. Keep the final source backup, source manifests, certificates, charts, and images until the rollback deadline passes.

Validate the Upgrade​

Before ending the write freeze:

  1. Confirm every CP component reports 3.10.6 and is healthy.

  2. Confirm every production DP reports Healthy and Compatible with no compatibility errors. For dual-cluster, evaluate target DP records separately from copied source records that report LostConnection.

  3. Run adc diff for every gateway group against its matching pre-upgrade export:

    adc diff -f "api7ee-{GATEWAY_GROUP}-3.9.19.yaml" \
      --backend api7ee \
      --server "https://{DASHBOARD_ADDR}" \
      --gateway-group "{GATEWAY_GROUP}"

  4. Verify services, routes, upstreams, consumers, credentials, SSL resources, plugin metadata, global rules, API product links, IAM policies, and the active license.

  5. Run representative public and authenticated traffic through the production Service.

  6. Verify custom plugins, authentication, rate limiting, caching, logging, metrics, health checks, and upstream integrations.

  7. Review CP, DP, and PostgreSQL logs and metrics through the predefined observation period.

Resume management writes only after the named decision owner records that every acceptance criterion passed before the rollback deadline.

Roll Back​

Choose the rollback section for the failed component and strategy. Keep management writes frozen until the restored deployment and required reconciliation are complete.

Roll Back Only the Data Plane​

If no target-only configuration has been written, remove target DPs from the load balancer and roll the main release back to gateway chart 3.9.11, the saved source values, and image 3.9.19. Keep maxSurge: 1 and maxUnavailable: 0, verify each source pod before a target pod is removed, and confirm the source DPs report Healthy and Partially Compatible with zero errors against the target CP.

If the API7 Support plan rewrote LDAP Auth user_dn values for the target DPs, restore the source plain-string values before returning LDAP traffic to 3.9.19 DPs.

If target-only configuration was written, do not return traffic to 3.9.19 DPs. Perform the full CP and database rollback below.

Roll Back the Control Plane and Database​

Do not rely on helm rollback or --atomic; Helm does not restore PostgreSQL state.

  1. Stop and isolate every 3.10.6 DP so it cannot connect to the restored source CP.
  2. Stop every 3.10.6 CP process and other database writer.
  3. Restore the immutable 3.9.19 backup into a new database.
  4. Restore CP chart 3.9.7, the saved source values, and explicit v3.9.19 images against the new database. Start one Dashboard first, verify it, then restore the remaining source CP and DP Manager replicas.
  5. Restore gateway chart 3.9.11, the saved source values and certificates, and image 3.9.19.
  6. Confirm the source CP reports 3.9.19, the license and representative resources are present, every source DP is Healthy and Compatible, and public and authenticated traffic passes.
  7. Keep the migrated database and target components isolated until the failed upgrade is understood.

Changes made after the final source backup are not present after restoration. Keep writes frozen until rollback verification and any required reconciliation are complete.

Roll Back a Dual-Cluster Upgrade​

Redirect traffic to the source cluster only while rollback-relevant writes remained frozen or after the tested reconciliation procedure completed. Target-only operational state is abandoned with the target cluster. If the databases diverged without a tested reconciliation path, do not perform a traffic-only rollback.