Upgrade from 3.8 LTS to 3.10 LTS

Some API7 Gateway operators move directly from 3.8 LTS to 3.10 LTS without deploying 3.9. This guide covers the verified 3.8.23 to 3.10.6 route.

For an in-place upgrade, the target Control Plane (CP) applies the intermediate database migrations, including the new service model. After the target CP is healthy, the Data Plane (DP) remains available while 3.8.23 nodes are replaced with 3.10.6 nodes. For a dual-cluster upgrade, the source cluster remains unchanged while an independent target cluster is built and receives traffic gradually. Confirm that the deployment is within the supported scope before scheduling either strategy.

Applicability

The direct route is self-service only when the source, target, database, and deployment method match the following boundaries.

Area	Supported scope
Release-line route	Upgrade an earlier 3.8.x source to 3.8.23 before following this guide.
Direct upgrade anchors	3.8.23 to 3.10.6
CP in-place production upgrade	External PostgreSQL 15.x with the Helm and Kubernetes procedure in this guide. Contact API7 Support before using the chart's built-in PostgreSQL, MySQL, Microsoft SQL Server, or another database.
Dual-cluster production upgrade	Self-service with two independent external PostgreSQL 15.x databases, the exact Helm and Kubernetes artifacts below, and an external load balancer. Contact API7 Support for another database or deployment topology.
Helm and Kubernetes	Exact released artifacts listed below. Single- and multi-replica CP deployments use one Dashboard replica for the migration. Gateway releases use a Deployment workload (apisix.kind: Deployment).
Other deployment methods	Contact API7 Support for RPM, a change of installation method, a DaemonSet gateway, or Helm chart and image combinations outside the versions above.
DP rolling upgrade after in-place CP upgrade	A 3.8.23 DP can report Partially Compatible when its compatibility report has no errors. If it reports Incompatible, stop the rollout and follow the documented rollback or API7 Support guidance.
LDAP Auth with structural DN characters	API7 Support-assisted. Source and target DPs require different stored user_dn representations for affected usernames, so the plan must coordinate the target rewrite, traffic switch, and DP-only rollback.
Developer Portal	Existing identities and resource ownership require an API7 Support-approved migration plan. This guide does not provide a self-service Portal migration.

The self-service route was verified with these released artifacts:

Component	Source	Target
Control Plane	Chart 0.17.37, explicit images v3.8.23	Chart 3.10.8, images v3.10.6
Data Plane	Gateway chart 0.2.40, image 3.8.23	Gateway chart 3.10.13, image 3.10.6

Helm chart and product versions are independent. CP chart 3.10.8 deploys API7 Gateway 3.10.6 components; it is not product version 3.10.8.

Known Dashboard restart in product version 3.10.6

The first 3.10.6 Dashboard process can terminate once with SQLSTATE 0A000 and cached plan must not change result type after migrating a 3.8.23 PostgreSQL database. Kubernetes restarts the pod automatically. The replacement process can complete startup because the schema migration has already finished.

This restart extends the management-plane interruption, but existing DPs continue proxying their last valid configuration.

Continue only when these conditions pass:

• The previous logs contain this exact error and no unrelated migration failure.
• The pod restart count is exactly one.
• The replacement becomes Ready, reports 3.10.6, and has no current migration error.
• All required resources pass validation.

A second restart, another error, a skipped-resource warning, a pod that does not become Ready, or missing migrated data requires rollback. If the production policy does not permit this bounded restart, contact API7 Support or wait until a product release containing the fix is available and this guide has been updated.

The database must also satisfy the target requirements in Supported Versions and Interoperability.

If Developer Portal is enabled, do not begin the production CP/DP upgrade or Portal domain transition until an API7 Support-approved migration plan has succeeded in staging. The plan must state whether Portal remains offline during a phased gateway upgrade or switches domains in the same maintenance window. If Portal is not enabled, skip the Portal migration section.

Review Critical Changes

The direct route applies all intermediate migrations at once. Review these product changes before working through the detailed compatibility checklist.

Service Model Migration

Service Templates, Service Hub publishing, service versions, and service rollback are removed. During startup, the target CP migrates each published service into a gateway-group-owned service. It preserves legacy primary records for investigation but does not provide a reverse migration through those records.

The migration moves published runtime state. It does not migrate unpublished template drafts or historical versions. Publish what must remain active and archive any draft design content needed for reference before the final backup.

Developer Portal Replacement

The built-in Portal frontend and SSO model from 3.8 are replaced by a separate Developer Portal frontend and Portal-level authentication. If the source deployment uses Developer Portal, obtain and validate the API7 Support-approved plan described in Plan Developer Portal Migration before scheduling the production upgrade. The plan must include the later Portal domain switch. Deployments that do not use Developer Portal can skip those requirements.

Gateway Runtime Upgrade

The gateway moves from OpenResty 1.21.4.4 to 1.29.2.4. Test custom plugins, NGINX snippets, modules, and operational scripts against the target image. The generated NGINX configuration also uses the newer HTTP/2 directive syntax.

Custom Plugin Scope Migration

Uploaded custom plugins become gateway-group resources in 3.10.6. The migration creates one target record for each gateway group where a plugin is deployed. A plugin that is not assigned to any gateway group is not migrated.

Inventory the source files and assignments before upgrading. After migration, verify every required gateway group has the expected plugin and confirm the first target DP loads it. Code uploaded after the upgrade is stored only in the new table, so a full rollback restores the code represented by the pre-upgrade database and saved source files.

Inventory automation that lists or uploads custom plugins. Replace /api/custom_plugins calls with /api/gateway_groups/{gateway_group_id}/custom_plugins, add gateway_group_id to GET /api/plugins, and remove gateway_groups from custom-plugin payloads. Test target list and upload operations with the affected automation before shifting production traffic; the old custom-plugin endpoints return HTTP 410.

Compatibility Checklist

Complete the following reviews and changes before upgrading. Features and fixes that require no operator action remain in the release notes.

Correct the Source Configuration

Apply and validate these source-state corrections before the production upgrade window. Treat each behavior-changing correction as an independent production change with its own validation and rollback plan. Do not batch unverified corrections immediately before the final backup.

Area	Required review or action
Service Templates	Publish and verify every configuration that must remain active. Archive unpublished drafts and historical versions that must be retained for reference.
HMAC Auth	signed_headers defaults to ["date"]. Ensure clients sign the Date header, or explicitly configure signed_headers to match the client signature. An incompatible signature begins failing when traffic reaches a target DP, not during the CP database migration.
OpenID Connect	For non-bearer-only flows, configure a shared session.secret of at least 16 characters. Explicitly choose the TLS verification policy.
SAML Auth	Configure the same secret of 8 to 32 characters on every DP, then test multi-node sign-in and logout.
CAS Auth	Configure the same cookie.secret of at least 32 characters on every DP.
LDAP Auth	Inventory consumers whose user_dn contains a structural DN character in the username. Preserve the current plain-string values in the source backup and do not rewrite them while 3.8.23 DPs serve traffic. If any are present, obtain the API7 Support-assisted transition and rollback plan required by the applicability table.
Workflow	Add valid rules to every configuration, or remove the plugin where it is no longer needed. The field is required when a target DP first loads the existing configuration.
Logger body limits	Replace zero, negative, or quoted max_req_body_bytes and max_resp_body_bytes values. Invalid resources appear in the compatibility report and are not published to the DP.
Prometheus metadata	Remove structural metric labels, such as type on latency and code on status, from disabled_labels. Non-structural labels can still be disabled.
OpenTelemetry and AI moderation	Replace arrays or objects in OpenTelemetry resource attributes and collector headers with scalar values. Set AI moderation deny_code to an integer from 200 through 599.

Prepare the Target Deployment

Add these changes to the target manifests and apply them only when deploying the target components. Do not alter the running source DP configuration before the final backup.

Area	Required review or action
Gateway runtime	Test custom plugins, NGINX snippets, modules, and scripts against the target image.
OpenID Connect trust	If ssl_verify is enabled, add the identity provider CA to the target DP trust store and verify the certificate chain.
OpenAPI to MCP	OpenAPI2MCP is no longer bundled in the gateway image. Enable openapiToMcp.enabled in the target Helm values when using openapi-to-mcp or mcp-tools-acl. If the openapi-to-mcp plugin attribute uses a non-default port, set openapiToMcp.port to the same value and validate the rendered manifest.
Gateway memory	The target raises several shared-memory defaults. Raise memory requests and limits in the target DP manifests, or explicitly size unused dictionaries. See Shared Dictionary Sizing.
Helm DNS	The target chart no longer supplies fixed public resolvers by default. An empty resolver list uses the pod's /etc/resolv.conf. Set explicit resolvers in the target values only when cluster DNS is unsuitable, and test service discovery and domain-name upstreams.
Alibaba Cloud Logging trust	Certificate verification is enabled by default through ssl_verify: true. Add the issuing CA to the target DP trust store. Disable verification only when the risk is explicitly accepted and documented.
Forwarded-header trust	Inventory and test every proxy hop. Configure apisix.trusted_addresses only when the deployment needs an explicit trust boundary or upstream applications depend on trusted forwarded headers. Include every trusted load balancer and reverse-proxy address or CIDR.

During the CP Migration

Verify these changes after the target CP starts.

Area	Target behavior and verification
Services and IAM	The CP migrates published services and rewrites backed-up Service Template and published-service IAM references. Verify services, routes, upstreams, OpenAPI documents, API product links, and IAM policies. Contact API7 Support if IAM references are not migrated correctly. See the 3.10.0 release notes.
Custom plugins	Verify every plugin assigned to a gateway group was migrated into that group. Re-upload any unassigned source plugin that is still required to each target gateway group that needs it. Confirm roles that read plugin source have the gateway:GetCustomPlugin permission.
Dashboard access	Existing sessions are invalidated once, URLs no longer use the /zh prefix, failed-login lockout is enabled, and new passwords require at least 12 characters. Sign in again, update saved links, and review the login failure policy. Integrations for users with 2FA must use access tokens instead of HTTP Basic Auth.
SQL Server (API7 Support-assisted only)	The first target startup enables READ_COMMITTED_SNAPSHOT and can disconnect existing sessions and in-flight transactions once. Plan for this interruption. This does not apply to PostgreSQL or MySQL.

During the Mixed-Version Window

The 3.10.6 CP can temporarily manage 3.8.23 DPs, which may report Partially Compatible while continuing to serve their last valid configuration. Do not run 3.8.23 and 3.10.6 CP processes against the same database.

More credential-bearing plugin fields are encrypted at rest by the target CP. Older DPs cannot decrypt values newly written by the target CP. Do not create or update gateway resources, enable target-only plugins or fields, or change secret references until every DP is upgraded and reports Compatible.

Validate Target-DP Behavior

These changes take effect when a target DP loads the configuration. Test every applicable behavior directly on the first target DP before adding it to the production load balancer. Repeat representative checks after every DP is upgraded and before ending the configuration freeze.

Area	Target behavior and required action
JWT Auth	exp and nbf claims are verified by default when claims_to_verify is absent or empty. Test token expiry and not-before handling. Set claims_to_verify explicitly only when a narrower set is intentional.
OpenID Connect	ssl_verify defaults to true, and refresh_session_interval no longer defaults to 900. Verify the identity provider trust chain. Set refresh_session_interval: 900 if the deployment relies on the old periodic silent re-authentication. Test required_scopes, audience matching, and issuer validation because 3.10.6 enforces these configured authorization checks in flows that previously skipped them.
Limit Conn and Limit Req	policy is required on the next create or update. Existing stored configurations continue running. Add an explicit value, such as local, before the next change.
Batch Requests	Batches default to at most 1000 entries, extra fields are rejected, and timeout must be at least 1 millisecond. Set plugin metadata max_pipeline_items for larger batches and remove undocumented fields.
Logging backlog	Logging batch processors default to a maximum of 8192 pending entries. Review max_pending_entries with request volume, body size, batch_max_size, and log-server outage behavior; entries are discarded after the limit is reached.
LDAP Auth	Consumer matching now uses the escaped bind DN. For affected usernames, follow the API7 Support-assisted plan to schedule the target RFC 4514 rewrite and restore the source plain-string value before a DP-only rollback. Schema validation and the compatibility report do not detect this mismatch.
AI Proxy Multi	Duplicate instance names are rejected on new writes and reported as compatibility errors for existing configurations. Rename duplicate instances before accepting the target DPs.
Rate limiting	limit-count includes Redis Sentinel, sliding windows, multiple rules, variables, and sync_interval. Redis-backed counters use new versioned keys and reset once. Existing limit-count-advanced configurations remain supported, so the temporary migration required on the 3.9 path is not required for this direct upgrade.
Proxy Cache	Authenticated requests are isolated by consumer by default. Both storage strategies skip responses with Set-Cookie; memory storage also skips Cache-Control: private, no-store, and no-cache. Set consumer_isolation: false only for intentional sharing. cache_set_cookie: true affects memory storage only; review on-disk NGINX cache rules separately. Existing memory-cache entries are fetched again once because 3.10.6 changes the cache-key layout.
GraphQL Proxy Cache	Default keys include host, route, service, and consumer identity. Responses with Set-Cookie are skipped by default. Set consumer_isolation: false only for intentional sharing; host, route, and service remain in the key. cache_set_cookie: true affects memory storage only.
Request and response bodies	Plugins that buffer bodies enforce limits. Common defaults are 64 MiB, and logger capture defaults to 512 KiB. Compare them with client_max_body_size and the largest production payload.
Health check API	GET /v1/healthcheck and GET /v1/healthcheck/{src_type}/{src_id} return a new target shape using ip, port, status, and failure counters instead of healthy_nodes. Update parsers, dashboards, and alerts.
LLM metrics	apisix_llm_ttft is replaced by apisix_llm_latency{type="ttft"}. Update Prometheus queries, recording rules, alerts, and Grafana dashboards.
Alibaba Cloud Logging	Confirm logging succeeds with the target certificate-verification policy and trust store.
Forwarded headers	For an untrusted peer, the gateway overwrites X-Forwarded-Proto, X-Forwarded-Host, and X-Forwarded-Port, and clears Forwarded. X-Forwarded-For is preserved when apisix.trusted_addresses is unset; after a trust boundary is configured, it is cleared for peers outside that boundary. Test the values received by upstream applications. In custom NGINX log formats, $http_x_forwarded_proto, $http_x_forwarded_host, and $http_x_forwarded_port now record client-supplied values. Use $scheme, $var_x_forwarded_host, and $var_x_forwarded_port when the overwritten values are required.

Plan Developer Portal Migration

Skip this section if the source deployment does not use Developer Portal.

Deploy the separate Developer Portal frontend, plan the domain switch, and migrate customizations. Configure and test the required identity providers with Configure SSO for the Developer Portal.

The new frontend stores users and sessions in a separate Portal database. Restoring and migrating the 3.8 CP database does not make old local credentials valid. New frontend organizations also receive different developer external IDs, so registration or SSO alone does not transfer ownership of migrated applications, subscriptions, or credentials.

Before scheduling the upgrade, inventory developers and their resources and obtain a supported linking, reassignment, or recreation procedure from API7 Support. The procedure must cover the CP database, the separate target frontend database, write freezes, backup and restoration, post-snapshot reconciliation, the domain switch, and session behavior. Test the complete procedure before switching the Portal domain. Do not copy password hashes or manually rewrite ownership in the database.

Review the remaining Portal changes:

Area	Required review or action
Draft API products	Authentication rules are synchronized only for published API products. Publish a product or apply another access control if draft-product routes must remain protected.
Credential secrets	Key Auth keys and Basic Auth passwords are returned only when created or regenerated. Update integrations to capture each secret at that time.
Sign-up consent	tosURL and beforeSignUpButtonHtml are replaced by signUpConsentLabel. Move the consent content to signUpConsentLabel; consent is enforced only when it is configured.
SSO-only domains	Local sign-in, sign-up, magic-link, and password-reset endpoints are rejected for SSO-only email domains. Confirm affected developers use the configured SSO provider.
Portal API proxy	The frontend proxy is default-deny and exposes only supported organization-scoped resources. Move other integrations to a supported path or call the CP API directly.

Prepare the Upgrade

Preparation establishes the source state, target artifacts, rollback authority, and production decision gates before any database migration begins.

1. If the source is earlier than 3.8.23, follow that patch upgrade's backup and rollback procedure before continuing. Verify that the CP and every DP report 3.8.23 and are healthy.

2. Review the target supported versions and interoperability for the database, Kubernetes, Helm, and client tools used by the deployment.

3. For Helm and Kubernetes, confirm that the source CP uses chart 0.17.37 with explicit v3.8.23 Dashboard, DP Manager, and optional-component image tags. Confirm that each source gateway release uses chart 0.2.40 with image tag 3.8.23. Record the actual images running in every pod; the source charts otherwise default to v3.8.21 for CP components and 3.8.21 for the gateway.

4. Save the user-supplied values and rendered manifest for every CP and gateway release. These artifacts can contain credentials and Secret data. Store them in an approved encrypted location with restricted access, and do not commit them to source control.

   umask 077
   
   helm get values {CP_RELEASE} --namespace {CP_NAMESPACE} --output yaml \
     > api7ee-cp-3.8.23-values.yaml
   
   helm get manifest {CP_RELEASE} --namespace {CP_NAMESPACE} \
     > api7ee-cp-3.8.23-manifest.yaml

   Repeat both commands for every gateway release, using gateway-group-specific filenames.

5. Build the target values from the defaults for CP chart 3.10.8 and gateway chart 3.10.13, then reapply the required production settings. Do not use --reuse-values or submit the source chart's complete values unchanged; both approaches can retain source image tags and omit target defaults.

6. Complete the source corrections, target deployment preparation, and Developer Portal actions that apply to the deployment.

7. Define the production decision plan: the change owner, observation duration, acceptance criteria, abort thresholds, rollback decision owner, and deadline for choosing rollback before writes can resume. For dual-cluster or Developer Portal deployments, include how any post-snapshot writes would be blocked or reconciled across the old and target databases.

8. Export every gateway group with a current ADC release. Specify --gateway-group and use a distinct, source-versioned filename for each group; otherwise ADC operates on the default group and output can be overwritten. Run adc lint on every export. See Backup and Restoration.

   adc dump -o "api7ee-{GATEWAY_GROUP}-3.8.23.yaml" \
     --backend api7ee \
     --server "https://{DASHBOARD_ADDR}" \
     --gateway-group "{GATEWAY_GROUP}"
   
   adc lint -f "api7ee-{GATEWAY_GROUP}-3.8.23.yaml"

9. Create a database-native backup and successfully restore it into an isolated database. Verify the restored source version and representative CP resources. This backup is authoritative because ADC does not include all CP resources, such as users, roles, API products, and audit data.

10. Save the source gateway configuration, custom plugin source, certificates, and all image tags in addition to the deployment artifacts recorded above.

11. Prepare the target images and deployment configuration, including the Developer Portal frontend, OpenAPI2MCP sidecar, file server, or other enabled components.

12. Rehearse the selected upgrade and rollback strategy in staging with representative authenticated traffic, cache behavior, rate limiting, logging, custom plugins, and Portal flows.

13. Start a CP write freeze before taking the final production backup. Block Dashboard, Admin API, Developer Portal, ADC, deployment automation, scheduled jobs, API usage services, direct database integrations, and every other path that can write the CP database. Repeat and lint the ADC exports after the freeze and record the freeze start time. Do not resume writes until the predefined acceptance criteria pass or rollback and reconciliation are complete.

ADC lint and diff detect stored gateway configuration differences. ADC does not verify runtime defaults, client authentication, custom code, or every CP resource, so complete all other checks in this guide before accepting the upgrade.

Upgrade the Control Plane

Choose the strategy that matches the production topology. Use the linked strategy pages for deployment-specific image, Helm, traffic-shifting, and node-replacement instructions. If guidance differs, follow the stricter safety requirement in this guide.

Strategy	Production impact	Database and rollback	Database scope
In-place	Existing DPs continue proxying, but the Dashboard and Admin API are unavailable while the CP is stopped. Requires less parallel capacity.	Migrates the existing CP database. Rollback requires restoring the immutable source backup into a new database.	External PostgreSQL 15.x. Contact API7 Support for the built-in chart database, MySQL, Microsoft SQL Server, or another database.
Dual-cluster	Requires capacity for two complete clusters and a rollback-relevant business-write freeze across both. Proxy traffic remains on the old DPs until traffic shifting begins.	Restores the source backup into a new target database. Rollback can steer traffic to the old cluster only while rollback-relevant business writes remain blocked or are reconciled.	Two independent external PostgreSQL 15.x databases with the Helm and Kubernetes procedure in this guide. Contact API7 Support for another database or topology.

Developer Portal deployments require an API7 Support-approved identity, ownership, database, domain, and rollback procedure before using either strategy.

In-Place Upgrade

The in-place strategy stops all source CP database clients before one target Dashboard replica migrates the database. Existing DPs continue proxying during this management-plane interruption.

Helm and Kubernetes

The Helm route begins with CP chart 0.17.37 and explicit v3.8.23 image tags. It upgrades to CP chart 3.10.8, which uses v3.10.6 images by default.

Prepare three values files from the chart 3.10.8 defaults. Set postgresql.builtin to false in every file so all target components use the external production database.

Values file	Purpose	Components to run
Migration	Apply the database migration with one writer.	One Dashboard. Stop DP Manager, Developer Portal, API usage, file server, and every other target database client.
Activation	Validate the migrated CP before restoring full capacity.	One Dashboard, one DP Manager, and only the optional components required for verification.
Final	Restore the production CP topology after validation.	The approved production replica counts and remaining optional components.

The migration values should include:

postgresql:
  builtin: false
dashboard:
  replicaCount: 1
dp_manager:
  replicaCount: 0
developer_portal:
  replicaCount: 0
api_usage:
  enable: false
file_server:
  enabled: false

1. Keep the write freeze in effect. Update the source Helm release with the source chart and values, setting every source CP database client to zero replicas. Wait until all 3.8.23 Dashboard, DP Manager, Developer Portal backend, API usage, and other database-writing pods have stopped. Also stop external writers and automation.

2. Take the final database backup and restore it into an isolated verification database. Do not continue until the restored source version and representative resources pass the go/no-go checks.

3. Record the external production database DSN used by the stopped source CP. Verify that dashboard_configuration.database.dsn in the migration values points to that exact database; the target chart defaults to its chart-local PostgreSQL service. Then upgrade the stopped CP release with the migration values. Do not add --atomic or --reuse-values: an automatic Helm rollback does not restore the migrated database, and reused source values can retain old images or omit target defaults.

   helm upgrade {CP_RELEASE} api7/api7ee3 \
     --namespace {CP_NAMESPACE} \
     --version 3.10.8 \
     --values api7ee-cp-3.10.6-migration-values.yaml \
     --wait \
     --timeout {TIMEOUT}

4. Watch the migration Dashboard pod and inspect its previous logs if it restarts. For this exact route, accept only one restart where the previous process ends with SQLSTATE 0A000 and cached plan must not change result type. Before starting another target component, confirm the replacement pod is Ready, reports 3.10.6, has no current migration error, and contains the expected services, routes, consumers, credentials, gateway groups, API products, IAM policies, and active license.

   A second restart, a different panic or migration error, a skipped-resource warning, a pod that does not become Ready, or missing migrated data is an abort condition. Preserve the logs and migrated database, keep writes frozen, and restore the immutable source backup into a new database.

5. Verify that every target CP component in the activation and final values uses the recorded production database DSN. Apply the activation values and verify the single Dashboard replica, one DP Manager replica, and required optional components. Then apply the final values to restore the remaining target replicas and components. Increase Dashboard replicas only after the migration replica is healthy; never start source and target Dashboard pods against the same database.

6. Sign in again and verify the Dashboard version, migrated resources, IAM policies, and every gateway group's compatibility report before upgrading the DPs.

For the in-place strategy, skip the dual-cluster section and continue with Upgrade the Data Plane.

Dual-Cluster Upgrade

The verified dual-cluster route keeps the source cluster running while the target cluster uses an independently restored database. Follow the dual-cluster strategy with these pair-specific requirements:

1. Keep the rollback-relevant business-write freeze in effect and leave the source CP, DPs, and database unchanged through the rollback window.
2. Take the final 3.8.23 database backup, restore it into a new external PostgreSQL 15.x database, and verify representative resources before starting the target CP.
3. Deploy CP chart 3.10.8 against the target database with one 3.10.6 Dashboard replica and every other target database client stopped. Apply the same bounded one-restart validation used by the in-place procedure. A second restart, a different error, a skipped-resource warning, readiness failure, or missing migrated data is an abort condition.
4. Validate the target version, migrated resources, IAM policies, active license, and database health. Apply the activation and final target values only after the migration gate passes.
5. Generate new DP certificates and values from the target Dashboard. Deploy gateway chart 3.10.13 and image 3.10.6 as a separate target gateway release outside the production load balancer.
6. Confirm every target DP reports Healthy and Compatible with zero errors, then run representative traffic directly through the target Service. Copied source DP records appear as LostConnection in the target Dashboard; evaluate the new target records separately. The copied records can be removed after they report Offline.
7. Shift traffic in the stages and observation periods defined by the production plan. Keep enough source capacity for rollback and keep writes frozen until acceptance or traffic rollback completes.

After completing the traffic move, skip the in-place DP rolling section and continue with Validate the Upgrade.

Upgrade the Data Plane

For the in-place CP strategy, the general Data Plane Rolling Upgrade explains the replacement mechanics. The version anchors, compatibility gates, and Helm settings below are authoritative for this pair.

Keep the immutable final 3.8.23 backup for the entire rollback window. If the rolling strategy or local procedure calls for another database backup after CP migration, store it as a separately named target-version snapshot. It does not replace the source backup required for full rollback.

Upgrade one gateway group at a time. The first target DP is a canary for the cumulative runtime and plugin changes; the remaining nodes follow only after that canary passes direct traffic checks.

Helm and Kubernetes

The verified Helm route applies to gateway Deployments. It upgrades gateway chart 0.2.40 with image tag 3.8.23 to gateway chart 3.10.13 with image tag 3.10.6. Contact API7 Support for a DaemonSet gateway.

1. Generate new target Helm values and a DP certificate from the target Dashboard for the gateway group. Reapply the source release's production resources, scheduling, service, autoscaling, resolver, and plugin settings to the target chart values after reviewing their target defaults.

2. Deploy one 3.10.6 DP as a separate canary Helm release. Set apisix.kind to Deployment, apisix.replicaCount to 1, and autoscaling.enabled to false. Give the canary its own Kubernetes Service and keep that Service out of the production load balancer.

3. Wait for the canary to report Healthy and Compatible, with no compatibility errors. Send representative public and authenticated requests directly through its Service and verify the target-DP behaviors in the compatibility checklist.

4. Before suspending autoscaling, record the Deployment's current desired and Ready replica counts. Set apisix.replicaCount to a verified safe capacity that is not lower than either count and satisfies the production load requirement. Then configure the main gateway release to add a target pod before making a source pod unavailable:

   apisix:
     kind: Deployment
     replicaCount: {SAFE_REPLICA_COUNT}
   updateStrategy:
     type: RollingUpdate
     rollingUpdate:
       maxSurge: 1
       maxUnavailable: 0
   autoscaling:
     enabled: false

5. Upgrade the main gateway release to chart 3.10.13 with image tag 3.10.6:

   helm upgrade {GATEWAY_RELEASE} api7/gateway \
     --namespace {GATEWAY_NAMESPACE} \
     --version 3.10.13 \
     --values api7-gateway-3.10.6-values.yaml \
     --wait \
     --timeout {TIMEOUT}

   Watch the pods and Gateway Instances throughout the rollout. Kubernetes should start and ready each target pod before terminating a source pod.

6. Confirm every main-release DP reports Healthy and Compatible and that representative production traffic passes. Remove the canary release only after the main rollout is accepted, then restore the production autoscaling policy.

Keep enough drained 3.8.23 DP capacity, manifests, certificates, and images available to restore traffic until the rollback deadline passes. Do not enable target-only plugins, fields, or secret references while any source DP remains.

Validate the Upgrade

Complete these checks before ending the configuration freeze:

1. Confirm every CP component is healthy and reports 3.10.6.

2. Confirm every production DP reports Healthy and Compatible, with no errors in the compatibility report. For dual-cluster, evaluate target DP records separately from copied source records that report LostConnection.

3. Run adc diff for every gateway group, using the matching pre-upgrade export:

   adc diff -f "api7ee-{GATEWAY_GROUP}-3.8.23.yaml" \
     --backend api7ee \
     --server "https://{DASHBOARD_ADDR}" \
     --gateway-group "{GATEWAY_GROUP}"

4. Verify migrated services, routes, upstreams, consumers, credentials, SSL resources, plugin metadata, global rules, API product links, IAM policies, and the active license.

5. If Developer Portal is enabled, verify registration or SSO and confirm each migrated developer can access and manage the expected applications, subscriptions, and credentials.

6. Run representative public and authenticated requests, including rate limiting, caching, logging, custom plugins, and large request bodies where applicable.

7. Review gateway and CP logs, metrics, database health, and resource utilization through the predefined observation duration. Compare them with the documented acceptance criteria and abort thresholds.

An empty ADC diff does not prove that runtime defaults, client authentication, custom plugins, or Portal identities are compatible. If an abort threshold is reached, keep writes frozen and execute the predefined rollback. Resume writes only after the named decision owner records that every acceptance criterion passed before the rollback deadline.

Roll Back

Choose the rollback path for the component and strategy that failed. Keep management writes frozen until the restored deployment and any required reconciliation are complete.

Roll Back Only the Data Plane

If a problem appears during DP replacement, first confirm that no target-only configuration was written. If none was written, remove the new nodes from the load balancer and return traffic to retained 3.8.23 nodes. The 3.10.6 CP can manage them temporarily as partially compatible DPs while you investigate.

For a Helm-managed DP, restore gateway chart 0.2.40, the saved source values, and image tag 3.8.23. Keep the explicit add-before-drain strategy. Validate source pods before allowing Kubernetes to remove the target pods.

If the API7 Support plan rewrote LDAP Auth user_dn values for the target DPs, restore the source plain-string values before returning LDAP traffic to 3.8.23 DPs.

If target-only configuration was written, do not return traffic to 3.8.23 DPs. Perform the full CP and database rollback below, or keep the affected nodes isolated and obtain API7 Support guidance.

Roll Back the Control Plane and Database

Use the pre-upgrade database backup instead of relying on legacy records in the migrated database.

For Helm and Kubernetes, do not run helm rollback by itself and do not rely on an earlier --atomic rollback. Helm restores Kubernetes objects, not the pre-migration PostgreSQL state. Stop the target CP first and restore the backup into a new database. Then restore CP chart 0.17.37, the saved source values, and the explicit v3.8.23 image tags against that database.

1. Keep the CP write freeze in effect. Stop every 3.10.6 CP process and other database writer. Network-isolate the management connections of the 3.10.6 DPs from both source and target CP endpoints.
2. If the target DPs are still proxying safely, keep them in the load balancer while rebuilding the source path. If continued target traffic is unsafe, remove them immediately and treat the resulting request interruption as an emergency outage until source DPs are ready.
3. Restore the immutable final 3.8.23 backup into a new database.
4. Restore the saved 3.8.23 CP configuration and image tags. Point the source CP components only to the restored database, using a management endpoint that the isolated 3.10.6 DPs cannot reach.
5. Start the 3.8.23 CP and the retained or redeployed 3.8.23 DPs. Verify the source version, active license, original Service Hub resources, and representative traffic directly on the source DPs.
6. Shift production traffic to the verified 3.8.23 DPs, then stop and isolate every 3.10.6 DP.
7. Compare the restored deployment with the pre-upgrade ADC exports and run representative traffic checks.
8. Keep the migrated database and target components isolated for investigation until the failed upgrade is understood.

Changes made after the final backup are not present in the restored database. Keep writes frozen until rollback verification and any required reconciliation are complete. If Developer Portal was enabled under an API7 Support-approved migration plan, also follow that plan for the separate frontend database, domain, and sessions.

Roll Back a Dual-Cluster Upgrade

Redirecting traffic to the old DPs is safe only when rollback-relevant business writes were blocked or the tested reconciliation procedure has completed. Target-only disposable operational state is abandoned with the target cluster. Otherwise, do not perform a traffic-only rollback; keep both clusters isolated as required by the incident plan and contact API7 Support. Restore the old database only if it was migrated, corrupted, or otherwise made unsuitable for the source cluster.