Skip to main content
Version: 1.4.0

Offline Resilience

Temporary control-plane connectivity loss separates the live traffic path from management workflows. A running AISIX gateway can keep serving from its latest accepted configuration while it reconnects, but it cannot receive new resources and some AISIX Cloud services remain connectivity-dependent.

Offline resilience protects traffic that depends on configuration already held by the gateway. It does not make the control plane optional or make external upstreams available during their own outages.

Continue Serving Accepted Configuration

After a gateway has applied valid projected configuration, it keeps that snapshot in memory and continues serving it while the configuration connection is unavailable. AISIX does not withdraw the gateway from traffic merely because no newer configuration event has arrived. Use configuration freshness as an operator alert instead of a load-balancer health condition.

No resource changes reach the gateway until connectivity returns. Changes that the control plane accepts during the outage wait for projection and do not affect the snapshot already in service.

Restart from Cached Configuration

On-disk snapshot caching is disabled by default whether the gateway receives resources from AISIX Cloud or reads them directly from etcd. A running gateway still retains its accepted configuration in memory while disconnected. To also recover that configuration across process restarts, explicitly enable persistence:

config.yaml
managed:
snapshot_cache_enabled: true
snapshot_cache_path: "/var/lib/aisix/config_cache.json"

managed.snapshot_cache_enabled: false disables cache reads, encoding, and writes even when a path is configured. When enabled, an omitted or null path uses /var/lib/aisix/config_cache.json; an empty path disables persistence. See AISIX Cloud startup configuration.

Upgrade behavior

Earlier versions enabled the cache by default in managed mode. To retain disk recovery after upgrading, set snapshot_cache_enabled: true; an existing snapshot_cache_path alone no longer enables it.

A restarted gateway can serve while reconnecting only when that cache is enabled and contains a valid snapshot. AISIX ignores a missing, unreadable, corrupt, or incompatible cache.

Without a usable snapshot, the gateway has nothing to serve until it connects and applies valid configuration. After any initial etcd connection completes, GET /status/ready on the metrics/status listener returns 503 while the gateway waits. A stalled credentialed dial occurs before AISIX creates any listener, so the metrics/status port is also refused during that earlier stage. /readyz returns nothing throughout: the proxy listener that carries it is not bound until the first configuration is applied. See Startup and the First Configuration.

When persistence is enabled, give each gateway instance its own writable state directory. The api7/aisix Helm chart uses a per-Pod emptyDir by default: it survives a container restart within the same Pod but is lost when that Pod is replaced. Recovery across Pod replacement requires storage that survives the replacement, with a separate cache location per replica. A replacement Pod without a usable cache must reconnect and obtain configuration before its proxy listener opens.

The snapshot includes provider credentials and other sensitive configuration without encryption. Restrict access to the cache directory and any backups as you would for the configuration store; Base64 encoding does not protect secrets.

Connectivity-Dependent Workflows

The gateway handles control-plane-dependent workflows separately:

WorkflowBehavior while disconnected
Resource projectionNew and updated resources do not reach the gateway. The latest accepted snapshot remains in service.
AISIX Cloud budgetsAISIX reuses the last decision for up to AISIX_DP_BUDGET_STALE_MAX_SECONDS, which defaults to 600. Without a cached decision, it denies the request. After the stale window, it applies the failure mode returned with the last decision. See Budget Availability and Caching.
AISIX Cloud usage telemetryFailed batches are dropped. AISIX does not persist, retry, or replay them after connectivity returns. Live requests are not failed or delayed to preserve telemetry.
External observability exportersThe gateway sends directly to each configured destination, so control-plane loss alone does not interrupt them. Export still depends on connectivity from the gateway to that destination.
Heartbeats and certificate rotationHeartbeat status stops updating. Certificate rotation requires control-plane connectivity; the gateway continues using its current certificate while it remains valid.

Detect and Recover

A running gateway that is serving a snapshot remains traffic-ready. Check management-path health separately:

  • GET /status/config reports source.connected: false and retains the applied revision and configuration hash.
  • GET /status/ready remains 200 after a valid configuration has been applied. It is 503 when a restarted gateway has no usable snapshot.
  • The aisix_config_* metrics expose connection state, applied configuration, and load failures for alerting.
  • AISIX Cloud gateway status becomes stale when heartbeats stop.

See Configuration Status for the complete status fields, metrics, and alert examples. Keep the metrics and status listener private because it is unauthenticated.

To recover, restore DNS, network, and TLS connectivity from the gateway to the AISIX Cloud control-plane endpoint. Confirm that heartbeats resume, compare the published and applied revisions, and send a caller-visible request through the recovered gateway. A successful live request confirms the traffic path, but it does not recover telemetry batches that were dropped during the outage.

Next Steps

Use High Availability for the broader failure-domain design. Use Resource Projection to verify that a recovered gateway applied the expected revision.