Health Checks

AISIX exposes separate health and status endpoints for process, traffic, configuration, and model state. The same endpoints apply to the open-source AISIX gateway and to AISIX gateways connected to AISIX Cloud. Use the endpoint that matches the condition you want to monitor. To verify the complete caller-to-provider path, send a test request through the gateway.

Question	Endpoint	Listener
Should this instance be restarted?	GET /livez	Proxy
Should this instance receive proxy traffic?	GET /readyz	Proxy
Has this instance applied any valid configuration?	GET /status/ready	Metrics/status
What configuration is this instance serving?	GET /status/config	Metrics/status
Is a model available for routing?	GET /status/models	Metrics/status

These endpoints do not require authentication. The metrics/status endpoints are available when Prometheus metrics are enabled, as they are by default. Keep that listener private to monitoring and operations systems.

Proxy Liveness

Use /livez for process liveness:

curl -i "http://127.0.0.1:3000/livez"

A healthy process returns 200 OK with an ok body.

A draining instance answers 200 as well. Liveness decides whether to restart the instance, and draining is deliberate work: an instance that has been told to shut down is finishing the requests it already accepted, and restarting it would kill exactly those. It is /readyz that reports the drain, so that traffic is withdrawn without the instance being replaced under its own in-flight work.

Append ?verbose=1 when investigating manually. Do not make an automated probe depend on the verbose response body.

Liveness is intentionally narrow. It does not prove that a model is available or that a provider request can succeed.

It does imply that a configuration has been loaded, but only as a side effect of when the listener carrying it exists. A gateway that reads its resources from etcd or from AISIX Cloud does not bind the proxy listener until it has applied its first configuration, so before that point /livez is not answered 503 — it is not answered at all. See Startup and the First Configuration.

Traffic Readiness

Use /readyz to decide whether an instance should receive traffic:

curl -i "http://127.0.0.1:3000/readyz"

It returns 503 Service Unavailable while the instance is draining. After a valid configuration is available, the gateway remains ready for as long as it can serve that configuration.

Before the first configuration is applied, /readyz behaves differently depending on the resource source, and on an etcd-backed or AISIX Cloud gateway the answer is not a 503: the proxy listener has not been bound yet, so the probe sees a refused connection. Use GET /status/ready on the metrics/status listener to observe that phase; see Startup and the First Configuration.

A control-plane or configuration-store interruption does not make a running gateway unready merely because no recent update arrived. A stalled source commonly affects every instance, so withdrawing them would remove the traffic path rather than shift traffic to a healthy peer. Monitor configuration freshness separately.

Append ?verbose=1 when investigating why an instance is not ready. Do not make an automated probe depend on the verbose response body.

For Kubernetes, point the liveness and readiness probes at /livez and /readyz on the proxy listener, and add a startupProbe on the same listener so that neither of the other two acts while the gateway is still reaching its configuration source. Startup and the First Configuration explains the budget that probe needs. Give the gateway enough termination time to drain in-flight and streaming requests.

Startup and the First Configuration

A gateway that reads resources from etcd or AISIX Cloud does not bind the proxy listener until it has applied its first configuration. This prevents traffic from reaching an instance that has no resources to serve. After any initial etcd connection completes, the listeners behave as follows:

Listener	Observable State
Proxy	Not bound. /livez, /readyz, and every caller-facing endpoint refuse the connection.
Metrics/status	Bound when Prometheus metrics are enabled. GET /status/ready answers 503 Service Unavailable with no configuration available, and GET /status/config reports state never_loaded.
Admin	Bound during the wait, on an open-source gateway that enabled it. Its own /livez answers 200 and its own /readyz answers 503 throughout the wait.

A credentialed initial etcd dial happens before AISIX creates any listener. If that dial stalls, none of these endpoints is available. That window is bounded: boot dials two providers one after the other, so it lasts at most etcd.dial_timeout_ms multiplied by the number of endpoints, multiplied by two. Only a deployment that sets etcd.user reaches this case, because without credentials the dial performs no I/O.

A ratelimit.redis or a cache.redis the gateway cannot reach does not delay any listener. Both backends bind and serve in a degraded mode, and attach in the background.

A resources-file gateway binds immediately after loading the file. An etcd-backed or AISIX Cloud gateway also binds immediately when it restores a usable on-disk snapshot.

The metrics/status listener is therefore the primary cold-start signal. If it is unavailable, see The Gateway Runs but the Proxy Port Refuses Connections for the symptoms, log messages, and corrective actions. The exact etcd timeout scope and defaults are documented under etcd Configuration Store.

Size a Kubernetes Startup Probe

A startupProbe on the proxy listener prevents liveness and readiness checks from acting before the first configuration is available. Its budget, periodSeconds x failureThreshold, must cover both reaching the configuration source and applying its resources. The api7/aisix Helm chart uses a 300-second budget: periodSeconds: 2 and failureThreshold: 150.

That budget provides headroom for a large configuration; it is not a measured upper bound on startup time.

The gateway retries failed reads after approximately 0, 1, 3, 7, 15, 31, and 63 seconds, then once per minute. Avoid ending the probe budget immediately before a retry. If the source remains unavailable past the budget, Kubernetes restarts the container and can report CrashLoopBackOff. Diagnose the source instead of pointing the probe at another listener.

Shutdown and Draining

AISIX separates becoming unready from closing the listener so that a load balancer has time to withdraw the instance. On SIGTERM or SIGINT, the gateway:

1. Answers /readyz with 503 Service Unavailable immediately, so the next health check withdraws it. /livez stays 200: the process is healthy and must not be restarted while it drains.
2. Keeps accepting new connections for at least shutdown.min_drain_secs, which defaults to 30 seconds.
3. Adds Connection: close to HTTP/1.1 responses and sends HTTP/2 clients a GOAWAY frame, so clients stop reusing the connection without interrupting in-flight work.
4. After the minimum window, waits without a deadline of its own for the in-flight count to reach zero, then stops accepting new connections.
5. On an etcd-backed or AISIX Cloud gateway that keeps a snapshot cache, waits up to a further five seconds for a cache write still in flight to reach disk, then exits.

Set min_drain_secs above the time the load balancer needs to stop routing after /readyz changes. For Kubernetes, this is periodSeconds x failureThreshold; for an external load balancer, use its check interval multiplied by its retry count.

config.yaml

shutdown:
  min_drain_secs: 30

The window is a minimum, not a shutdown deadline. After it elapses, the gateway still waits for the in-flight count to reach zero. Setting it to 0 drops the minimum window and is appropriate only when no health-check-driven system routes traffic to the instance.

Because the wait for in-flight work to reach zero is unbounded, the deployment platform sets the real limit on the whole sequence: terminationGracePeriodSeconds in Kubernetes, TimeoutStopSec under systemd. Size it above shutdown.min_drain_secs, plus your longest request or stream, plus the five-second snapshot-cache drain when persistence is enabled, with operational margin. Add the duration of any preStop hook because it consumes the same Kubernetes termination budget.

The snapshot-cache allowance is additional time because that wait starts only after in-flight requests finish. The five-second limit is fixed. If it expires, the gateway logs a warning and exits; atomic file replacement prevents a partial snapshot, but the next process can restore an older snapshot or no snapshot.

note

Point the health check of an external load balancer at /readyz rather than at a bare TCP connect. A TCP check cannot observe readiness, so the only signal it ever receives is the listener closing — the very event the drain window exists to avoid.

Watch a Drain in the Logs

Use the drain log fields to distinguish unfinished requests from connections that remain open or continue arriving:

Message	Fields	Written
draining — /readyz now reports 503, still accepting new connections	min_drain_secs, in_flight, open_connections	Once, when the signal arrives.
still draining in-flight requests	in_flight, open_connections	Periodically, while the drain runs.
request arrived while draining	method, path, plus the request's own request_id, peer, and downstream_request_id	Per request that reaches the gateway after the signal.
accepted a new downstream connection while draining	peer, open_connections	Per connection accepted after the signal.

in_flight counts active requests, including streaming responses. open_connections also includes idle pooled connections and health-probe connections. If in_flight remains above zero, AISIX is still finishing accepted work. If open_connections grows, another component is still opening connections to the draining instance.

For a failed request during a rollout, match request arrived while draining with accepted a new downstream connection while draining by peer. A matching accept record means the connection arrived after draining began; without one, the client reused an older pooled connection. Health checks can produce accept records but are excluded from the request-arrival record.

peer and downstream_request_id provide the arrival context described in Access Logs and Request Correlation. Match peer to the fronting proxy's connection record and downstream_request_id to its request record.

Configuration State

The metrics/status listener provides two configuration checks.

GET /status/ready is a configuration-only startup gate:

• 503 Service Unavailable before the first valid configuration is applied;
• 200 OK after a valid configuration is available, including while AISIX serves a last-known-good snapshot after a later update fails.

On an etcd-backed or AISIX Cloud gateway this is the endpoint to watch before the first configuration is applied, because the proxy listener is not bound until then. See Startup and the First Configuration.

GET /status/config explains what AISIX observed and applied:

curl -sS "http://127.0.0.1:9090/status/config"

Use it when a resource update does not appear in proxy behavior. Compare the source and applied state, then inspect rejected resources and the latest load failure. See Configuration Status for the complete response fields, state meanings, Prometheus metrics, and alert examples.

Configuration status does not replace caller-facing verification. After the expected snapshot applies, query GET /v1/models and send the request whose behavior changed. See Configuration Propagation.

Per-Model Runtime Health

Use GET /status/models when the gateway is ready but routing avoids a model or reports no eligible target:

curl -sS "http://127.0.0.1:9090/status/models"

Each configured model reports one of these high-level states:

• healthy: available for routing;
• cooldown: temporarily removed after recent upstream failures; only a model that enables cooldown enters this state;
• unhealthy: excluded after failed background model checks;
• not_applicable: a virtual model whose availability derives from its targets.

The status view helps identify cooldown and background-check failures, but it does not validate caller access or provider credentials. A model can report healthy while a caller API key, provider key, or upstream response still prevents a request from succeeding.

See Configuration Status for the full response fields.

Combined Health Signals

Work from the earliest failing layer:

Signal	Next Check
/livez or /readyz refuses the connection.	Whether the proxy listener is bound at all. On an etcd-backed or AISIX Cloud gateway it is not, until the first configuration is applied; check /status/ready and the configuration source.
/livez fails.	Process state, listener binding, and listener TLS
/readyz fails.	Drain state
/status/ready fails.	Initial configuration source and load errors
/status/config is degraded or out_of_sync.	Rejected resources, source connectivity, and last_failure
/status/models reports cooldown or unhealthy.	Provider credential, provider availability, model checks, and outbound network
Every health endpoint succeeds but the request fails.	Caller access, provider path, policy enforcement, and upstream response

Finish with the same path the application uses:

AISIX_API_KEY="YOUR_CALLER_API_KEY"

curl -sS "http://127.0.0.1:3000/v1/models" \
  -H "Authorization: Bearer ${AISIX_API_KEY}"

Then send a real request through the required endpoint and model. That final probe verifies conditions that runtime health endpoints deliberately do not evaluate.

Next Steps

Use Troubleshooting to narrow a failed health or request-path check to configuration, caller policy, AISIX Cloud projection, or the upstream provider.