Health Checks

AISIX exposes separate health and status endpoints for process, traffic, configuration, and model state. The same endpoints apply to the open-source AISIX gateway and to AISIX gateways connected to AISIX Cloud. Use the endpoint that matches the condition you want to monitor. To verify the complete caller-to-provider path, send a test request through the gateway.

Question	Endpoint	Listener
Should this instance be restarted?	GET /livez	Proxy
Should this instance receive proxy traffic?	GET /readyz	Proxy
Has this instance applied any valid configuration?	GET /status/ready	Metrics/status
What configuration is this instance serving?	GET /status/config	Metrics/status
Is a model available for routing?	GET /status/models	Metrics/status

These endpoints do not require authentication. The metrics/status endpoints are available when Prometheus metrics are enabled, as they are by default. Keep that listener private to monitoring and operations systems.

Proxy Liveness

Use /livez for process liveness:

curl -i "http://127.0.0.1:3000/livez"

A healthy process returns 200 OK with an ok body.

A draining instance answers 200 as well. Liveness decides whether to restart the instance, and draining is deliberate work: an instance that has been told to shut down is finishing the requests it already accepted, and restarting it would kill exactly those. It is /readyz that reports the drain, so that traffic is withdrawn without the instance being replaced under its own in-flight work.

Append ?verbose=1 when investigating manually. Do not make an automated probe depend on the verbose response body.

Liveness is intentionally narrow. It does not prove that a model is available or that a provider request can succeed.

It does imply that a configuration has been loaded, but only as a side effect of when the listener carrying it exists. A gateway that reads its resources from etcd or from AISIX Cloud does not bind the proxy listener until it has applied its first configuration, so before that point /livez is not answered 503 — it is not answered at all. See Startup and the First Configuration.

Traffic Readiness

Use /readyz to decide whether an instance should receive traffic:

curl -i "http://127.0.0.1:3000/readyz"

It returns 503 Service Unavailable while the instance is draining. After a valid configuration is available, the gateway remains ready for as long as it can serve that configuration.

Before the first configuration is applied, /readyz behaves differently depending on the resource source, and on an etcd-backed or AISIX Cloud gateway the answer is not a 503: the proxy listener has not been bound yet, so the probe sees a refused connection. Use GET /status/ready on the metrics/status listener to observe that phase; see Startup and the First Configuration.

A control-plane or configuration-store interruption does not make a running gateway unready merely because no recent update arrived. A stalled source commonly affects every instance, so withdrawing them would remove the traffic path rather than shift traffic to a healthy peer. Monitor configuration freshness separately.

Append ?verbose=1 when investigating why an instance is not ready. Do not make an automated probe depend on the verbose response body.

For Kubernetes, point the liveness and readiness probes at /livez and /readyz on the proxy listener, and add a startupProbe on the same listener so that neither of the other two acts while the gateway is still reaching its configuration source. Startup and the First Configuration explains the budget that probe needs. Give the gateway enough termination time to drain in-flight and streaming requests.

Startup and the First Configuration

A gateway whose resources come from etcd or from AISIX Cloud binds the proxy listener only after it has applied its first configuration. This is unconditional and has no setting: an instance that has never applied a configuration has nothing to serve, and a traffic layer that reads "the port accepts" as "this instance is ready" would otherwise route callers to it and see every request rejected.

Two kinds of boot are unaffected and bind immediately, because they already hold a configuration by the time the listener is set up: a gateway configured from a resources file, which loads that file at startup and exits if it cannot, and an etcd-backed or AISIX Cloud gateway that restored a usable snapshot from its on-disk cache.

While a gateway waits, the listeners split:

Listener	During the wait
Proxy	Not bound. /livez, /readyz, and every caller-facing endpoint refuse the connection.
Metrics/status	Bound from process start, when Prometheus metrics are enabled. GET /status/ready answers 503 Service Unavailable with no configuration available, and GET /status/config reports state never_loaded.
Admin	Bound from process start, on an open-source gateway that enabled it. Its own /livez answers 200 and its own /readyz answers 503 throughout the wait.

So the metrics/status listener is where a cold start is observed. On a gateway connected to AISIX Cloud, which never binds an admin listener, GET /status/ready is the only health endpoint that answers at all, and the only way to distinguish a gateway still reaching its configuration source from one that is not running. An open-source gateway that bound an admin listener has that listener's /livez and /readyz as well — but only the metrics/status endpoints say why the wait is happening.

The address and any proxy.tls material are still checked at startup, before the wait begins, so an unusable address or unreadable certificate file remains a boot failure rather than a deferred one.

The gateway does not give up. While no configuration has been applied it neither exits nor binds a degraded listener, and it binds the listener the moment a read succeeds. A read that fails is retried on an exponential backoff that grows from one second to a sixty-second ceiling. It records the wait in its own log:

Message	Level	Written
waiting for the first configuration before binding the proxy listener	INFO	Once, when the wait starts.
proxy listener still not bound: no configuration has been applied yet	WARN	Every ten seconds, while the wait continues.
first configuration applied — binding the proxy listener	INFO	Once, when the listener binds.

Size a Kubernetes Startup Probe

A startupProbe on the proxy listener holds liveness and readiness off for the whole of boot, that wait included. Its budget is periodSeconds x failureThreshold, and it has to cover reaching the configuration source and applying what it holds — not merely starting the process. The api7/aisix Helm chart ships a 60-second budget for it.

Size that budget for reaching the source — DNS, TLS, and control-plane or etcd availability — rather than for the size of the configuration. Applying a configuration is not usually what consumes it; a 20,000-model configuration, whose single response from the configuration store exceeds 4 MiB, is covered by the gateway's own end-to-end tests. What does consume the budget is an unreachable or slow source.

A pod whose configuration source stays unreachable for longer than the budget has its container killed and restarted by the kubelet, and repeated restarts show as CrashLoopBackOff. That is the intended outcome, not a fault to work around: the instance has never had anything to serve. The restarted container resumes the same wait, and binds as soon as the source comes back. Diagnose it from GET /status/ready and the log lines above, not by pointing the probes somewhere else.

Shutdown and Draining

A load balancer learns that an instance is withdrawing on its next health check, not the moment the instance decides to. Between those two points it keeps routing new connections. Closing the listener as soon as the shutdown signal arrives would refuse every connection routed inside that interval. Callers then see gateway errors during an ordinary rolling update or scale-down.

So the gateway separates the two events. On SIGTERM or SIGINT it:

1. Answers /readyz with 503 Service Unavailable immediately, so the next health check withdraws it. /livez stays 200: the process is healthy and must not be restarted while it drains.
2. Keeps accepting new connections for at least shutdown.min_drain_secs, which defaults to 30 seconds.
3. Adds Connection: close to every HTTP/1.1 response, so a client that pools connections retires them as it uses them instead of holding idle ones open. HTTP/2 forbids that header, so an HTTP/2 client is sent a GOAWAY frame instead, at the moment the drain starts. GOAWAY asks the peer to finish the streams it has already opened and to start no new ones; it does not close the connection or interrupt anything in flight.
4. After the minimum window, waits without a deadline of its own for the in-flight count to reach zero, then stops accepting new connections and exits.

Set min_drain_secs above the detection latency of whatever load-balances the instance. A Kubernetes readiness probe needs periodSeconds x failureThreshold. An external load balancer needs its own check interval multiplied by its retry count. Setting it too low closes the listener while traffic is still arriving. Setting it too high only delays the exit.

config.yaml

shutdown:
  min_drain_secs: 30

The window is a minimum, not a deadline. After it elapses the gateway still waits for the in-flight count to reach zero, so a load balancer slower than configured cannot make it close under live traffic. 0 drops the window entirely and is only correct when nothing routes to the instance by health check.

Because the wait for in-flight work to reach zero is unbounded, the deployment platform sets the real limit on the whole sequence: terminationGracePeriodSeconds in Kubernetes, TimeoutStopSec under systemd. Size it above shutdown.min_drain_secs plus your longest request or stream, with operational margin. Add the duration of any preStop hook because it consumes the same Kubernetes termination budget.

note

Point the health check of an external load balancer at /readyz rather than at a bare TCP connect. A TCP check cannot observe readiness, so the only signal it ever receives is the listener closing — the very event the drain window exists to avoid.

Watch a Drain in the Logs

The drain logs its own progress, because the record a request normally leaves is written when the request completes — and a request still running when the platform's grace period expires never gets there. These lines are what separate work the gateway had already taken on from traffic still being routed at it after the signal:

Message	Fields	Written
draining — /readyz now reports 503, still accepting new connections	min_drain_secs, in_flight, open_connections	Once, when the signal arrives.
still draining in-flight requests	in_flight, open_connections	Periodically, while the drain runs.
request arrived while draining	method, path, plus the request's own request_id, peer, and downstream_request_id	Per request that reaches the gateway after the signal.
accepted a new downstream connection while draining	peer, open_connections	Per connection accepted after the signal.

The two counts answer different questions and neither is derivable from the other. in_flight counts requests being served, and a streaming response holds its slot for as long as bytes may still flow. open_connections counts downstream connections open on the proxy listener, including pooled ones sitting idle with no request on them. So a drain that will not end while in_flight stays up is the gateway finishing work it already accepted; one where in_flight has reached zero while open_connections stays up is clients holding connections they are not using; one where open_connections keeps climbing is something still routing new connections here.

Together, the two per-event lines explain a request that failed during a rolling update. An arrival line with a matching accept line for the same peer means the connection itself was routed here after the gateway had already asked to be withdrawn — a load balancer that has not caught up, which is the case min_drain_secs exists to absorb. An arrival line with no matching accept line means the connection predates the signal and the client reused it out of its pool, which is what the Connection: close header retires.

The accept line cannot tell your callers from your platform, though. /livez and /readyz are served on the proxy listener too, and probes keep arriving on their own connections throughout the drain, so they raise open_connections and produce accept lines like any other connection. The arrival line is where that judgement is made: it knows the path, and deliberately excludes those two endpoints so the handful of real arrivals is not buried under a probe every few seconds. Read the accept line for what the arrival line cannot see — a connection opened and never used.

peer and downstream_request_id are the connection identity described in Metrics and Logs. Matching them against the records of whatever fronts the gateway is how a gateway line and a balancer line are shown to be about the same connection.

Configuration State

The metrics/status listener provides two configuration checks.

GET /status/ready is a configuration-only startup gate:

• 503 Service Unavailable before the first valid configuration is applied;
• 200 OK after a valid configuration is available, including while AISIX serves a last-known-good snapshot after a later update fails.

On an etcd-backed or AISIX Cloud gateway this is the endpoint to watch before the first configuration is applied, because the proxy listener is not bound until then. See Startup and the First Configuration.

GET /status/config explains what AISIX observed and applied:

curl -sS "http://127.0.0.1:9090/status/config"

Use it when a resource update does not appear in proxy behavior. Compare the source and applied state, then inspect rejected resources and the latest load failure. See Configuration Status for the complete response fields, state meanings, Prometheus metrics, and alert examples.

Configuration status does not replace caller-facing verification. After the expected snapshot applies, query GET /v1/models and send the request whose behavior changed. See Configuration Propagation.

Per-Model Runtime Health

Use GET /status/models when the gateway is ready but routing avoids a model or reports no eligible target:

curl -sS "http://127.0.0.1:9090/status/models"

Each configured model reports one of these high-level states:

• healthy: available for routing;
• cooldown: temporarily removed after recent upstream failures; only a model that enables cooldown enters this state;
• unhealthy: excluded after failed background model checks;
• not_applicable: a virtual model whose availability derives from its targets.

The status view helps identify cooldown and background-check failures, but it does not validate caller access or provider credentials. A model can report healthy while a caller API key, provider key, or upstream response still prevents a request from succeeding.

See Configuration Status for the full response fields.

Combined Health Signals

Work from the earliest failing layer:

Signal	Next Check
/livez or /readyz refuses the connection.	Whether the proxy listener is bound at all. On an etcd-backed or AISIX Cloud gateway it is not, until the first configuration is applied; check /status/ready and the configuration source.
/livez fails.	Process state, listener binding, and listener TLS
/readyz fails.	Drain state
/status/ready fails.	Initial configuration source and load errors
/status/config is degraded or out_of_sync.	Rejected resources, source connectivity, and last_failure
/status/models reports cooldown or unhealthy.	Provider credential, provider availability, model checks, and outbound network
Every health endpoint succeeds but the request fails.	Caller access, provider path, policy enforcement, and upstream response

Finish with the same path the application uses:

AISIX_API_KEY="YOUR_CALLER_API_KEY"

curl -sS "http://127.0.0.1:3000/v1/models" \
  -H "Authorization: Bearer ${AISIX_API_KEY}"

Then send a real request through the required endpoint and model. That final probe verifies conditions that runtime health endpoints deliberately do not evaluate.

Next Steps

Use Troubleshooting to narrow a failed health or request-path check to configuration, caller policy, AISIX Cloud projection, or the upstream provider.