# Configure Upstream Health Checks Health checking determines whether upstream services are healthy or unhealthy based on their responsiveness. APISIX excludes unhealthy nodes while at least one healthy node remains. If all nodes are unhealthy, APISIX falls back to the original node set so that a reachable node can still serve traffic. There are two general approaches to health check: * **Active checks**: APISIX proactively and periodically sends requests to upstream services and determines the health of those based on the responses to these requests. * **Passive checks**: APISIX determines the health of upstream services based on how they respond to client requests, without proactively probing. This guide will show you how to configure both active and passive health checks for your upstream services. ## Prerequisite(s)[​](#prerequisites "Direct link to Prerequisite(s)") * Install [Docker](https://docs.docker.com/get-docker). * Install [cURL](https://curl.se/) to send requests to the services for validation. * Follow the [Getting Started tutorial](https://docs.api7.ai/apisix/getting-started.md) to start a new APISIX instance in Docker or on Kubernetes. * If you use the Admin API examples and authentication is enabled, export an admin-role key as `ADMIN_API_KEY`. Configure the key under `deployment.admin.admin_key`. The Docker quickstart disables authentication for local evaluation; keep it enabled in production. ## Start Sample Upstream Services[​](#start-sample-upstream-services "Direct link to Start Sample Upstream Services") Start two NGINX instances as sample upstream services in the same Docker network as APISIX: ``` DOCKER_NETWORK=apisix-quickstart-net docker run -d -p 8080:80 --network=${DOCKER_NETWORK} --name nginx1 nginx docker run -d -p 8081:80 --network=${DOCKER_NETWORK} --name nginx2 nginx ``` Verify both NGINX instances are running: ``` for port in 8080 8081; do curl -s "http://127.0.0.1:$port" | grep -q "Welcome to nginx" && echo "NGINX welcome page available on port $port." done ``` You should see the following response: ``` NGINX welcome page available on port 8080. NGINX welcome page available on port 8081. ``` ## Configure Active Health Checks[​](#configure-active-health-checks "Direct link to Configure Active Health Checks") Active checks determine the health of upstream services by periodically sending requests, or probes, to the services and seeing how they respond. The examples below show how to: * send a custom method and body in an active probe * detect changes in upstream status with active checks * observe how APISIX forwards requests when all upstream statuses are unhealthy Active HTTP and HTTPS checks use `GET` with an empty body by default. Set `http_method` to any supported HTTP method and use `http_req_body` when the health endpoint expects a request payload. A nonempty body adds the corresponding `Content-Length` header automatically. ### Send a POST Health Check[​](#send-a-post-health-check "Direct link to Send a POST Health Check") Configure the first sample NGINX service with a health endpoint that accepts only `POST` and logs the probe method, path, and content length: ``` docker exec nginx1 /bin/sh -c 'cat > /etc/nginx/conf.d/default.conf <<"EOF" log_format health_probe "$request_method $uri $content_length"; server { listen 80; access_log /tmp/health-probe.log health_probe; location = /health { if ($request_method != POST) { return 405; } return 200 "healthy\n"; } location = /404 { return 404; } location / { return 200 "Welcome to nginx!\n"; } } EOF nginx -s reload' ``` Create a route with a POST active check and a JSON body: ``` curl "http://127.0.0.1:9180/apisix/admin/routes/post-health-check" -X PUT \ -H "X-API-KEY: ${ADMIN_API_KEY}" \ -d '{ "uri": "/post-health-check", "upstream": { "type": "roundrobin", "nodes": { "nginx1:80": 1 }, "checks": { "active": { "type": "http", "http_method": "POST", "http_path": "/health", "http_req_body": "{\"status\":\"check\"}", "req_headers": [ "Content-Type: application/json" ], "healthy": { "interval": 2, "successes": 1 }, "unhealthy": { "interval": 1, "http_failures": 1 } } } } }' ``` Send a request through the route to start its health checker, wait for a probe, and inspect the NGINX access log: ``` curl -i "http://127.0.0.1:9080/post-health-check" sleep 2 docker exec nginx1 /bin/sh -c "grep 'POST /health' /tmp/health-probe.log | tail -n 1" ``` The route request should return `200 OK`. The log should confirm that the active probe used POST and sent the 18-byte body: ``` POST /health 18 ``` ### Example: Status Change in Upstream Services[​](#example-status-change-in-upstream-services "Direct link to Example: Status Change in Upstream Services") The following example demonstrates how APISIX active health checks respond in situations where healthy upstream services have become: partially unavailable, all unavailable, and all recovered. Create a route to the two services and configure active health checks that run every 2 seconds: * Admin API * ADC ``` curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \ -H "X-API-KEY: ${ADMIN_API_KEY}" \ -d '{ "id": "example-hc-route", "uri":"/", "upstream": { "type":"roundrobin", "nodes": { "nginx1:80": 1, "nginx2:80": 1 }, "checks": { "active": { "type": "http", "http_path": "/", "healthy": { "interval": 2, "successes": 1 }, "unhealthy": { "interval": 1, "timeouts": 3 } } } } }' ``` ❶ `type`: the type of active health checks. ❷ `http_path`: the HTTP request path to actively probe. ❸ `healthy.interval`: the time interval in seconds for periodically checking healthy nodes. ❹ `healthy.successes`: the success count threshold for ruling if an upstream node is considered healthy. ❺ `unhealthy.interval`: the time interval in seconds for periodically checking unhealthy nodes. ❻ `unhealthy.timeouts`: the timeout count threshold for ruling if an upstream node is considered unhealthy. adc.yaml ``` services: - name: Nginx Service labels: docs-example: upstream-health-check routes: - uris: - / name: example-hc-route upstream: type: roundrobin nodes: - host: nginx1 port: 80 weight: 1 - host: nginx2 port: 80 weight: 1 checks: active: type: http http_path: / healthy: interval: 2 successes: 1 unhealthy: interval: 1 timeouts: 3 ``` ❶ `type`: the type of active health checks. ❷ `http_path`: the HTTP request path to actively probe. ❸ `healthy.interval`: the time interval in seconds for periodically checking healthy nodes. ❹ `healthy.successes`: the success count threshold for ruling if an upstream node is considered healthy. ❺ `unhealthy.interval`: the time interval in seconds for periodically checking unhealthy nodes. ❻ `unhealthy.timeouts`: the timeout count threshold for ruling if an upstream node is considered unhealthy. ADC reconciles services as desired state. The label selector limits this guide to its own service throughout the health-check scenarios. Preview the scoped changes and confirm that they contain no unintended updates or deletions: ``` adc diff -f adc.yaml \ --include-resource-type service \ --label-selector docs-example=upstream-health-check ``` Synchronize the reviewed service configuration: ``` adc sync -f adc.yaml \ --include-resource-type service \ --label-selector docs-example=upstream-health-check ``` ### Verify[​](#verify "Direct link to Verify") You will be verifying the above configurations to understand how APISIX upstream health checks respond in different scenarios: * when [all upstream services are healthy](#verify-both-upstream-services-are-healthy) * when [only partial services are healthy](#verify-when-one-upstream-service-is-unavailable) * when [none of the services is healthy](#verify-both-upstream-services-are-unavailable) * when [all services are recovered](#verify-both-upstream-services-are-recovered) If you started APISIX with the Docker [Getting Started quickstart](https://docs.api7.ai/apisix/getting-started.md#get-apisix), the Control API is available on host port `9090`. #### Verify Both Upstream Services Are Healthy[​](#verify-both-upstream-services-are-healthy "Direct link to Verify Both Upstream Services Are Healthy") Send a request to the route to start health checks: ``` curl "http://127.0.0.1:9080/" ``` To see upstream health statuses, send a request to the health check endpoint in [Control API](https://docs.api7.ai/apisix/reference/control-api.md): ``` curl "http://127.0.0.1:9090/v1/healthcheck" ``` You should see a response similar to the following: ``` [ { "name": "/apisix/routes/example-hc-route", "type": "http", "nodes": [ { "port": 80, "counter": { "http_failure": 0, "tcp_failure": 0, "timeout_failure": 0, "success": 0 }, "ip": "172.24.0.5", "status": "healthy" }, { "port": 80, "counter": { "http_failure": 0, "tcp_failure": 0, "timeout_failure": 0, "success": 0 }, "ip": "172.24.0.4", "status": "healthy" } ] } ] ``` #### Verify When One Upstream Service Is Unavailable[​](#verify-when-one-upstream-service-is-unavailable "Direct link to Verify When One Upstream Service Is Unavailable") Make one upstream service temporarily unavailable to verify if APISIX reports one of the upstream services unhealthy: ``` docker container stop nginx1 ``` Wait for a few seconds and send a request to the health check endpoint: ``` curl "http://127.0.0.1:9090/v1/healthcheck" ``` You should see a response similar to the following, showing one of the upstream nodes has 3 timeout failures and marked unhealthy: ``` [ { "name": "/apisix/routes/example-hc-route", "type": "http", "nodes": [ { "port": 80, "counter": { "http_failure": 0, "tcp_failure": 0, "timeout_failure": 0, "success": 0 }, "ip": "172.24.0.5", "status": "healthy" }, { "port": 80, "counter": { "http_failure": 0, "tcp_failure": 0, "timeout_failure": 3, "success": 0 }, "ip": "172.24.0.4", "status": "unhealthy" } ] } ] ``` Send a request to the route to see if APISIX forwards the request to the other healthy node: ``` curl -i "http://127.0.0.1:9080/" ``` You should receive an `HTTP/1.1 200 OK` response. #### Verify Both Upstream Services Are Unavailable[​](#verify-both-upstream-services-are-unavailable "Direct link to Verify Both Upstream Services Are Unavailable") Make the other upstream service temporarily unavailable to verify if APISIX reports both upstream services unhealthy: ``` docker container stop nginx2 ``` Wait for a few seconds and send a request to the health check endpoint: ``` curl "http://127.0.0.1:9090/v1/healthcheck" ``` You should see a response similar to the following, showing both upstream nodes have 3 timeout failures and marked unhealthy: ``` [ { "name": "/apisix/routes/example-hc-route", "type": "http", "nodes": [ { "port": 80, "counter": { "http_failure": 0, "tcp_failure": 0, "timeout_failure": 3, "success": 0 }, "ip": "172.24.0.5", "status": "unhealthy" }, { "port": 80, "counter": { "http_failure": 0, "tcp_failure": 0, "timeout_failure": 3, "success": 0 }, "ip": "172.24.0.4", "status": "unhealthy" } ] } ] ``` Send a request to the route: ``` curl -i "http://127.0.0.1:9080/" ``` You should receive an `HTTP/1.1 502 Bad Gateway` response. When every node is unhealthy, APISIX falls back to the original node set. In this example, both fallback nodes are stopped and unreachable, so the request returns `502 Bad Gateway`. #### Verify Both Upstream Services Are Recovered[​](#verify-both-upstream-services-are-recovered "Direct link to Verify Both Upstream Services Are Recovered") Make both services available again to verify if APISIX reports both upstream services healthy: ``` docker container start nginx1 nginx2 ``` Wait for a few seconds and send a request to the health check endpoint: ``` curl "http://127.0.0.1:9090/v1/healthcheck" ``` You should see a response showing both upstream nodes are healthy, similar to [when both services are healthy at the start](#verify-both-upstream-services-are-healthy). ### Example: Forward Requests When Statuses Are Unhealthy[​](#example-forward-requests-when-statuses-are-unhealthy "Direct link to Example: Forward Requests When Statuses Are Unhealthy") The following example demonstrates that APISIX would still forward client requests to upstream services even when all upstream health statuses are unhealthy. Create a route to the two services and configure active health checks that run every 2 seconds: * Admin API * ADC ``` curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \ -H "X-API-KEY: ${ADMIN_API_KEY}" \ -d '{ "id": "example-hc-route", "uri":"/", "upstream": { "type":"roundrobin", "nodes": { "nginx1:80": 1, "nginx2:80": 1 }, "checks": { "active": { "type": "http", "http_path": "/404", "healthy": { "interval": 2, "successes": 1 }, "unhealthy": { "interval": 1, "http_failures": 2 } } } } }' ``` ❶ `type`: the type of active health checks. ❷ `http_path`: the HTTP request path to actively probe. For the convenience of demonstration, this is set to `/404`, which is a path that does not exist in upstream services. Consequently, both services should always be considered unhealthy by the active health checks. ❸ `unhealthy.http_failures`: the HTTP failure count threshold for ruling if an upstream node is considered unhealthy. adc.yaml ``` services: - name: Nginx Service labels: docs-example: upstream-health-check routes: - uris: - / name: example-hc-route upstream: type: roundrobin nodes: - host: nginx1 port: 80 weight: 1 - host: nginx2 port: 80 weight: 1 checks: active: type: http http_path: /404 healthy: interval: 2 successes: 1 unhealthy: interval: 1 http_failures: 2 ``` ❶ `type`: the type of active health checks. ❷ `http_path`: the HTTP request path to actively probe. For the convenience of demonstration, this is set to `/404`, which is a path that does not exist in upstream services. Consequently, both services should always be considered unhealthy by the active health checks. ❸ `unhealthy.http_failures`: the HTTP failure count threshold for ruling if an upstream node is considered unhealthy. Preview the changes within the same example-owned service scope: ``` adc diff -f adc.yaml \ --include-resource-type service \ --label-selector docs-example=upstream-health-check ``` Synchronize the reviewed service configuration: ``` adc sync -f adc.yaml \ --include-resource-type service \ --label-selector docs-example=upstream-health-check ``` ### Verify[​](#verify-1 "Direct link to Verify") If you started APISIX with the Docker [Getting Started quickstart](https://docs.api7.ai/apisix/getting-started.md#get-apisix), the Control API is available on host port `9090`. Send a request to the route to start health checks: ``` curl -i "http://127.0.0.1:9080/" ``` You should receive an `HTTP/1.1 200 OK` response. Send a request to the health check endpoint: ``` curl "http://127.0.0.1:9090/v1/healthcheck" ``` You should see a response similar to the following: ``` [ { "name": "/apisix/routes/example-hc-route", "nodes": [ { "counter": { "timeout_failure": 0, "http_failure": 2, "success": 0, "tcp_failure": 0 }, "port": 80, "ip": "172.25.0.5", "status": "unhealthy" }, { "counter": { "timeout_failure": 0, "http_failure": 2, "success": 0, "tcp_failure": 0 }, "port": 80, "ip": "172.25.0.4", "status": "unhealthy" } ], "type": "http" } ] ``` Send a request to the route to see if APISIX still forwards the request: ``` curl -i "http://127.0.0.1:9080/" ``` You should receive an `HTTP/1.1 200 OK` response. When no healthy node remains, APISIX falls back to the original node set. Both NGINX services are reachable in this example, so one of them can still serve the request despite being marked unhealthy. ## Configure Passive Health Checks[​](#configure-passive-health-checks "Direct link to Configure Passive Health Checks") APISIX requires the use of active health checks with passive health checks. When an upstream service becomes unhealthy, the active health check is in place to periodically check if the upstream service has recovered. ### Example: Status Change in Upstream Services[​](#example-status-change-in-upstream-services-1 "Direct link to Example: Status Change in Upstream Services") Create a route to the two services, and configure both active and passive health checks: * Admin API * ADC ``` curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \ -H "X-API-KEY: ${ADMIN_API_KEY}" \ -d '{ "id": "example-hc-route", "uri": "/404", "upstream": { "type": "roundrobin", "nodes": { "nginx1:80": 1, "nginx2:80": 1 }, "checks": { "active": { "type": "http", "http_path": "/", "healthy": { "interval": 99999, "successes": 1 }, "unhealthy": { "interval": 30 } }, "passive": { "healthy": { "http_statuses": [200,201,202,300,301,302], "successes": 1 }, "unhealthy": { "http_statuses": [429,404,500,501,502,503,504,505], "http_failures": 3 } } } } }' ``` ❶ `uri`: the URI path that the route matches. For the convenience of demonstration, this is set to `/404`, which is a path that does not exist in upstream services. Consequently, when a request is made, both upstream services should respond with a `404` status code. ❷ `active.healthy.interval`: the time interval in seconds for periodically checking healthy nodes. ❸ `active.unhealthy.interval`: the time interval in seconds for periodically checking unhealthy nodes. ❹ `passive.healthy.http_statuses`: the response HTTP status codes that are considered healthy. ❺ `passive.unhealthy.http_statuses`: the response HTTP status codes that are considered unhealthy. The unhealthy responses are counted towards the `http_failures`. ❻ `passive.unhealthy.http_failures`: the HTTP failure count threshold for ruling if an upstream node is considered unhealthy. adc.yaml ``` services: - name: Nginx Service labels: docs-example: upstream-health-check routes: - uris: - /404 name: example-hc-route upstream: type: roundrobin nodes: - host: nginx1 port: 80 weight: 1 - host: nginx2 port: 80 weight: 1 checks: active: type: http http_path: / healthy: interval: 99999 successes: 1 unhealthy: interval: 30 passive: healthy: http_statuses: - 200 - 201 - 202 - 300 - 301 - 302 successes: 1 unhealthy: http_statuses: - 429 - 404 - 500 - 501 - 502 - 503 - 504 - 505 http_failures: 3 ``` ❶ `uris`: the URI paths that the route matches. For the convenience of demonstration, this is set to `/404`, which is a path that does not exist in upstream services. Consequently, when a request is made, both upstream services should respond with a `404` status code. ❷ `active.healthy.interval`: the time interval in seconds for periodically checking healthy nodes. ❸ `active.unhealthy.interval`: the time interval in seconds for periodically checking unhealthy nodes. ❹ `passive.healthy.http_statuses`: the response HTTP status codes that are considered healthy. ❺ `passive.unhealthy.http_statuses`: the response HTTP status codes that are considered unhealthy. The unhealthy responses are counted towards the `http_failures`. ❻ `passive.unhealthy.http_failures`: the HTTP failure count threshold for ruling if an upstream node is considered unhealthy. Preview the changes within the same example-owned service scope: ``` adc diff -f adc.yaml \ --include-resource-type service \ --label-selector docs-example=upstream-health-check ``` Synchronize the reviewed service configuration: ``` adc sync -f adc.yaml \ --include-resource-type service \ --label-selector docs-example=upstream-health-check ``` ### Verify[​](#verify-2 "Direct link to Verify") If you started APISIX with the Docker [Getting Started quickstart](https://docs.api7.ai/apisix/getting-started.md#get-apisix), the Control API is available on host port `9090`. Send a request to the route to start health checks: ``` curl -i "http://127.0.0.1:9080/404" ``` You should see an `HTTP/1.1 404 Not Found` response. Send a request to the health check endpoint: ``` curl "http://127.0.0.1:9090/v1/healthcheck" ``` You should see a response similar to the following: ``` [ { "name": "/apisix/routes/example-hc-route", "nodes": [ { "counter": { "timeout_failure": 0, "http_failure": 1, "success": 0, "tcp_failure": 0 }, "port": 80, "ip": "172.25.0.5", "status": "mostly_healthy" }, { "counter": { "timeout_failure": 0, "http_failure": 0, "success": 0, "tcp_failure": 0 }, "port": 80, "ip": "172.25.0.4", "status": "healthy" } ], "type": "http" } ] ``` ❶ `http_failure` has a count of 1 due to the previous request with a 404 response. ❷ `mostly_healthy` status means the current node status is healthy, but APISIX starts to receive unhealthy indications during health checks. Generate consecutive requests to invoke `404` responses: ``` resp=$(seq 10 | xargs -I{} curl "http://127.0.0.1:9080/404" -o /dev/null -s -w "%{http_code}\n") && \ count=$(echo "$resp" | grep "404" | wc -l) && \ echo "Invoked $count responses with 404 status code." ``` Send a request to the health check endpoint: ``` curl "http://127.0.0.1:9090/v1/healthcheck" ``` You should see a response similar to the following: ``` [ { "name": "/apisix/routes/example-hc-route", "nodes": [ { "counter": { "timeout_failure": 0, "http_failure": 3, "success": 0, "tcp_failure": 0 }, "port": 80, "ip": "172.25.0.4", "status": "unhealthy" }, { "counter": { "timeout_failure": 0, "http_failure": 3, "success": 0, "tcp_failure": 0 }, "port": 80, "ip": "172.25.0.5", "status": "unhealthy" } ], "type": "http" } ] ``` Wait at least 30 seconds for active checks to probe the upstream services at `/` and mark them as healthy. Then, send a request to the health check endpoint: ``` curl "http://127.0.0.1:9090/v1/healthcheck" ``` You should see a response similar to the following: ``` [ { "name": "/apisix/routes/example-hc-route", "nodes": [ { "counter": { "timeout_failure": 0, "http_failure": 0, "success": 1, "tcp_failure": 0 }, "port": 80, "ip": "172.25.0.4", "status": "healthy" }, { "counter": { "timeout_failure": 0, "http_failure": 0, "success": 1, "tcp_failure": 0 }, "port": 80, "ip": "172.25.0.5", "status": "healthy" } ], "type": "http" } ] ``` ## Inspect Plugin-Owned Health Checks[​](#inspect-plugin-owned-health-checks "Direct link to Inspect Plugin-Owned Health Checks") The Control API health listing includes plugin-owned checkers. For example, `ai-proxy-multi` can check each configured AI instance independently of an ordinary upstream resource. Each plugin entry includes the plugin name and can include metadata identifying the target. To inspect all checkers owned by a route, replace `ai-route` with its ID: ``` curl "http://127.0.0.1:9090/v1/healthcheck/routes/ai-route/checkers" ``` The response is an array containing that resource's upstream and plugin-owned checkers, or `[]` when the resource exists but has no checkers. Query the service or upstream itself if it owns the configuration; this endpoint does not follow resource references. The older endpoint without `/checkers` still returns a single upstream checker object and does not list plugin-owned checkers. A configured checker can report an empty `nodes` array before its targets are initialized. Send traffic to the applicable route before interpreting that state as a failure. These endpoints show local instance state, not the health of an entire deployment. The Control API does not authenticate requests, so keep it on a loopback or trusted internal interface. See the [Control API reference](https://docs.api7.ai/apisix/reference/control-api.md#tag/Health-Check) for response details. ## Disable All Health Checks[​](#disable-all-health-checks "Direct link to Disable All Health Checks") You can disable all upstream health checks globally. This is useful in scenarios such as emergency maintenance, where health checks might interfere with routing or fallback behavior. To disable all health checks, update your [configuration file](https://docs.api7.ai/apisix/reference/configuration-files.md#configyaml-and-configyamlexample) as follows: conf/config.yaml ``` apisix: disable_upstream_healthcheck: true ``` [Reload APISIX](https://docs.api7.ai/apisix/reference/apisix-cli.md#apisix-reload) for configuration changes to take effect: ``` docker exec apisix-quickstart apisix reload ``` ## Next Steps[​](#next-steps "Direct link to Next Steps") You have now learned how to configure active and passive health checks for upstream services in APISIX. To learn more about the available configuration options for upstream health checks, see [Admin API, Upstream](https://docs.api7.ai/apisix/reference/admin-api.md#tag/Upstreams/paths/~1apisix~1admin~1upstreams/post) for reference. APISIX also offers an `api-breaker` plugin, which implements circuit breaker functionality based on the health of upstream services and helps improve application resilience.