Skip to main content
API7.ai Logo
Version: 3.20.0

Configure Upstream Health Checks

Health checking determines whether upstream services are healthy or unhealthy based on their responsiveness. APISIX excludes unhealthy nodes while at least one healthy node remains. If all nodes are unhealthy, APISIX falls back to the original node set so that a reachable node can still serve traffic.

There are two general approaches to health check:

  • Active checks: APISIX proactively and periodically sends requests to upstream services and determines the health of those based on the responses to these requests.
  • Passive checks: APISIX determines the health of upstream services based on how they respond to client requests, without proactively probing.

This guide will show you how to configure both active and passive health checks for your upstream services.

Prerequisite(s)​

  • Install Docker.
  • Install cURL to send requests to the services for validation.
  • Follow the Getting Started tutorial to start a new APISIX instance in Docker or on Kubernetes.
  • If you use the Admin API examples and authentication is enabled, export an admin-role key as ADMIN_API_KEY. Configure the key under deployment.admin.admin_key. The Docker quickstart disables authentication for local evaluation; keep it enabled in production.

Start Sample Upstream Services​

Start two NGINX instances as sample upstream services in the same Docker network as APISIX:

DOCKER_NETWORK=apisix-quickstart-net
docker run -d -p 8080:80 --network=${DOCKER_NETWORK} --name nginx1 nginx
docker run -d -p 8081:80 --network=${DOCKER_NETWORK} --name nginx2 nginx

Verify both NGINX instances are running:

for port in 8080 8081; do
  curl -s "http://127.0.0.1:$port" | grep -q "Welcome to nginx" &&
  echo "NGINX welcome page available on port $port."
done

You should see the following response:

NGINX welcome page available on port 8080.
NGINX welcome page available on port 8081.

Configure Active Health Checks​

Active checks determine the health of upstream services by periodically sending requests, or probes, to the services and seeing how they respond.

The examples below show how to:

  • send a custom method and body in an active probe
  • detect changes in upstream status with active checks
  • observe how APISIX forwards requests when all upstream statuses are unhealthy

Active HTTP and HTTPS checks use GET with an empty body by default. Set http_method to any supported HTTP method and use http_req_body when the health endpoint expects a request payload. A nonempty body adds the corresponding Content-Length header automatically.

Send a POST Health Check​

Configure the first sample NGINX service with a health endpoint that accepts only POST and logs the probe method, path, and content length:

docker exec nginx1 /bin/sh -c 'cat > /etc/nginx/conf.d/default.conf <<"EOF"
log_format health_probe "$request_method $uri $content_length";

server {
  listen 80;
  access_log /tmp/health-probe.log health_probe;

  location = /health {
    if ($request_method != POST) {
      return 405;
    }
    return 200 "healthy\n";
  }

  location = /404 {
    return 404;
  }

  location / {
    return 200 "Welcome to nginx!\n";
  }
}
EOF
nginx -s reload'

Create a route with a POST active check and a JSON body:

curl "http://127.0.0.1:9180/apisix/admin/routes/post-health-check" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d '{
    "uri": "/post-health-check",
    "upstream": {
      "type": "roundrobin",
      "nodes": {
        "nginx1:80": 1
      },
      "checks": {
        "active": {
          "type": "http",
          "http_method": "POST",
          "http_path": "/health",
          "http_req_body": "{\"status\":\"check\"}",
          "req_headers": [
            "Content-Type: application/json"
          ],
          "healthy": {
            "interval": 2,
            "successes": 1
          },
          "unhealthy": {
            "interval": 1,
            "http_failures": 1
          }
        }
      }
    }
  }'

Send a request through the route to start its health checker, wait for a probe, and inspect the NGINX access log:

curl -i "http://127.0.0.1:9080/post-health-check"
sleep 2
docker exec nginx1 /bin/sh -c "grep 'POST /health' /tmp/health-probe.log | tail -n 1"

The route request should return 200 OK. The log should confirm that the active probe used POST and sent the 18-byte body:

POST /health 18

Example: Status Change in Upstream Services​

The following example demonstrates how APISIX active health checks respond in situations where healthy upstream services have become: partially unavailable, all unavailable, and all recovered.

Create a route to the two services and configure active health checks that run every 2 seconds:

curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d '{
  "id": "example-hc-route",
  "uri":"/",
  "upstream": {
    "type":"roundrobin",
    "nodes": {
      "nginx1:80": 1,
      "nginx2:80": 1
    },
    "checks": {
      "active": {
        "type": "http",
        "http_path": "/",
        "healthy": {
          "interval": 2,
          "successes": 1
        },
        "unhealthy": {
          "interval": 1,
          "timeouts": 3
        }
      }
    }
  }
}'

❶ type: the type of active health checks.

❷ http_path: the HTTP request path to actively probe.

❸ healthy.interval: the time interval in seconds for periodically checking healthy nodes.

❹ healthy.successes: the success count threshold for ruling if an upstream node is considered healthy.

❺ unhealthy.interval: the time interval in seconds for periodically checking unhealthy nodes.

❻ unhealthy.timeouts: the timeout count threshold for ruling if an upstream node is considered unhealthy.

Verify​

You will be verifying the above configurations to understand how APISIX upstream health checks respond in different scenarios:

If you started APISIX with the Docker Getting Started quickstart, the Control API is available on host port 9090.

Verify Both Upstream Services Are Healthy​

Send a request to the route to start health checks:

curl "http://127.0.0.1:9080/"

To see upstream health statuses, send a request to the health check endpoint in Control API:

curl "http://127.0.0.1:9090/v1/healthcheck"

You should see a response similar to the following:

[
  {
    "name": "/apisix/routes/example-hc-route",
    "type": "http",
    "nodes": [
      {
        "port": 80,
        "counter": {
          "http_failure": 0,
          "tcp_failure": 0,
          "timeout_failure": 0,
          "success": 0
        },
        "ip": "172.24.0.5",
        "status": "healthy"
      },
      {
        "port": 80,
        "counter": {
          "http_failure": 0,
          "tcp_failure": 0,
          "timeout_failure": 0,
          "success": 0
        },
        "ip": "172.24.0.4",
        "status": "healthy"
      }
    ]
  }
]

Verify When One Upstream Service Is Unavailable​

Make one upstream service temporarily unavailable to verify if APISIX reports one of the upstream services unhealthy:

docker container stop nginx1

Wait for a few seconds and send a request to the health check endpoint:

curl "http://127.0.0.1:9090/v1/healthcheck"

You should see a response similar to the following, showing one of the upstream nodes has 3 timeout failures and marked unhealthy:

[
  {
    "name": "/apisix/routes/example-hc-route",
    "type": "http",
    "nodes": [
      {
        "port": 80,
        "counter": {
          "http_failure": 0,
          "tcp_failure": 0,
          "timeout_failure": 0,
          "success": 0
        },
        "ip": "172.24.0.5",
        "status": "healthy"
      },
      {
        "port": 80,
        "counter": {
          "http_failure": 0,
          "tcp_failure": 0,
          "timeout_failure": 3,
          "success": 0
        },
        "ip": "172.24.0.4",
        "status": "unhealthy"
      }
    ]
  }
]

Send a request to the route to see if APISIX forwards the request to the other healthy node:

curl -i "http://127.0.0.1:9080/"

You should receive an HTTP/1.1 200 OK response.

Verify Both Upstream Services Are Unavailable​

Make the other upstream service temporarily unavailable to verify if APISIX reports both upstream services unhealthy:

docker container stop nginx2

Wait for a few seconds and send a request to the health check endpoint:

curl "http://127.0.0.1:9090/v1/healthcheck"

You should see a response similar to the following, showing both upstream nodes have 3 timeout failures and marked unhealthy:

[
  {
    "name": "/apisix/routes/example-hc-route",
    "type": "http",
    "nodes": [
      {
        "port": 80,
        "counter": {
          "http_failure": 0,
          "tcp_failure": 0,
          "timeout_failure": 3,
          "success": 0
        },
        "ip": "172.24.0.5",
        "status": "unhealthy"
      },
      {
        "port": 80,
        "counter": {
          "http_failure": 0,
          "tcp_failure": 0,
          "timeout_failure": 3,
          "success": 0
        },
        "ip": "172.24.0.4",
        "status": "unhealthy"
      }
    ]
  }
]

Send a request to the route:

curl -i "http://127.0.0.1:9080/"

You should receive an HTTP/1.1 502 Bad Gateway response.

When every node is unhealthy, APISIX falls back to the original node set. In this example, both fallback nodes are stopped and unreachable, so the request returns 502 Bad Gateway.

Verify Both Upstream Services Are Recovered​

Make both services available again to verify if APISIX reports both upstream services healthy:

docker container start nginx1 nginx2

Wait for a few seconds and send a request to the health check endpoint:

curl "http://127.0.0.1:9090/v1/healthcheck"

You should see a response showing both upstream nodes are healthy, similar to when both services are healthy at the start.

Example: Forward Requests When Statuses Are Unhealthy​

The following example demonstrates that APISIX would still forward client requests to upstream services even when all upstream health statuses are unhealthy.

Create a route to the two services and configure active health checks that run every 2 seconds:

curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d '{
  "id": "example-hc-route",
  "uri":"/",
  "upstream": {
    "type":"roundrobin",
    "nodes": {
      "nginx1:80": 1,
      "nginx2:80": 1
    },
    "checks": {
      "active": {
        "type": "http",
        "http_path": "/404",
        "healthy": {
          "interval": 2,
          "successes": 1
        },
        "unhealthy": {
          "interval": 1,
          "http_failures": 2
        }
      }
    }
  }
}'

❶ type: the type of active health checks.

❷ http_path: the HTTP request path to actively probe. For the convenience of demonstration, this is set to /404, which is a path that does not exist in upstream services. Consequently, both services should always be considered unhealthy by the active health checks.

❸ unhealthy.http_failures: the HTTP failure count threshold for ruling if an upstream node is considered unhealthy.

Verify​

If you started APISIX with the Docker Getting Started quickstart, the Control API is available on host port 9090.

Send a request to the route to start health checks:

curl -i "http://127.0.0.1:9080/"

You should receive an HTTP/1.1 200 OK response.

Send a request to the health check endpoint:

curl "http://127.0.0.1:9090/v1/healthcheck"

You should see a response similar to the following:

[
  {
    "name": "/apisix/routes/example-hc-route",
    "nodes": [
      {
        "counter": {
          "timeout_failure": 0,
          "http_failure": 2,
          "success": 0,
          "tcp_failure": 0
        },
        "port": 80,
        "ip": "172.25.0.5",
        "status": "unhealthy"
      },
      {
        "counter": {
          "timeout_failure": 0,
          "http_failure": 2,
          "success": 0,
          "tcp_failure": 0
        },
        "port": 80,
        "ip": "172.25.0.4",
        "status": "unhealthy"
      }
    ],
    "type": "http"
  }
]

Send a request to the route to see if APISIX still forwards the request:

curl -i "http://127.0.0.1:9080/"

You should receive an HTTP/1.1 200 OK response. When no healthy node remains, APISIX falls back to the original node set. Both NGINX services are reachable in this example, so one of them can still serve the request despite being marked unhealthy.

Configure Passive Health Checks​

APISIX requires the use of active health checks with passive health checks. When an upstream service becomes unhealthy, the active health check is in place to periodically check if the upstream service has recovered.

Example: Status Change in Upstream Services​

Create a route to the two services, and configure both active and passive health checks:

curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d '{
  "id": "example-hc-route",
  "uri": "/404",
  "upstream": {
    "type": "roundrobin",
    "nodes": {
      "nginx1:80": 1,
      "nginx2:80": 1
    },
    "checks": {
      "active": {
        "type": "http",
        "http_path": "/",
        "healthy": {
          "interval": 99999,
          "successes": 1
        },
        "unhealthy": {
          "interval": 30
        }
      },
      "passive": {
        "healthy": {
          "http_statuses": [200,201,202,300,301,302],
          "successes": 1
        },
        "unhealthy": {
          "http_statuses": [429,404,500,501,502,503,504,505],
          "http_failures": 3
        }
      }
    }
  }
}'

❶ uri: the URI path that the route matches. For the convenience of demonstration, this is set to /404, which is a path that does not exist in upstream services. Consequently, when a request is made, both upstream services should respond with a 404 status code.

❷ active.healthy.interval: the time interval in seconds for periodically checking healthy nodes.

❸ active.unhealthy.interval: the time interval in seconds for periodically checking unhealthy nodes.

❹ passive.healthy.http_statuses: the response HTTP status codes that are considered healthy.

❺ passive.unhealthy.http_statuses: the response HTTP status codes that are considered unhealthy. The unhealthy responses are counted towards the http_failures.

❻ passive.unhealthy.http_failures: the HTTP failure count threshold for ruling if an upstream node is considered unhealthy.

Verify​

If you started APISIX with the Docker Getting Started quickstart, the Control API is available on host port 9090.

Send a request to the route to start health checks:

curl -i "http://127.0.0.1:9080/404"

You should see an HTTP/1.1 404 Not Found response.

Send a request to the health check endpoint:

curl "http://127.0.0.1:9090/v1/healthcheck"

You should see a response similar to the following:

[
  {
    "name": "/apisix/routes/example-hc-route",
    "nodes": [
      {
        "counter": {
          "timeout_failure": 0,
          "http_failure": 1,
          "success": 0,
          "tcp_failure": 0
        },
        "port": 80,
        "ip": "172.25.0.5",
        "status": "mostly_healthy"
      },
      {
        "counter": {
          "timeout_failure": 0,
          "http_failure": 0,
          "success": 0,
          "tcp_failure": 0
        },
        "port": 80,
        "ip": "172.25.0.4",
        "status": "healthy"
      }
    ],
    "type": "http"
  }
]

❶ http_failure has a count of 1 due to the previous request with a 404 response.

❷ mostly_healthy status means the current node status is healthy, but APISIX starts to receive unhealthy indications during health checks.

Generate consecutive requests to invoke 404 responses:

resp=$(seq 10 | xargs -I{} curl "http://127.0.0.1:9080/404" -o /dev/null -s -w "%{http_code}\n") && \
  count=$(echo "$resp" | grep "404" | wc -l) && \
  echo "Invoked $count responses with 404 status code."

Send a request to the health check endpoint:

curl "http://127.0.0.1:9090/v1/healthcheck"

You should see a response similar to the following:

[
  {
    "name": "/apisix/routes/example-hc-route",
    "nodes": [
      {
        "counter": {
          "timeout_failure": 0,
          "http_failure": 3,
          "success": 0,
          "tcp_failure": 0
        },
        "port": 80,
        "ip": "172.25.0.4",
        "status": "unhealthy"
      },
      {
        "counter": {
          "timeout_failure": 0,
          "http_failure": 3,
          "success": 0,
          "tcp_failure": 0
        },
        "port": 80,
        "ip": "172.25.0.5",
        "status": "unhealthy"
      }
    ],
    "type": "http"
  }
]

Wait at least 30 seconds for active checks to probe the upstream services at / and mark them as healthy. Then, send a request to the health check endpoint:

curl "http://127.0.0.1:9090/v1/healthcheck"

You should see a response similar to the following:

[
  {
    "name": "/apisix/routes/example-hc-route",
    "nodes": [
      {
        "counter": {
          "timeout_failure": 0,
          "http_failure": 0,
          "success": 1,
          "tcp_failure": 0
        },
        "port": 80,
        "ip": "172.25.0.4",
        "status": "healthy"
      },
      {
        "counter": {
          "timeout_failure": 0,
          "http_failure": 0,
          "success": 1,
          "tcp_failure": 0
        },
        "port": 80,
        "ip": "172.25.0.5",
        "status": "healthy"
      }
    ],
    "type": "http"
  }
]

Inspect Plugin-Owned Health Checks​

The Control API health listing includes plugin-owned checkers. For example, ai-proxy-multi can check each configured AI instance independently of an ordinary upstream resource. Each plugin entry includes the plugin name and can include metadata identifying the target.

To inspect all checkers owned by a route, replace ai-route with its ID:

curl "http://127.0.0.1:9090/v1/healthcheck/routes/ai-route/checkers"

The response is an array containing that resource's upstream and plugin-owned checkers, or [] when the resource exists but has no checkers. Query the service or upstream itself if it owns the configuration; this endpoint does not follow resource references. The older endpoint without /checkers still returns a single upstream checker object and does not list plugin-owned checkers.

A configured checker can report an empty nodes array before its targets are initialized. Send traffic to the applicable route before interpreting that state as a failure. These endpoints show local instance state, not the health of an entire deployment. The Control API does not authenticate requests, so keep it on a loopback or trusted internal interface. See the Control API reference for response details.

Disable All Health Checks​

You can disable all upstream health checks globally. This is useful in scenarios such as emergency maintenance, where health checks might interfere with routing or fallback behavior.

To disable all health checks, update your configuration file as follows:

conf/config.yaml
apisix:
  disable_upstream_healthcheck: true

Reload APISIX for configuration changes to take effect:

docker exec apisix-quickstart apisix reload

Next Steps​

You have now learned how to configure active and passive health checks for upstream services in APISIX. To learn more about the available configuration options for upstream health checks, see Admin API, Upstream for reference.

APISIX also offers an api-breaker plugin, which implements circuit breaker functionality based on the health of upstream services and helps improve application resilience.