Skip to main content
API7.ai Logo

Configure Upstream Health Checks

Health checking is a mechanism that determines whether upstream services are healthy or unhealthy based on their responsiveness. With health checks enabled, APISIX will only forward requests to upstream services that are considered healthy and will not forward requests to services that are considered unhealthy.

There are two general approaches to health check:

  • Active checks: APISIX proactively and periodically sends requests to upstream services and determines the health of those based on the responses to these requests.
  • Passive checks: APISIX determines the health of upstream services based on how they respond to client requests, without proactively probing.

This guide will show you how to configure both active and passive health checks for your upstream services.

Prerequisite(s)​

  • Install Docker.
  • Install cURL to send requests to the services for validation.
  • Follow the Getting Started tutorial to start a new APISIX instance in Docker or on Kubernetes.

Start Sample Upstream Services​

Start two NGINX instances as sample upstream services in the same Docker network as APISIX:

DOCKER_NETWORK=apisix-quickstart-net
docker run -d -p 8080:80 --network=${DOCKER_NETWORK} --name nginx1 nginx
docker run -d -p 8081:80 --network=${DOCKER_NETWORK} --name nginx2 nginx

Verify both NGINX instances are running:

for port in 8080 8081; do
  curl -s "http://127.0.0.1:$port" | grep -q "Welcome to nginx" &&
  echo "NGINX welcome page available on port $port."
done

You should see the following response:

NGINX welcome page available on port 8080.
NGINX welcome page available on port 8081.

Configure Active Health Checks​

Active checks determine the health of upstream services by periodically sending requests, or probes, to the services and seeing how they respond.

The examples below show how to:

  • send a custom method and body in an active probe
  • detect changes in upstream status with active checks
  • observe how APISIX forwards requests when all upstream statuses are unhealthy

Active HTTP and HTTPS checks use GET with an empty body by default. Set http_method to any supported HTTP method and use http_req_body when the health endpoint expects a request payload. A nonempty body adds the corresponding Content-Length header automatically.

Send a POST Health Check​

Configure the first sample NGINX service with a health endpoint that accepts only POST and logs the probe method, path, and content length:

docker exec nginx1 /bin/sh -c 'cat > /etc/nginx/conf.d/default.conf <<"EOF"
log_format health_probe "$request_method $uri $content_length";

server {
  listen 80;
  access_log /tmp/health-probe.log health_probe;

  location = /health {
    if ($request_method != POST) {
      return 405;
    }
    return 200 "healthy\n";
  }

  location / {
    return 200 "Welcome to nginx!\n";
  }
}
EOF
nginx -s reload'

Create a route with a POST active check and a JSON body:

curl "http://127.0.0.1:9180/apisix/admin/routes/post-health-check" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d '{
    "uri": "/post-health-check",
    "upstream": {
      "type": "roundrobin",
      "nodes": {
        "nginx1:80": 1
      },
      "checks": {
        "active": {
          "type": "http",
          "http_method": "POST",
          "http_path": "/health",
          "http_req_body": "{\"status\":\"check\"}",
          "req_headers": [
            "Content-Type: application/json"
          ],
          "healthy": {
            "interval": 2,
            "successes": 1
          },
          "unhealthy": {
            "interval": 1,
            "http_failures": 1
          }
        }
      }
    }
  }'

Send a request through the route to start its health checker, wait for a probe, and inspect the NGINX access log:

curl -i "http://127.0.0.1:9080/post-health-check"
sleep 2
docker exec nginx1 /bin/sh -c "grep 'POST /health' /tmp/health-probe.log | tail -n 1"

The route request should return 200 OK. The log should confirm that the active probe used POST and sent the 18-byte body:

POST /health 18

Example: Status Change in Upstream Services​

The following example demonstrates how APISIX active health checks respond in situations where healthy upstream services have become: partially unavailable, all unavailable, and all recovered.

Create a route to the two services and configure active health checks that run every 2 seconds:

curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT -d '
{
  "id": "example-hc-route",
  "uri":"/",
  "upstream": {
    "type":"roundrobin",
    "nodes": {
      "nginx1:80": 1,
      "nginx2:80": 1
    },
    "checks": {
      "active": {
        "type": "http",
        "http_path": "/",
        "healthy": {
          "interval": 2,
          "successes": 1
        },
        "unhealthy": {
          "interval": 1,
          "timeouts": 3
        }
      }
    }
  }
}'

❶ type: the type of active health checks.

❷ http_path: the HTTP request path to actively probe.

❸ healthy.interval: the time interval in seconds for periodically checking healthy nodes.

❹ healthy.successes: the success count threshold for ruling if an upstream node is considered healthy.

❺ unhealthy.interval: the time interval in seconds for periodically checking unhealthy nodes.

❻ unhealthy.timeouts: the timeout count threshold for ruling if an upstream node is considered unhealthy.

Verify​

You will be verifying the above configurations to understand how APISIX upstream health checks respond in different scenarios:

If you started APISIX in Docker with Getting Started quickstart, Control API port 9090 is already mapped (-p 9090:9090).

Verify Both Upstream Services Are Healthy​

Send a request to the route to start health checks:

curl "http://127.0.0.1:9080/"

To see upstream health statuses, send a request to the health check endpoint in Control API:

curl "http://127.0.0.1:9090/v1/healthcheck"

You should see a response similar to the following:

[
  {
    "name": "/apisix/routes/example-hc-route",
    "type": "http",
    "nodes": [
      {
        "port": 80,
        "counter": {
          "http_failure": 0,
          "tcp_failure": 0,
          "timeout_failure": 0,
          "success": 0
        },
        "ip": "172.24.0.5",
        "status": "healthy"
      },
      {
        "port": 80,
        "counter": {
          "http_failure": 0,
          "tcp_failure": 0,
          "timeout_failure": 0,
          "success": 0
        },
        "ip": "172.24.0.4",
        "status": "healthy"
      }
    ]
  }
]

Verify When One Upstream Service Is Unavailable​

Make one upstream service temporarily unavailable to verify if APISIX reports one of the upstream services unhealthy:

docker container stop nginx1

Wait for a few seconds and send a request to the health check endpoint:

curl "http://127.0.0.1:9090/v1/healthcheck"

You should see a response similar to the following, showing one of the upstream nodes has 3 timeout failures and marked unhealthy:

[
  {
    "name": "/apisix/routes/example-hc-route",
    "type": "http",
    "nodes": [
      {
        "port": 80,
        "counter": {
          "http_failure": 0,
          "tcp_failure": 0,
          "timeout_failure": 0,
          "success": 0
        },
        "ip": "172.24.0.5",
        "status": "healthy"
      },
      {
        "port": 80,
        "counter": {
          "http_failure": 0,
          "tcp_failure": 0,
          "timeout_failure": 3,
          "success": 0
        },
        "ip": "172.24.0.4",
        "status": "unhealthy"
      }
    ]
  }
]

Send a request to the route to see if APISIX forwards the request to the other healthy node:

curl -i "http://127.0.0.1:9080/"

You should receive an HTTP/1.1 200 OK response.

Verify Both Upstream Services Are Unavailable​

Make the other upstream service temporarily unavailable to verify if APISIX reports both upstream services unhealthy:

docker container stop nginx2

Wait for a few seconds and send a request to the health check endpoint:

curl "http://127.0.0.1:9090/v1/healthcheck"

You should see a response similar to the following, showing both upstream nodes have 3 timeout failures and marked unhealthy:

[
  {
    "name": "/apisix/routes/example-hc-route",
    "type": "http",
    "nodes": [
      {
        "port": 80,
        "counter": {
          "http_failure": 0,
          "tcp_failure": 0,
          "timeout_failure": 3,
          "success": 0
        },
        "ip": "172.24.0.5",
        "status": "unhealthy"
      },
      {
        "port": 80,
        "counter": {
          "http_failure": 0,
          "tcp_failure": 0,
          "timeout_failure": 3,
          "success": 0
        },
        "ip": "172.24.0.4",
        "status": "unhealthy"
      }
    ]
  }
]

Send a request to the route:

curl -i "http://127.0.0.1:9080/"

You should receive an HTTP/1.1 502 Bad Gateway response.

Verify Both Upstream Services Are Recovered​

Make both services available again to verify if APISIX reports both upstream services healthy:

docker container start nginx1 nginx2

Wait for a few seconds and send a request to the health check endpoint:

curl "http://127.0.0.1:9090/v1/healthcheck"

You should see a response showing both upstream nodes are healthy, similar to when both services are healthy at the start.

Example: Forward Requests When Statuses Are Unhealthy​

The following example demonstrates that APISIX would still forward client requests to upstream services even when all upstream health statuses are unhealthy.

Create a route to the two services and configure active health checks that run every 2 seconds:

curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT -d '
{
  "id": "example-hc-route",
  "uri":"/",
  "upstream": {
    "type":"roundrobin",
    "nodes": {
      "nginx1:80": 1,
      "nginx2:80": 1
    },
    "checks": {
      "active": {
        "type": "http",
        "http_path": "/404",
        "healthy": {
          "interval": 2,
          "successes": 1
        },
        "unhealthy": {
          "interval": 1,
          "http_failures": 2
        }
      }
    }
  }
}'

❶ type: the type of active health checks.

❷ http_path: the HTTP request path to actively probe. For the convenience of demonstration, this is set to /404, which is a path that does not exist in upstream services. Consequently, both services should always be considered unhealthy by the active health checks.

❸ unhealthy.http_failures: the HTTP failure count threshold for ruling if an upstream node is considered unhealthy.

Verify​

If you started APISIX in Docker with Getting Started quickstart, Control API port 9090 is already mapped (-p 9090:9090).

Send a request to the route to start health checks:

curl -i "http://127.0.0.1:9080/"

You should receive an HTTP/1.1 200 OK response.

Send a request to the health check endpoint:

curl "http://127.0.0.1:9090/v1/healthcheck"

You should see a response similar to the following:

[
  {
    "name": "/apisix/routes/example-hc-route",
    "nodes": [
      {
        "counter": {
          "timeout_failure": 0,
          "http_failure": 2,
          "success": 0,
          "tcp_failure": 0
        },
        "port": 80,
        "ip": "172.25.0.5",
        "status": "unhealthy"
      },
      {
        "counter": {
          "timeout_failure": 0,
          "http_failure": 2,
          "success": 0,
          "tcp_failure": 0
        },
        "port": 80,
        "ip": "172.25.0.4",
        "status": "unhealthy"
      }
    ],
    "type": "http"
  }
]

Send a request to the route to see if APISIX still forwards the request:

curl -i "http://127.0.0.1:9080/"

You should receive an HTTP/1.1 200 OK response. This verifies that APISIX would still forward client requests to upstream services, despite both services being marked as unhealthy.

Configure Passive Health Checks​

APISIX requires the use of active health checks with passive health checks. When an upstream service becomes unhealthy, the active health check is in place to periodically check if the upstream service has recovered.

note

There is a known issue where the health check data displayed through the Control API does not accurately reflect the actual health statuses, so your testing results may differ from the example shown. The issue is being actively resolved. However, the passive health check mechanism itself is functioning correctly and continues to route requests as expected.

Example: Status Change in Upstream Services​

Create a route to the two services, and configure both active and passive health checks:

curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT -d '
{
  "id": "example-hc-route",
  "uri": "/404",
  "upstream": {
    "type": "roundrobin",
    "nodes": {
      "nginx1:80": 1,
      "nginx2:80": 1
    },
    "checks": {
      "active": {
        "type": "http",
        "http_path": "/",
        "healthy": {
          "interval": 99999,
          "successes": 1
        },
        "unhealthy": {
          "interval": 30
        }
      },
      "passive": {
        "healthy": {
          "http_statuses": [200,201,202,300,301,302],
          "successes": 1
        },
        "unhealthy": {
          "http_statuses": [429,404,500,501,502,503,504,505],
          "http_failures": 3
        }
      }
    }
  }
}'

❶ uri: the URI path that the route matches. For the convenience of demonstration, this is set to /404, which is a path that does not exist in upstream services. Consequently, when a request is made, both upstream services should respond with a 404 status code.

❷ active.healthy.interval: the time interval in seconds for periodically checking healthy nodes.

❸ active.unhealthy.interval: the time interval in seconds for periodically checking unhealthy nodes.

❹ passive.healthy.http_statuses: the response HTTP status codes that are considered healthy.

❺ passive.unhealthy.http_statuses: the response HTTP status codes that are considered unhealthy. The unhealthy responses are counted towards the http_failures.

❻ passive.unhealthy.http_failures: the HTTP failure count threshold for ruling if an upstream node is considered unhealthy.

Verify​

If you started APISIX in Docker with Getting Started quickstart, Control API port 9090 is already mapped (-p 9090:9090).

Send a request to the route to start health checks:

curl -i "http://127.0.0.1:9080/404"

You should see an HTTP/1.1 404 Not Found response.

Send a request to the health check endpoint:

curl "http://127.0.0.1:9090/v1/healthcheck"

You should see a response similar to the following:

[
  {
    "name": "/apisix/routes/example-hc-route",
    "nodes": [
      {
        "counter": {
          "timeout_failure": 0,
          "http_failure": 1,
          "success": 0,
          "tcp_failure": 0
        },
        "port": 80,
        "ip": "172.25.0.5",
        "status": "mostly_healthy"
      },
      {
        "counter": {
          "timeout_failure": 0,
          "http_failure": 0,
          "success": 0,
          "tcp_failure": 0
        },
        "port": 80,
        "ip": "172.25.0.4",
        "status": "healthy"
      }
    ],
    "type": "http"
  }
]

❶ http_failure has a count of 1 due to the previous request with a 404 response.

❷ mostly_healthy status means the current node status is healthy, but APISIX starts to receive unhealthy indications during health checks.

Generate consecutive requests to invoke 404 responses:

resp=$(seq 10 | xargs -I{} curl "http://127.0.0.1:9080/404" -o /dev/null -s -w "%{http_code}\n") && \
  count=$(echo "$resp" | grep "404" | wc -l) && \
  echo "Invoked $count responses with 404 status code."

Send a request to the health check endpoint:

curl "http://127.0.0.1:9090/v1/healthcheck"

You should see a response similar to the following:

[
  {
    "name": "/apisix/routes/example-hc-route",
    "nodes": [
      {
        "counter": {
          "timeout_failure": 0,
          "http_failure": 3,
          "success": 0,
          "tcp_failure": 0
        },
        "port": 80,
        "ip": "172.25.0.4",
        "status": "unhealthy"
      },
      {
        "counter": {
          "timeout_failure": 0,
          "http_failure": 4,
          "success": 0,
          "tcp_failure": 0
        },
        "port": 80,
        "ip": "172.25.0.5",
        "status": "unhealthy"
      }
    ],
    "type": "http"
  }
]

Wait at least 30 seconds for active checks to probe the upstream services at / and mark them as healthy. Then, send a request to the health check endpoint:

curl "http://127.0.0.1:9090/v1/healthcheck"

You should see a response similar to the following:

[
  {
    "name": "/apisix/routes/example-hc-route",
    "nodes": [
      {
        "counter": {
          "timeout_failure": 0,
          "http_failure": 0,
          "success": 1,
          "tcp_failure": 0
        },
        "port": 80,
        "ip": "172.25.0.4",
        "status": "healthy"
      },
      {
        "counter": {
          "timeout_failure": 0,
          "http_failure": 0,
          "success": 1,
          "tcp_failure": 0
        },
        "port": 80,
        "ip": "172.25.0.5",
        "status": "healthy"
      }
    ],
    "type": "http"
  }
]

Disable All Health Checks​

You can disable all upstream health checks globally. This is useful in scenarios such as emergency maintenance, where health checks might interfere with routing or fallback behavior.

To disable all health checks, update your configuration file as follows:

conf/config.yaml
apisix:
  disable_upstream_healthcheck: true

Reload APISIX for configuration changes to take effect:

docker exec apisix-quickstart apisix reload

Next Steps​

You have now learned how to configure active and passive health checks for upstream services in APISIX. To learn more about the available configuration options for upstream health checks, see Admin API, Upstream for reference.

APISIX also offers an api-breaker plugin, which implements circuit breaker functionality based on the health of upstream services and helps improve application resilience.