Skip to main content
API7.ai Logo
Version: 3.10.x

Configure Upstream Health Checks

Health checks are crucial for maintaining the availability of your services. By monitoring the health of upstream nodes, the Gateway can automatically remove unhealthy nodes from the load-balancing pool and redirect traffic only to healthy ones.

API7 Enterprise supports two types of health checks:

  • Active Health Check: The Gateway periodically sends probe requests to upstream nodes to verify their status.
  • Passive Health Check: The Gateway monitors the responses of real client requests to detect failures.

Prerequisites​

  • An API7 Enterprise instance is running.
  • A Gateway Group is created and a Gateway instance is running.
  • A token from the Dashboard.

Start Sample Upstream Services​

For local validation, start two sample upstream services on the host:

docker run -d --name tm-health-1 -p 18083:80 kennethreitz/httpbin
docker run -d --name tm-health-2 -p 18084:80 kennethreitz/httpbin

The examples below assume the gateway can reach the host on 192.168.215.1.

Workflow Overview​

The Gateway sends active probes and monitors real traffic to detect unhealthy nodes. Traffic is only routed to nodes that are marked as healthy.

Configure Active Health Check​

Active health checks allow the Gateway to detect failures even when there is no client traffic.

# 1. Create a Service with two upstream nodes and active checks
curl -k "https://localhost:7443/apisix/admin/services/upstream-with-health-check?gateway_group_id={gateway_group_id}" -X PUT \
  -H "X-API-KEY: ${API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "upstream-with-health-check",
    "upstream": {
      "type": "roundrobin",
      "nodes": [
        {
          "host": "192.168.215.1",
          "port": 18083,
          "weight": 100
        },
        {
          "host": "192.168.215.1",
          "port": 18084,
          "weight": 100
        }
      ],
      "checks": {
        "active": {
          "type": "http",
          "http_path": "/status/200",
          "timeout": 1,
          "concurrency": 10,
          "healthy": {
            "interval": 2,
            "successes": 1,
            "http_statuses": [200]
          },
          "unhealthy": {
            "interval": 1,
            "tcp_failures": 2,
            "timeouts": 3,
            "http_failures": 3,
            "http_statuses": [429, 404, 500, 502, 503, 504]
          }
        }
      }
    }
  }'

# 2. Create a Route for validation traffic
curl -k "https://localhost:7443/apisix/admin/routes/health-check-route?gateway_group_id={gateway_group_id}" -X PUT \
  -H "X-API-KEY: ${API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "health-check-route",
    "methods": ["GET"],
    "paths": ["/status/200"],
    "service_id": "upstream-with-health-check"
  }'

Configure Passive Health Check​

Passive health checks complement active ones by monitoring real-world traffic patterns to detect failures that active probes might miss.

# Update the Service to add passive checks alongside active checks
curl -k "https://localhost:7443/apisix/admin/services/upstream-with-health-check?gateway_group_id={gateway_group_id}" -X PUT \
  -H "X-API-KEY: ${API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "upstream-with-health-check",
    "upstream": {
      "type": "roundrobin",
      "nodes": [
        {
          "host": "192.168.215.1",
          "port": 18083,
          "weight": 100
        },
        {
          "host": "192.168.215.1",
          "port": 18084,
          "weight": 100
        }
      ],
      "checks": {
        "active": {
          "type": "http",
          "http_path": "/status/200",
          "healthy": {
            "interval": 30,
            "successes": 1,
            "http_statuses": [200]
          },
          "unhealthy": {
            "interval": 30,
            "http_failures": 3,
            "tcp_failures": 2,
            "timeouts": 3,
            "http_statuses": [429, 404, 500, 502, 503, 504]
          }
        },
        "passive": {
          "healthy": {
            "http_statuses": [200, 201],
            "successes": 1
          },
          "unhealthy": {
            "http_statuses": [404, 500, 502, 503],
            "http_failures": 3,
            "tcp_failures": 2,
            "timeouts": 3
          }
        }
      }
    }
  }'

Passive health checks are reactive and require some amount of client traffic to detect failures.

Combine Active and Passive Health Checks​

For maximum reliability, it is recommended to use both active and passive health checks together.

Validate the Configuration​

  1. Simulate a failure: Stop one of your backend upstream nodes, for example docker stop tm-health-1.
  2. Observe the Gateway: Wait for the health check to detect the failure based on the configured thresholds.
  3. Check availability: Send requests to the route. The Gateway should continue to return 200 OK while at least one node is healthy.
  4. Inspect health state: Query the Control API from a temporary container that shares the gateway container's network. The gateway image does not include curl:
docker run --rm --network container:gateway curlimages/curl:8.10.1 -s http://127.0.0.1:9090/v1/healthcheck

In the local validation environment, the health state was reported under the upstream path, such as /apisix/upstreams/upstream-with-health-check.

Each entry names the resource the checker belongs to, the checker type, and the nodes it tracks with their current health state. From API7 Enterprise 3.9.20 and 3.10.7, an entry created by a plugin that runs health checks of its own carries two more fields. plugin names that plugin, and meta describes what the checker stands for. For AI Proxy Multi, meta.instance names the LLM instance being checked.

A route or a service can own more than one checker, which the single object returned by /v1/healthcheck/{src_type}/{src_id} cannot express. From API7 Enterprise 3.9.20 and 3.10.7, request the checkers sub-resource instead to get every checker as an array:

docker run --rm --network container:gateway curlimages/curl:8.10.1 -s http://127.0.0.1:9090/v1/healthcheck/services/upstream-with-health-check/checkers

A resource that has no health check at all is not an error here, it simply owns an empty set of checkers and the response is [].

while true; do curl -s -o /dev/null -w '%{http_code}\n' "http://127.0.0.1:9080/status/200"; sleep 1; done

Troubleshooting​

  • Health Checks Not Working: Check that the http_path exists on your backend and that the host and port are correctly configured.
  • Firewall Issues: Ensure the Gateway nodes have permission to reach the upstream nodes' health check endpoints.
  • Wait Time: If a node is not being removed immediately, check your interval and http_failures/successes settings.
  • DNS Issues: If you are using hostnames in your upstream nodes, ensure the Gateway nodes can resolve them.

Next Steps​