Skip to main content
API7.ai Logo

Configure Upstream Health Checks

Health checks help the gateway identify responsive upstream nodes and detect when a node fails or recovers.

There are two types of health checks:

  • Active checks: the gateway proactively and periodically sends requests to upstream services and determines the health of those based on the responses to these requests.
  • Passive checks: the gateway determines the health of upstream services based on how they respond to client requests, without proactively probing.

The examples configure both types against two NGINX services, then distinguish probe failure from endpoint removal.

Prerequisite​

  1. Complete Set Up Ingress Controller and Gateway.

Start Example Upstream Services​

Create a Kubernetes manifest file for the deployment of two NGINX instances as example upstream services:

nginx.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: aic
  name: nginx1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx1
  template:
    metadata:
      labels:
        app: nginx1
    spec:
      containers:
        - name: nginx
          image: nginx
          ports:
            - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  namespace: aic
  name: nginx1
spec:
  selector:
    app: nginx1
  ports:
    - name: http
      port: 8080
      targetPort: 80
---
apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: aic
  name: nginx2
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx2
  template:
    metadata:
      labels:
        app: nginx2
    spec:
      containers:
        - name: nginx
          image: nginx
          ports:
            - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  namespace: aic
  name: nginx2
spec:
  selector:
    app: nginx2
  ports:
    - name: http
      port: 8081
      targetPort: 80

Apply the configuration to your cluster:

kubectl apply -f nginx.yaml

Expose the NGINX service ports to your local machine by port forwarding:

kubectl port-forward svc/nginx1 -n aic 8080:8080 &
kubectl port-forward svc/nginx2 -n aic 8081:8081 &

Verify both NGINX instances are running:

for port in 8080 8081; do
  curl -s "http://127.0.0.1:$port" | grep -q "Welcome to nginx" &&
  echo "NGINX welcome page available on port $port."
done

You should see the following response:

NGINX welcome page available on port 8080.
NGINX welcome page available on port 8081.

Enable Control API​

Upgrade your gateway to enable Control API.

Export all values (including defaults):

helm get values -n aic apisix --all > values.yaml

In the values file, update the following section values as such:

values.yaml
apisix:
  enabled: true
control:
  enabled: true

Upgrade the release:

helm upgrade -n aic apisix apisix/apisix --version 2.18.0 -f values.yaml

Port-forward the Control API port:

kubectl port-forward service/apisix-control -n aic 9090:9090 &

Configure Active Health Checks​

Active checks determine the health of upstream services by periodically sending requests, or probes, to the services and seeing how they respond.

In this section, you will find two examples to understand:

  • How changes in upstream statuses can be detected by active checks.
  • How the gateway forwards client requests to upstream services when all upstream statuses are unhealthy.

Example: Status Change in Upstream Services​

The following example distinguishes four states: both upstreams are healthy, active probes fail and recover while endpoints remain available, endpoints are removed, and the upstreams are restored.

Create a route to the two services. Configure probes for healthy nodes to run every 2 seconds and probes for unhealthy nodes to run every second:

active-health-checks.yaml
apiVersion: apisix.apache.org/v1alpha1
kind: BackendTrafficPolicy
metadata:
  namespace: aic
  name: nginx1
spec:
  targetRefs:
    - name: nginx1
      kind: Service
      group: ""
      # Apply this policy only to the named Service port.
      sectionName: http
  healthCheck:
    active:
      type: http
      httpPath: /
      healthy:
        interval: 2s
        successes: 1
      unhealthy:
        interval: 1s
        timeouts: 3
---
apiVersion: apisix.apache.org/v1alpha1
kind: BackendTrafficPolicy
metadata:
  namespace: aic
  name: nginx2
spec:
  targetRefs:
    - name: nginx2
      kind: Service
      group: ""
      sectionName: http
  healthCheck:
    active:
      type: http
      httpPath: /
      healthy:
        interval: 2s
        successes: 1
      unhealthy:
        interval: 1s
        timeouts: 3
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  namespace: aic
  name: example-hc-route
spec:
  parentRefs:
    - name: apisix
  rules:
    - matches:
        - path:
            type: Exact
            value: /
      backendRefs:
        - name: nginx1
          port: 8080
          weight: 1
        - name: nginx2
          port: 8081
          weight: 1

Apply the configuration to your cluster:

kubectl apply -f active-health-checks.yaml

sectionName selects a Service port by spec.ports[].name. If the named port does not exist, the policy is not attached. Its status reports Accepted=False with a message such as No section name http found in Service aic/nginx1. Check policy status with:

kubectl describe backendtrafficpolicy nginx1 -n aic

Verify​

The gateway normally selects healthy nodes. When every node is marked unhealthy but endpoints still exist, the balancer can fall back to those nodes; this differs from having no endpoints at all.

You will be verifying the above configurations to understand how the upstream health checks respond in different scenarios:

Before proceeding, expose the gateway’s service port to your local machine:

# replace with your gateway’s service name
kubectl port-forward svc/<gateway-service-name> -n aic 9080:80 &

Verify Both Upstream Services Are Healthy​

Generate traffic through the route while the controller applies the configuration and initializes the upstream checkers. Poll http://127.0.0.1:9090/v1/healthcheck until both Service upstreams have healthy nodes. A single request does not guarantee that every worker has initialized its checker.

for attempt in $(seq 10); do
  curl -sS --max-time 5 "http://127.0.0.1:9080/" -o /dev/null
  curl -sS --max-time 5 "http://127.0.0.1:9090/v1/healthcheck"
  sleep 2
done

Repeat the requests if the checkers are still initializing. Confirm both upstreams are healthy before starting the failure test.

The response contains separate upstream checker entries for nginx1 and nginx2, each with its own nodes and counters. Names and pod IPs are generated; do not expect one route-named checker containing both Services.

Verify Probe Failure and Recovery​

Keep both Deployments running so their endpoints remain present. Select the resource kind for the configuration tab you applied:

# Gateway API:
export HEALTH_RESOURCE="backendtrafficpolicies.apisix.apache.org"
# For the APISIX CRD tab, use this instead:
# export HEALTH_RESOURCE="apisixupstreams.apisix.apache.org"

Make the first upstream's probes request /404, which returns an unhealthy HTTP status, and set its failure threshold:

kubectl patch "$HEALTH_RESOURCE" nginx1 -n aic --type merge \
  -p '{"spec":{"healthCheck":{"active":{"httpPath":"/404","unhealthy":{"httpFailures":2}}}}}'
curl -i "http://127.0.0.1:9080/"
curl "http://127.0.0.1:9090/v1/healthcheck"

Repeat traffic and checker requests until the nginx1 checker reports HTTP failures and becomes unhealthy while nginx2 remains healthy. EndpointSlices should still contain both backends. Counters and timing vary; the transition is not guaranteed after a single request.

Apply the same change to the second upstream:

kubectl patch "$HEALTH_RESOURCE" nginx2 -n aic --type merge \
  -p '{"spec":{"healthCheck":{"active":{"httpPath":"/404","unhealthy":{"httpFailures":2}}}}}'
curl -i "http://127.0.0.1:9080/"
curl "http://127.0.0.1:9090/v1/healthcheck"

Poll until both checkers report unhealthy nodes. Requests to / can still succeed through the all-unhealthy fallback because the servers and their endpoints remain available.

Restore the probe path on both resources:

for service in nginx1 nginx2; do
  kubectl patch "$HEALTH_RESOURCE" "$service" -n aic --type merge \
    -p '{"spec":{"healthCheck":{"active":{"httpPath":"/"}}}}'
done
curl -i "http://127.0.0.1:9080/"
curl "http://127.0.0.1:9090/v1/healthcheck"

Wait for both upstream checkers to become healthy again before the endpoint-removal test. This verifies HTTP probe failure and recovery, not timeout counters or endpoint removal.

Verify Endpoint Removal​

Scale one backend to zero:

kubectl scale deployment nginx1 -n aic --replicas=0

Wait for its EndpointSlices and gateway configuration to converge. The removed endpoints disappear from the upstream; removal does not demonstrate three failed probes or increment a timeout counter.

With HTTPRoute, the remaining usable backend can serve requests. With the ApisixRoute example, the empty backend retains its weight, so repeated requests can include both successful responses and 503 responses.

Scale the second backend to zero:

kubectl scale deployment nginx2 -n aic --replicas=0

After convergence, neither backend has a usable endpoint and requests return 503, not 502. To demonstrate probe timeouts instead, keep endpoints present while preventing their probe responses.

Restore the Upstreams​

kubectl scale deployment nginx1 -n aic --replicas=1
kubectl scale deployment nginx2 -n aic --replicas=1
kubectl rollout status deployment/nginx1 -n aic --timeout=120s
kubectl rollout status deployment/nginx2 -n aic --timeout=120s

Wait for endpoints, gateway configuration, and healthy checker status before continuing. Pod readiness alone does not prove the gateway has applied the recovered endpoints.

Example: Forward Requests When Statuses Are Unhealthy​

This example tests the all-unhealthy fallback while both Services still have endpoints.

Create a route to the two services. Configure healthy probes to run every 2 seconds and unhealthy probes every second:

active-health-checks.yaml
apiVersion: apisix.apache.org/v1alpha1
kind: BackendTrafficPolicy
metadata:
  namespace: aic
  name: nginx1
spec:
  targetRefs:
    - name: nginx1
      kind: Service
      group: ""
  healthCheck:
    active:
      type: http
      httpPath: /404
      healthy:
        interval: 2s
        successes: 1
      unhealthy:
        interval: 1s
        httpFailures: 2
---
apiVersion: apisix.apache.org/v1alpha1
kind: BackendTrafficPolicy
metadata:
  namespace: aic
  name: nginx2
spec:
  targetRefs:
    - name: nginx2
      kind: Service
      group: ""
  healthCheck:
    active:
      type: http
      httpPath: /404
      healthy:
        interval: 2s
        successes: 1
      unhealthy:
        interval: 1s
        httpFailures: 2
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  namespace: aic
  name: example-hc-route
spec:
  parentRefs:
    - name: apisix
  rules:
    - matches:
        - path:
            type: Exact
            value: /
      backendRefs:
        - name: nginx1
          port: 8080
          weight: 1
        - name: nginx2
          port: 8081
          weight: 1

Apply the configuration to your cluster:

kubectl apply -f active-health-checks.yaml

Verify​

Expose the gateway’s service port to your local machine:

# replace with your gateway’s service name
kubectl port-forward svc/<gateway-service-name> -n aic 9080:80 &

Send one request to initialize the upstream checkers. This request goes to /; the active probes then check /404 on their own schedule, so the first response does not demonstrate the fallback:

curl -i "http://127.0.0.1:9080/"

Poll the health check endpoint until both checkers report unhealthy nodes after reaching the two-failure threshold:

for attempt in $(seq 10); do
  curl -sS --max-time 5 "http://127.0.0.1:9090/v1/healthcheck"
  sleep 2
done

Repeat the check if the probes are still running. Once both checkers are unhealthy, the response looks similar to the following:

[
  {
    "name": "<generated-upstream-checker-for-nginx1>",
    "nodes": [
      {
        "counter": {
          "timeout_failure": 0,
          "http_failure": 2,
          "success": 0,
          "tcp_failure": 0
        },
        "port": 80,
        "ip": "172.25.0.5",
        "status": "unhealthy"
      }
    ],
    "type": "http"
  },
  {
    "name": "<generated-upstream-checker-for-nginx2>",
    "nodes": [
      {
        "counter": {
          "timeout_failure": 0,
          "http_failure": 2,
          "success": 0,
          "tcp_failure": 0
        },
        "port": 80,
        "ip": "172.25.0.4",
        "status": "unhealthy"
      }
    ],
    "type": "http"
  }
]

Send a request to the route to see if the gateway still forwards the request:

curl -i "http://127.0.0.1:9080/"

After both checkers show unhealthy nodes, a 200 response confirms fallback to an available endpoint.

Before testing passive failures, apply the configuration below. It restores the active probe path to /; wait until both checkers report healthy nodes before sending requests to /404.

Configure Passive Health Checks​

Pair passive checks with active checks so a node marked unhealthy by client traffic can be probed for recovery.

Example: Status Change in Upstream Services​

Create a route to the two services, and configure both active and passive health checks:

passive-health-checks.yaml
apiVersion: apisix.apache.org/v1alpha1
kind: BackendTrafficPolicy
metadata:
  namespace: aic
  name: nginx1
spec:
  targetRefs:
    - name: nginx1
      kind: Service
      group: ""
  healthCheck:
    active:
      type: http
      httpPath: /
      healthy:
        interval: 99999s
        successes: 1
      unhealthy:
        interval: 30s
    passive:
      healthy:
        httpCodes: [200, 201, 202, 300, 301, 302]
        successes: 1
      unhealthy:
        httpCodes: [429, 404, 500, 501, 502, 503, 504, 505]
        httpFailures: 3
---
apiVersion: apisix.apache.org/v1alpha1
kind: BackendTrafficPolicy
metadata:
  namespace: aic
  name: nginx2
spec:
  targetRefs:
    - name: nginx2
      kind: Service
      group: ""
  healthCheck:
    active:
      type: http
      httpPath: /
      healthy:
        interval: 99999s
        successes: 1
      unhealthy:
        interval: 30s
    passive:
      healthy:
        httpCodes: [200, 201, 202, 300, 301, 302]
        successes: 1
      unhealthy:
        httpCodes: [429, 404, 500, 501, 502, 503, 504, 505]
        httpFailures: 3
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  namespace: aic
  name: example-hc-route
spec:
  parentRefs:
    - name: apisix
  rules:
    - matches:
        - path:
            type: Exact
            value: /404
      backendRefs:
        - name: nginx1
          port: 8080
          weight: 1
        - name: nginx2
          port: 8081
          weight: 1

Apply the configuration to your cluster:

kubectl apply -f passive-health-checks.yaml

Verify​

Expose the gateway’s service port to your local machine:

# replace with your gateway’s service name
kubectl port-forward svc/<gateway-service-name> -n aic 9080:80 &

Send a request to the route to start health checks:

curl -i "http://127.0.0.1:9080/404"

You should see an HTTP/1.1 404 Not Found response.

Send a request to the health check endpoint:

curl "http://127.0.0.1:9090/v1/healthcheck"

You should see a response similar to the following:

[
  {
    "name": "<generated-upstream-checker-for-nginx1>",
    "nodes": [
      {
        "counter": {
          "timeout_failure": 0,
          "http_failure": 1,
          "success": 0,
          "tcp_failure": 0
        },
        "port": 80,
        "ip": "172.25.0.5",
        "status": "mostly_healthy"
      }
    ],
    "type": "http"
  },
  {
    "name": "<generated-upstream-checker-for-nginx2>",
    "nodes": [
      {
        "counter": {
          "timeout_failure": 0,
          "http_failure": 0,
          "success": 0,
          "tcp_failure": 0
        },
        "port": 80,
        "ip": "172.25.0.4",
        "status": "healthy"
      }
    ],
    "type": "http"
  }
]

❶ http_failure has a count of 1 due to the previous request with a 404 response.

❷ mostly_healthy status means the current node status is healthy, but the gateway starts to receive unhealthy indications during health checks.

Generate consecutive requests to invoke 404 responses:

resp=$(seq 10 | xargs -I{} curl "http://127.0.0.1:9080/404" -o /dev/null -s -w "%{http_code}\n") && \
  count=$(echo "$resp" | grep "404" | wc -l) && \
  echo "Invoked $count responses with 404 status code."

Send a request to the health check endpoint:

curl "http://127.0.0.1:9090/v1/healthcheck"

You should see a response similar to the following:

[
  {
    "name": "<generated-upstream-checker-for-nginx1>",
    "nodes": [
      {
        "counter": {
          "timeout_failure": 0,
          "http_failure": 3,
          "success": 0,
          "tcp_failure": 0
        },
        "port": 80,
        "ip": "172.25.0.4",
        "status": "unhealthy"
      }
    ],
    "type": "http"
  },
  {
    "name": "<generated-upstream-checker-for-nginx2>",
    "nodes": [
      {
        "counter": {
          "timeout_failure": 0,
          "http_failure": 4,
          "success": 0,
          "tcp_failure": 0
        },
        "port": 80,
        "ip": "172.25.0.5",
        "status": "unhealthy"
      }
    ],
    "type": "http"
  }
]

Wait at least 30 seconds for active checks to probe the upstream services at / and mark them as healthy. Then, send a request to the health check endpoint:

curl "http://127.0.0.1:9090/v1/healthcheck"

You should see a response similar to the following:

[
  {
    "name": "<generated-upstream-checker-for-nginx1>",
    "nodes": [
      {
        "counter": {
          "timeout_failure": 0,
          "http_failure": 0,
          "success": 1,
          "tcp_failure": 0
        },
        "port": 80,
        "ip": "172.25.0.4",
        "status": "healthy"
      }
    ],
    "type": "http"
  },
  {
    "name": "<generated-upstream-checker-for-nginx2>",
    "nodes": [
      {
        "counter": {
          "timeout_failure": 0,
          "http_failure": 0,
          "success": 1,
          "tcp_failure": 0
        },
        "port": 80,
        "ip": "172.25.0.5",
        "status": "healthy"
      }
    ],
    "type": "http"
  }
]