Configure Fault Injection

Chaos engineering is essential for building reliable systems. The fault-injection plugin allows you to test your application's resilience by simulating failures and latency at the Gateway level before requests reach your upstream services.

Prerequisites

• An API7 Enterprise instance is running.
• A Gateway Group is created and a Gateway instance is running.
• A token from the Dashboard.

Configure HTTP Aborts

You can return specific HTTP status codes to a percentage of requests to simulate service failures.

Admin API

curl -k "https://localhost:7443/apisix/admin/services/fault-injection-service?gateway_group_id={group_id}" -X PUT \
  -H "X-API-KEY: ${API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "fault-injection-service",
    "upstream": {
      "type": "roundrobin",
      "scheme": "http",
      "nodes": [
        {
          "host": "httpbin.org",
          "port": 80,
          "weight": 100
        }
      ]
    }
  }'

curl -k "https://localhost:7443/apisix/admin/routes/fault-injection-route?gateway_group_id={group_id}" -X PUT \
  -H "X-API-KEY: ${API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "fault-injection-route",
    "paths": ["/get"],
    "service_id": "fault-injection-service",
    "plugins": {
      "fault-injection": {
        "abort": {
          "http_status": 503,
          "body": "Service Temporarily Unavailable",
          "percentage": 10
        }
      }
    }
  }'

ADC

adc.yaml

services:
  - name: fault-injection-service
    upstream:
      nodes:
        - host: httpbin.org
          port: 80
          weight: 1
    routes:
      - name: fault-injection-route
        uris:
          - /get
        plugins:
          fault-injection:
            abort:
              http_status: 503
              body: "Service Temporarily Unavailable"
              percentage: 10

adc sync -f adc.yaml

Configure Response Delays

You can also simulate slow network conditions or backend latency by delaying responses.

Admin API

curl -k "https://localhost:7443/apisix/admin/services/delay-fault-injection-service?gateway_group_id={group_id}" -X PUT \
  -H "X-API-KEY: ${API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "delay-fault-injection-service",
    "upstream": {
      "type": "roundrobin",
      "scheme": "http",
      "nodes": [
        {
          "host": "httpbin.org",
          "port": 80,
          "weight": 100
        }
      ]
    }
  }'

curl -k "https://localhost:7443/apisix/admin/routes/delay-fault-injection-route?gateway_group_id={group_id}" -X PUT \
  -H "X-API-KEY: ${API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "delay-fault-injection-route",
    "paths": ["/delay/5"],
    "service_id": "delay-fault-injection-service",
    "plugins": {
      "fault-injection": {
        "delay": {
          "duration": 5,
          "percentage": 20
        }
      }
    }
  }'

ADC

adc.yaml

services:
  - name: delay-fault-injection-service
    upstream:
      nodes:
        - host: httpbin.org
          port: 80
          weight: 1
    routes:
      - name: delay-fault-injection-route
        uris:
          - /delay/5
        plugins:
          fault-injection:
            delay:
              duration: 5
              percentage: 20

adc sync -f adc.yaml

Validate the Configuration

To validate an abort configuration with a 10% percentage, send multiple requests:

for i in {1..20}; do curl -i "http://127.0.0.1:9080/get"; done

You should see some requests receiving the injected 503 Service Temporarily Unavailable response. For delays, the curl command should hang for the configured duration when the fault is injected.

To verify the delay example, measure a few requests against the delay route:

time curl -i "http://127.0.0.1:9080/delay/5"

If the route returns 404 immediately after you apply the configuration, wait a few seconds for the latest configuration to reach the gateway and retry.

Next Steps

• Configure Proxy Mirror — test your application with real production traffic.
• Configure Response Rewrite — modify response data before returning to the client.
• Configure Rate Limiting — protect your system from traffic spikes.