fault-injection

The fault-injection plugin is designed to test your application's resiliency by simulating controlled faults or delays. It executes before other configured plugins, ensuring that faults are applied consistently. This makes it ideal for scenarios like chaos engineering, where the behavior of your system under failure conditions is analyzed.

The plugin supports two key actions: abort, which immediately terminates a request with a specified HTTP status code (e.g., 503 Service Unavailable), skipping all subsequent plugins; and delay, which introduces a specified delay before processing the request further. These features allow you to simulate scenarios such as service outages or latency, helping you validate error-handling logic and improve system reliability.

Examples

The examples below demonstrate how you can configure the fault-injection plugin for different scenarios.

Inject Faults

The following example demonstrate how you can configure the fault-injection plugin on a route to intercept further request sending, and respond with a specific HTTP code.

Create a route using the fault-injection plugin with the abort action to respond any request with 404 and the specified response body:

Admin API
ADC
Ingress Controller

curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d '{
    "id": "fault-injection-route",
    "uri": "/anything",
    "plugins": {
      "fault-injection": {
        "abort": {
          "http_status": 404,
          "body": "APISIX Fault Injection"
        }
      }
    },
    "upstream": {
      "type": "roundrobin",
      "nodes": {
        "httpbin.org": 1
      }
    }
  }'

adc.yaml
services:
  - name: httpbin
    routes:
      - name: fault-injection-route
        uris:
          - /anything
        plugins:
          fault-injection:
            abort:
              http_status: 404
              body: "APISIX Fault Injection"
    upstream:
      type: roundrobin
      nodes:
        - host: httpbin.org
          port: 80
          weight: 1

Synchronize the configuration to the gateway:

adc sync -f adc.yaml

Gateway API
APISIX CRD

fault-injection-ic.yaml
apiVersion: v1
kind: Service
metadata:
  namespace: aic
  name: httpbin-external-domain
spec:
  type: ExternalName
  externalName: httpbin.org
---
apiVersion: apisix.apache.org/v1alpha1
kind: PluginConfig
metadata:
  namespace: aic
  name: fault-injection-plugin-config
spec:
  plugins:
    - name: fault-injection
      config:
        abort:
          http_status: 404
          body: "APISIX Fault Injection"
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  namespace: aic
  name: fault-injection-route
spec:
  parentRefs:
    - name: apisix
  rules:
    - matches:
        - path:
            type: Exact
            value: /anything
      filters:
        - type: ExtensionRef
          extensionRef:
            group: apisix.apache.org
            kind: PluginConfig
            name: fault-injection-plugin-config
      backendRefs:
        - name: httpbin-external-domain
          port: 80

Apply the configuration to your cluster:

kubectl apply -f fault-injection-ic.yaml

fault-injection-ic.yaml
apiVersion: apisix.apache.org/v2
kind: ApisixUpstream
metadata:
  namespace: aic
  name: httpbin-external-domain
spec:
  ingressClassName: apisix
  externalNodes:
  - type: Domain
    name: httpbin.org
---
apiVersion: apisix.apache.org/v2
kind: ApisixRoute
metadata:
  namespace: aic
  name: fault-injection-route
spec:
  ingressClassName: apisix
  http:
    - name: fault-injection-route
      match:
        paths:
          - /anything
      upstreams:
      - name: httpbin-external-domain
      plugins:
      - name: fault-injection
        enable: true
        config:
          abort:
            http_status: 404
            body: "APISIX Fault Injection"

Apply the configuration to your cluster:

kubectl apply -f fault-injection-ic.yaml

Send a request to the route:

curl -i "http://127.0.0.1:9080/anything"

You should receive an HTTP/1.1 404 Not Found response and see the following response body, without the request being forwarded to the upstream service:

APISIX Fault Injection

Inject Latencies

The following example demonstrate how you can configure the fault-injection plugin on a route to inject request latencies.

Create a route using the fault-injection plugin with the delay action to delay response sending for 3 seconds:

Admin API
ADC
Ingress Controller

curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d '{
    "id": "fault-injection-route",
    "uri": "/anything",
    "plugins": {
      "fault-injection": {
        "delay": {
          "duration": 3
        }
      }
    },
    "upstream": {
      "type": "roundrobin",
      "nodes": {
        "httpbin.org": 1
      }
    }
  }'

adc.yaml
services:
  - name: httpbin
    routes:
      - name: fault-injection-route
        uris:
          - /anything
        plugins:
          fault-injection:
            delay:
              duration: 3
    upstream:
      type: roundrobin
      nodes:
        - host: httpbin.org
          port: 80
          weight: 1

Synchronize the configuration to the gateway:

adc sync -f adc.yaml

Gateway API
APISIX CRD

fault-injection-ic.yaml
apiVersion: v1
kind: Service
metadata:
  namespace: aic
  name: httpbin-external-domain
spec:
  type: ExternalName
  externalName: httpbin.org
---
apiVersion: apisix.apache.org/v1alpha1
kind: PluginConfig
metadata:
  namespace: aic
  name: fault-injection-plugin-config
spec:
  plugins:
    - name: fault-injection
      config:
        delay:
          duration: 3
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  namespace: aic
  name: fault-injection-route
spec:
  parentRefs:
    - name: apisix
  rules:
    - matches:
        - path:
            type: Exact
            value: /anything
      filters:
        - type: ExtensionRef
          extensionRef:
            group: apisix.apache.org
            kind: PluginConfig
            name: fault-injection-plugin-config
      backendRefs:
        - name: httpbin-external-domain
          port: 80

Apply the configuration to your cluster:

kubectl apply -f fault-injection-ic.yaml

fault-injection-ic.yaml
apiVersion: apisix.apache.org/v2
kind: ApisixUpstream
metadata:
  namespace: aic
  name: httpbin-external-domain
spec:
  ingressClassName: apisix
  externalNodes:
  - type: Domain
    name: httpbin.org
---
apiVersion: apisix.apache.org/v2
kind: ApisixRoute
metadata:
  namespace: aic
  name: fault-injection-route
spec:
  ingressClassName: apisix
  http:
    - name: fault-injection-route
      match:
        paths:
          - /anything
      upstreams:
      - name: httpbin-external-domain
      plugins:
      - name: fault-injection
        enable: true
        config:
          delay:
            duration: 3

Apply the configuration to your cluster:

kubectl apply -f fault-injection-ic.yaml

Send a request to the route and use time command to summarize of how long the request took to complete:

time curl -i "http://127.0.0.1:9080/anything"

You should receive an HTTP/1.1 200 OK response from the upstream service and see the following timing summary:

01s user
01s system
0% cpu
685 total

Inject Faults Conditionally

The following example demonstrate how you can configure the fault-injection plugin on a route to intercept further request sending, and respond with a specific HTTP code.

Create a route using the fault-injection plugin with the abort action as such:

Admin API
ADC
Ingress Controller

curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d '{
    "id": "fault-injection-route",
    "uri": "/anything",
    "plugins": {
      "fault-injection": {
        "abort": {
          "http_status": 404,
          "body": "APISIX Fault Injection",
          "headers": {
            "X-APISIX-Remote-Addr": "$remote_addr"
          },
          "vars": [
            [
              [ "arg_name","==","john" ]
            ]
          ]
        }
      }
    },
    "upstream": {
      "type": "roundrobin",
      "nodes": {
        "httpbin.org": 1
      }
    }
  }'

adc.yaml
services:
  - name: httpbin
    routes:
      - name: fault-injection-route
        uris:
          - /anything
        plugins:
          fault-injection:
            abort:
              http_status: 404
              body: "APISIX Fault Injection"
              headers:
                X-APISIX-Remote-Addr: $remote_addr
              vars:
                - - - arg_name
                    - "=="
                    - john
    upstream:
      type: roundrobin
      nodes:
        - host: httpbin.org
          port: 80
          weight: 1

Synchronize the configuration to the gateway:

adc sync -f adc.yaml

Gateway API
APISIX CRD

fault-injection-ic.yaml
apiVersion: v1
kind: Service
metadata:
  namespace: aic
  name: httpbin-external-domain
spec:
  type: ExternalName
  externalName: httpbin.org
---
apiVersion: apisix.apache.org/v1alpha1
kind: PluginConfig
metadata:
  namespace: aic
  name: fault-injection-plugin-config
spec:
  plugins:
    - name: fault-injection
      config:
        abort:
          http_status: 404
          body: "APISIX Fault Injection"
          headers:
            X-APISIX-Remote-Addr: $remote_addr
          vars:
            - - - arg_name
                - "=="
                - john
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  namespace: aic
  name: fault-injection-route
spec:
  parentRefs:
    - name: apisix
  rules:
    - matches:
        - path:
            type: Exact
            value: /anything
      filters:
        - type: ExtensionRef
          extensionRef:
            group: apisix.apache.org
            kind: PluginConfig
            name: fault-injection-plugin-config
      backendRefs:
        - name: httpbin-external-domain
          port: 80

Apply the configuration to your cluster:

kubectl apply -f fault-injection-ic.yaml

fault-injection-ic.yaml
apiVersion: apisix.apache.org/v2
kind: ApisixUpstream
metadata:
  namespace: aic
  name: httpbin-external-domain
spec:
  ingressClassName: apisix
  externalNodes:
  - type: Domain
    name: httpbin.org
---
apiVersion: apisix.apache.org/v2
kind: ApisixRoute
metadata:
  namespace: aic
  name: fault-injection-route
spec:
  ingressClassName: apisix
  http:
    - name: fault-injection-route
      match:
        paths:
          - /anything
      upstreams:
      - name: httpbin-external-domain
      plugins:
      - name: fault-injection
        enable: true
        config:
          abort:
            http_status: 404
            body: "APISIX Fault Injection"
            headers:
              X-APISIX-Remote-Addr: $remote_addr
            vars:
              - - - arg_name
                  - "=="
                  - john

Apply the configuration to your cluster:

kubectl apply -f fault-injection-ic.yaml

❶ Respond requests with HTTP status code 404.

❷ Respond requests with APISIX Fault Injection as the body.

❸ Respond requests with header X-APISIX-Remote-Addr and the IP the request originates from.

❹ Respond requests with the above specifications only if the URL parameter name value is john.

Send a request to the route with the URL parameter name being john:

curl -i "http://127.0.0.1:9080/anything?name=john"

You should receive an HTTP/1.1 404 Not Found response:

HTTP/1.1 404 Not Found
...
X-APISIX-Remote-Addr: 192.168.65.1

APISIX Fault Injection

Send a request to the route with the URL parameter name being a different value:

curl -i "http://127.0.0.1:9080/anything?name=jane"

You should receive an HTTP/1.1 200 OK response from the upstream service without the injections.

Examples​

Inject Faults​

Inject Latencies​

Inject Faults Conditionally​

Examples

Inject Faults

Inject Latencies

Inject Faults Conditionally