fault-injection
The fault-injection plugin tests your application's resiliency by simulating controlled faults or delays. It runs in the rewrite phase. Within the same rewrite execution pass, plugins run in descending priority order, and _meta.priority can override a plugin's default priority. The fault-injection default of 11000 places it before lower-priority rewrite handlers and after higher-priority rewrite handlers in that pass. Global-rule plugins and route or service plugins run in separate rewrite passes. This makes the plugin useful for scenarios such as chaos engineering, where you analyze how your system behaves under failure conditions.
The plugin supports two actions. abort immediately ends request processing with a specified HTTP status code, such as 503 Service Unavailable, so later rewrite and access handlers do not run. Response-filter and log handlers can still run for the injected response. delay pauses request processing for a specified duration. These actions simulate service outages or latency so that you can validate error-handling logic and improve system reliability.
Examples
The examples below demonstrate how you can configure the fault-injection plugin for different scenarios.
Inject Faults
The following example demonstrate how you can configure the fault-injection plugin on a route to intercept further request sending, and respond with a specific HTTP code.
Create a route using the fault-injection plugin with the abort action to respond any request with 404 and the specified response body:
- Admin API
- ADC
- Ingress Controller
curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
-H "X-API-KEY: ${ADMIN_API_KEY}" \
-d '{
"id": "fault-injection-route",
"uri": "/anything",
"plugins": {
"fault-injection": {
"abort": {
"http_status": 404,
"body": "APISIX Fault Injection"
}
}
},
"upstream": {
"type": "roundrobin",
"nodes": {
"httpbin.org": 1
}
}
}'
services:
- name: httpbin
routes:
- name: fault-injection-route
uris:
- /anything
plugins:
fault-injection:
abort:
http_status: 404
body: "APISIX Fault Injection"
upstream:
type: roundrobin
nodes:
- host: httpbin.org
port: 80
weight: 1
Preview the changes and confirm that they only affect resources for this guide:
adc diff -f adc.yaml \
--include-resource-type service \
--label-selector docs-example=fault-injection
Synchronize the configuration to the gateway:
adc sync -f adc.yaml \
--include-resource-type service \
--label-selector docs-example=fault-injection
- Gateway API
- APISIX CRD
apiVersion: v1
kind: Service
metadata:
namespace: aic
name: httpbin-external-domain
spec:
type: ExternalName
externalName: httpbin.org
---
apiVersion: apisix.apache.org/v1alpha1
kind: PluginConfig
metadata:
namespace: aic
name: fault-injection-plugin-config
spec:
plugins:
- name: fault-injection
config:
abort:
http_status: 404
body: "APISIX Fault Injection"
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
namespace: aic
name: fault-injection-route
spec:
parentRefs:
- name: apisix
rules:
- matches:
- path:
type: Exact
value: /anything
filters:
- type: ExtensionRef
extensionRef:
group: apisix.apache.org
kind: PluginConfig
name: fault-injection-plugin-config
backendRefs:
- name: httpbin-external-domain
port: 80
Apply the configuration to your cluster:
kubectl apply -f fault-injection-ic.yaml
apiVersion: apisix.apache.org/v2
kind: ApisixUpstream
metadata:
namespace: aic
name: httpbin-external-domain
spec:
ingressClassName: apisix
externalNodes:
- type: Domain
name: httpbin.org
---
apiVersion: apisix.apache.org/v2
kind: ApisixRoute
metadata:
namespace: aic
name: fault-injection-route
spec:
ingressClassName: apisix
http:
- name: fault-injection-route
match:
paths:
- /anything
upstreams:
- name: httpbin-external-domain
plugins:
- name: fault-injection
enable: true
config:
abort:
http_status: 404
body: "APISIX Fault Injection"
Apply the configuration to your cluster:
kubectl apply -f fault-injection-ic.yaml
Send a request to the route:
curl -i "http://127.0.0.1:9080/anything"
You should receive an HTTP/1.1 404 Not Found response and see the following response body, without the request being forwarded to the upstream service:
APISIX Fault Injection
Inject Latencies
The following example demonstrate how you can configure the fault-injection plugin on a route to inject request latencies.
Create a route using the fault-injection plugin with the delay action to delay response sending for 3 seconds:
- Admin API
- ADC
- Ingress Controller
curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
-H "X-API-KEY: ${ADMIN_API_KEY}" \
-d '{
"id": "fault-injection-route",
"uri": "/anything",
"plugins": {
"fault-injection": {
"delay": {
"duration": 3
}
}
},
"upstream": {
"type": "roundrobin",
"nodes": {
"httpbin.org": 1
}
}
}'
services:
- name: httpbin
routes:
- name: fault-injection-route
uris:
- /anything
plugins:
fault-injection:
delay:
duration: 3
upstream:
type: roundrobin
nodes:
- host: httpbin.org
port: 80
weight: 1
Preview the changes and confirm that they only affect resources for this guide:
adc diff -f adc.yaml \
--include-resource-type service \
--label-selector docs-example=fault-injection
Synchronize the configuration to the gateway:
adc sync -f adc.yaml \
--include-resource-type service \
--label-selector docs-example=fault-injection
- Gateway API
- APISIX CRD
apiVersion: v1
kind: Service
metadata:
namespace: aic
name: httpbin-external-domain
spec:
type: ExternalName
externalName: httpbin.org
---
apiVersion: apisix.apache.org/v1alpha1
kind: PluginConfig
metadata:
namespace: aic
name: fault-injection-plugin-config
spec:
plugins:
- name: fault-injection
config:
delay:
duration: 3
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
namespace: aic
name: fault-injection-route
spec:
parentRefs:
- name: apisix
rules:
- matches:
- path:
type: Exact
value: /anything
filters:
- type: ExtensionRef
extensionRef:
group: apisix.apache.org
kind: PluginConfig
name: fault-injection-plugin-config
backendRefs:
- name: httpbin-external-domain
port: 80
Apply the configuration to your cluster:
kubectl apply -f fault-injection-ic.yaml
apiVersion: apisix.apache.org/v2
kind: ApisixUpstream
metadata:
namespace: aic
name: httpbin-external-domain
spec:
ingressClassName: apisix
externalNodes:
- type: Domain
name: httpbin.org
---
apiVersion: apisix.apache.org/v2
kind: ApisixRoute
metadata:
namespace: aic
name: fault-injection-route
spec:
ingressClassName: apisix
http:
- name: fault-injection-route
match:
paths:
- /anything
upstreams:
- name: httpbin-external-domain
plugins:
- name: fault-injection
enable: true
config:
delay:
duration: 3
Apply the configuration to your cluster:
kubectl apply -f fault-injection-ic.yaml
Send a request to the route and use time command to summarize of how long the request took to complete:
time curl -i "http://127.0.0.1:9080/anything"
You should receive an HTTP/1.1 200 OK response from the upstream service and see the following timing summary:
0.01s user
0.01s system
0% cpu
3.685 total
Inject Faults Conditionally
The following example demonstrate how you can configure the fault-injection plugin on a route to intercept further request sending, and respond with a specific HTTP code.
Create a route using the fault-injection plugin with the abort action as such:
- Admin API
- ADC
- Ingress Controller
curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
-H "X-API-KEY: ${ADMIN_API_KEY}" \
-d '{
"id": "fault-injection-route",
"uri": "/anything",
"plugins": {
"fault-injection": {
"abort": {
"http_status": 404,
"body": "APISIX Fault Injection",
"headers": {
"X-APISIX-Remote-Addr": "$remote_addr"
},
"vars": [
[
[ "arg_name","==","john" ]
]
]
}
}
},
"upstream": {
"type": "roundrobin",
"nodes": {
"httpbin.org": 1
}
}
}'
services:
- name: httpbin
routes:
- name: fault-injection-route
uris:
- /anything
plugins:
fault-injection:
abort:
http_status: 404
body: "APISIX Fault Injection"
headers:
X-APISIX-Remote-Addr: $remote_addr
vars:
- - - arg_name
- "=="
- john
upstream:
type: roundrobin
nodes:
- host: httpbin.org
port: 80
weight: 1
Preview the changes and confirm that they only affect resources for this guide:
adc diff -f adc.yaml \
--include-resource-type service \
--label-selector docs-example=fault-injection
Synchronize the configuration to the gateway:
adc sync -f adc.yaml \
--include-resource-type service \
--label-selector docs-example=fault-injection
- Gateway API
- APISIX CRD
apiVersion: v1
kind: Service
metadata:
namespace: aic
name: httpbin-external-domain
spec:
type: ExternalName
externalName: httpbin.org
---
apiVersion: apisix.apache.org/v1alpha1
kind: PluginConfig
metadata:
namespace: aic
name: fault-injection-plugin-config
spec:
plugins:
- name: fault-injection
config:
abort:
http_status: 404
body: "APISIX Fault Injection"
headers:
X-APISIX-Remote-Addr: $remote_addr
vars:
- - - arg_name
- "=="
- john
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
namespace: aic
name: fault-injection-route
spec:
parentRefs:
- name: apisix
rules:
- matches:
- path:
type: Exact
value: /anything
filters:
- type: ExtensionRef
extensionRef:
group: apisix.apache.org
kind: PluginConfig
name: fault-injection-plugin-config
backendRefs:
- name: httpbin-external-domain
port: 80
Apply the configuration to your cluster:
kubectl apply -f fault-injection-ic.yaml
apiVersion: apisix.apache.org/v2
kind: ApisixUpstream
metadata:
namespace: aic
name: httpbin-external-domain
spec:
ingressClassName: apisix
externalNodes:
- type: Domain
name: httpbin.org
---
apiVersion: apisix.apache.org/v2
kind: ApisixRoute
metadata:
namespace: aic
name: fault-injection-route
spec:
ingressClassName: apisix
http:
- name: fault-injection-route
match:
paths:
- /anything
upstreams:
- name: httpbin-external-domain
plugins:
- name: fault-injection
enable: true
config:
abort:
http_status: 404
body: "APISIX Fault Injection"
headers:
X-APISIX-Remote-Addr: $remote_addr
vars:
- - - arg_name
- "=="
- john
Apply the configuration to your cluster:
kubectl apply -f fault-injection-ic.yaml
❶ Respond requests with HTTP status code 404.
❷ Respond requests with APISIX Fault Injection as the body.
❸ Respond requests with header X-APISIX-Remote-Addr and the IP the request originates from.
❹ Respond requests with the above specifications only if the URL parameter name value is john.
Send a request to the route with the URL parameter name being john:
curl -i "http://127.0.0.1:9080/anything?name=john"
You should receive an HTTP/1.1 404 Not Found response:
HTTP/1.1 404 Not Found
...
X-APISIX-Remote-Addr: 192.168.65.1
APISIX Fault Injection
Send a request to the route with the URL parameter name being a different value:
curl -i "http://127.0.0.1:9080/anything?name=jane"
You should receive an HTTP/1.1 200 OK response from the upstream service without the injections.