a7-plugin-fault-injection

Overview

The fault-injection plugin in API7 Enterprise Edition (API7 EE) injects faults — delays and HTTP aborts — into requests for chaos engineering and resiliency testing. It runs in the rewrite phase with priority 11000 (very early), meaning it executes before most other plugins including authentication and rate limiting.

Execution order: delay first → abort second. If abort fires, subsequent plugins do NOT execute.

When to Use

• Chaos engineering: simulate upstream failures and slowdowns.
• Resiliency testing: verify timeout handling and circuit breakers.
• Load testing: add artificial latency to measure degradation.
• Canary fault testing: inject faults for specific users or conditions.

Plugin Configuration Reference

At least one of abort or delay must be specified.

abort Object

Field	Type	Required	Default	Description
http_status	integer	Yes	—	HTTP status code (≥ 200)
body	string	No	—	Response body; supports Nginx variables ($remote_addr)
headers	object	No	—	Response headers; values support Nginx variables
percentage	integer	No	100 (always)	Percentage of requests to abort (0–100)
vars	array	No	—	Conditional rules using expression syntax (max 20 items)

delay Object

Field	Type	Required	Default	Description
duration	number	Yes	—	Delay in seconds (supports decimals: 0.5, 1.5)
percentage	integer	No	100 (always)	Percentage of requests to delay (0–100)
vars	array	No	—	Conditional rules using expression syntax (max 20 items)

Vars Expression Syntax

The vars field uses logical expressions for conditional fault injection.

Structure

[
  [["condition1a"], ["condition1b"]],  // AND group 1
  [["condition2a"]]                     // AND group 2
]
// Groups joined by OR — first matching group triggers the fault

Variable Access

Prefix	Source	Example
arg_*	Query parameters	arg_name → ?name=value
http_*	Request headers	http_apikey → X-Api-Key header
(none)	Nginx built-ins	remote_addr, uri, request_method

Operators

Operator	Example
==	["arg_name", "==", "jack"]
~=	["arg_env", "~=", "prod"]
>, >=, <, <=	["arg_age", ">", 18]
~~	["arg_env", "~~", "[Dd]ev"] (regex)
~*	["arg_env", "~*", "dev"] (case-insensitive regex)
in	["arg_ver", "in", ["v1","v2"]]
!	["arg_age", "!", "<", 18] (negation → >=)
ipmatch	["remote_addr", "ipmatch", ["10.0.0.0/8"]]

Step-by-Step Examples

1. Fixed Delay (3 Seconds)

a7 route create --gateway-group default -f - <<'EOF'
{
  "id": "delay-test",
  "uri": "/api/*",
  "plugins": {
    "fault-injection": {
      "delay": {
        "duration": 3
      }
    }
  },
  "upstream": {
    "type": "roundrobin",
    "nodes": [{"host": "backend", "port": 8080, "weight": 1}]
  }
}
EOF

2. Percentage-Based Abort (50% Return 503)

a7 route create --gateway-group default -f - <<'EOF'
{
  "id": "abort-test",
  "uri": "/api/*",
  "plugins": {
    "fault-injection": {
      "abort": {
        "http_status": 503,
        "body": "Service temporarily unavailable",
        "percentage": 50
      }
    }
  },
  "upstream": {
    "type": "roundrobin",
    "nodes": [{"host": "backend", "port": 8080, "weight": 1}]
  }
}
EOF

3. Conditional Abort Based on Query Parameter

Only abort when ?name=jack:

a7 route create --gateway-group default -f - <<'EOF'
{
  "id": "conditional-abort",
  "uri": "/api/*",
  "plugins": {
    "fault-injection": {
      "abort": {
        "http_status": 403,
        "body": "Fault Injection!\n",
        "vars": [
          [["arg_name", "==", "jack"]]
        ]
      }
    }
  },
  "upstream": {
    "type": "roundrobin",
    "nodes": [{"host": "backend", "port": 8080, "weight": 1}]
  }
}
EOF

4. Complex Conditional Logic (AND/OR)

Abort when (name=jack AND age≥18) OR (has api-key header):

a7 route create --gateway-group default -f - <<'EOF'
{
  "id": "complex-fault",
  "uri": "/api/*",
  "plugins": {
    "fault-injection": {
      "abort": {
        "http_status": 403,
        "body": "Fault Injection!\n",
        "vars": [
          [
            ["arg_name", "==", "jack"],
            ["arg_age", "!", "<", 18]
          ],
          [
            ["http_apikey", "==", "api-key"]
          ]
        ]
      }
    }
  },
  "upstream": {
    "type": "roundrobin",
    "nodes": [{"host": "backend", "port": 8080, "weight": 1}]
  }
}
EOF

5. Custom Headers with Nginx Variables

a7 route create --gateway-group default -f - <<'EOF'
{
  "id": "headers-fault",
  "uri": "/api/*",
  "plugins": {
    "fault-injection": {
      "abort": {
        "http_status": 200,
        "body": "{\"uri\": \"$uri\"}",
        "headers": {
          "X-Fault-Injected": "true",
          "X-Request-URI": "$uri"
        }
      }
    }
  },
  "upstream": {
    "type": "roundrobin",
    "nodes": [{"host": "backend", "port": 8080, "weight": 1}]
  }
}
EOF

6. Canary Fault Testing

Only users with X-Canary: true header experience 10% fault rate:

a7 route create --gateway-group default -f - <<'EOF'
{
  "id": "canary-fault",
  "uri": "/api/*",
  "plugins": {
    "fault-injection": {
      "abort": {
        "http_status": 500,
        "percentage": 10,
        "vars": [
          [["http_x_canary", "==", "true"]]
        ]
      }
    }
  },
  "upstream": {
    "type": "roundrobin",
    "nodes": [{"host": "backend", "port": 8080, "weight": 1}]
  }
}
EOF

7. Combined Delay + Abort with Different Conditions

a7 route create --gateway-group default -f - <<'EOF'
{
  "id": "combined-fault",
  "uri": "/api/*",
  "plugins": {
    "fault-injection": {
      "delay": {
        "duration": 2,
        "vars": [
          [["http_x_slow", "==", "true"]]
        ]
      },
      "abort": {
        "http_status": 503,
        "vars": [
          [["http_x_fail", "==", "true"]]
        ]
      }
    }
  },
  "upstream": {
    "type": "roundrobin",
    "nodes": [{"host": "backend", "port": 8080, "weight": 1}]
  }
}
EOF

Config Sync Example

version: "1"
gateway_group: default
routes:
  - id: fault-injection-demo
    uri: /api/*
    plugins:
      fault-injection:
        delay:
          duration: 1
          percentage: 25
        abort:
          http_status: 503
          body: "Service unavailable"
          percentage: 5
    upstream:
      type: roundrobin
      nodes:
        - host: backend
          port: 8080
          weight: 1

Execution Behavior

1. Delay evaluated first: if vars match and percentage sampled → sleep(duration).
2. Abort evaluated second: if vars match and percentage sampled → return immediately.
3. Percentage sampling: math.random(1, 100) <= percentage.
4. When abort fires: subsequent plugins (auth, rate limiting) are skipped.

Plugin Priority Context

Priority 11000 means fault-injection runs very early:

• ✅ Tracing plugins (zipkin, skywalking) capture faults.
• ❌ Rate limiting won't prevent faults.
• ❌ Authentication won't block faults.

To apply faults only to authenticated users, use vars to check auth-related variables or headers.

Troubleshooting

Symptom	Cause	Fix
Fault never triggers	percentage: 0 or vars never match	Check vars expressions; set percentage > 0
Fault always triggers	No percentage set (defaults to 100%)	Set percentage to desired value
Auth bypass via fault	Plugin runs before auth (priority 11000)	Use vars to restrict fault scope
Body not interpolated	Missing $ prefix on variable	Use $uri not uri in body/headers
Abort + delay both fire	Delay runs first, then abort	This is expected behavior; delay always executes before abort check

---

This page is generated from a7-plugin-fault-injection/SKILL.md in the api7/a7 repository. Browse all skills on the AI Agent Skills page.