a6-recipe-circuit-breaker

Overview

A circuit breaker prevents cascading failures by detecting unhealthy upstream services and temporarily stopping requests to them. When the upstream returns too many errors, the circuit "opens" and APISIX returns errors immediately without forwarding requests. After a cooldown period, it "half-opens" to test if the upstream has recovered.

APISIX implements this via the api-breaker plugin, which tracks response status codes and manages circuit state automatically.

When to Use

• Protect your API from cascading failures when an upstream goes down
• Automatically stop sending traffic to failing backends
• Allow failing services time to recover before retrying
• Return fast error responses instead of waiting for timeouts

Circuit Breaker States

          ┌─────────┐
          │ CLOSED   │ ← Normal operation: requests flow through
          │(healthy) │
          └────┬─────┘
               │ Error count exceeds threshold
               ▼
          ┌─────────┐
          │  OPEN    │ ← Breaker tripped: returns 502 immediately
          │(tripped) │
          └────┬─────┘
               │ After cooldown period
               ▼
         ┌──────────┐
         │HALF-OPEN │ ← Test: allows one request through
         │ (testing) │
         └─────┬────┘
               │
       ┌───────┴───────┐
       │               │
   Success          Failure
       │               │
       ▼               ▼
   CLOSED           OPEN (longer cooldown)

Plugin Configuration Reference

Field	Type	Required	Default	Description
break_response_code	integer	Yes	—	HTTP status code returned when circuit is open (e.g., 502, 503).
break_response_body	string	No	—	Response body returned when circuit is open.
break_response_headers	array[object]	No	—	Response headers when circuit is open. Format: [{"key": "name", "value": "val"}].
unhealthy.http_statuses	array[integer]	No	[500]	HTTP status codes from upstream that count as unhealthy.
unhealthy.failures	integer	No	3	Number of consecutive unhealthy responses before opening the circuit.
healthy.http_statuses	array[integer]	No	[200]	HTTP status codes from upstream that count as healthy (for recovery).
healthy.successes	integer	No	3	Number of consecutive healthy responses to close the circuit.
max_breaker_sec	integer	No	300	Maximum circuit-open duration in seconds. Cooldown doubles each time but caps here.

Breaker Timing

When the circuit opens:

1. First open: 2 seconds cooldown
2. If it opens again: 4 seconds (doubles)
3. Next: 8 seconds, 16 seconds, ...
4. Caps at max_breaker_sec (default 300s = 5 minutes)

During cooldown, all requests get the break_response_code immediately.

Step-by-Step: Enable Circuit Breaker

1. Basic circuit breaker

a6 route create -f - <<'EOF'
{
  "id": "protected-api",
  "uri": "/api/*",
  "plugins": {
    "api-breaker": {
      "break_response_code": 502,
      "unhealthy": {
        "http_statuses": [500, 502, 503],
        "failures": 3
      },
      "healthy": {
        "http_statuses": [200],
        "successes": 3
      },
      "max_breaker_sec": 300
    }
  },
  "upstream": {
    "type": "roundrobin",
    "nodes": {
      "backend:8080": 1
    }
  }
}
EOF

After 3 consecutive 500/502/503 responses, the circuit opens and returns 502 immediately. After cooldown, it tests with one request. If 3 consecutive 200s come back, the circuit closes and normal operation resumes.

2. Circuit breaker with custom error body

a6 route create -f - <<'EOF'
{
  "id": "api-with-error-body",
  "uri": "/api/*",
  "plugins": {
    "api-breaker": {
      "break_response_code": 503,
      "break_response_body": "{\"error\": \"service temporarily unavailable\", \"retry_after\": 30}",
      "break_response_headers": [
        {"key": "Content-Type", "value": "application/json"},
        {"key": "Retry-After", "value": "30"}
      ],
      "unhealthy": {
        "http_statuses": [500, 502, 503, 504],
        "failures": 5
      },
      "healthy": {
        "http_statuses": [200, 201, 204],
        "successes": 2
      },
      "max_breaker_sec": 60
    }
  },
  "upstream": {
    "type": "roundrobin",
    "nodes": {
      "backend:8080": 1
    }
  }
}
EOF

3. Sensitive circuit breaker (trips on first error)

{
  "plugins": {
    "api-breaker": {
      "break_response_code": 503,
      "unhealthy": {
        "http_statuses": [500, 502, 503],
        "failures": 1
      },
      "healthy": {
        "http_statuses": [200],
        "successes": 1
      },
      "max_breaker_sec": 30
    }
  }
}

Trips on the very first 5xx error. Recovers after one successful response.

Combining with Health Checks

For production, combine the circuit breaker with upstream health checks. The circuit breaker handles per-route protection while health checks manage per-node health at the upstream level.

# Create upstream with health checks
a6 upstream create -f - <<'EOF'
{
  "id": "monitored-backend",
  "type": "roundrobin",
  "nodes": {
    "backend-1:8080": 1,
    "backend-2:8080": 1
  },
  "checks": {
    "active": {
      "type": "http",
      "http_path": "/health",
      "healthy": {
        "interval": 5,
        "successes": 2
      },
      "unhealthy": {
        "interval": 3,
        "http_failures": 3
      }
    }
  }
}
EOF

# Create route with circuit breaker
a6 route create -f - <<'EOF'
{
  "id": "api",
  "uri": "/api/*",
  "plugins": {
    "api-breaker": {
      "break_response_code": 503,
      "unhealthy": {
        "http_statuses": [500, 502, 503],
        "failures": 3
      },
      "healthy": {
        "http_statuses": [200],
        "successes": 3
      }
    }
  },
  "upstream_id": "monitored-backend"
}
EOF

Config Sync Example

version: "1"
routes:
  - id: protected-api
    uri: /api/*
    plugins:
      api-breaker:
        break_response_code: 503
        break_response_body: '{"error": "service unavailable"}'
        break_response_headers:
          - key: Content-Type
            value: application/json
          - key: Retry-After
            value: "30"
        unhealthy:
          http_statuses: [500, 502, 503]
          failures: 3
        healthy:
          http_statuses: [200]
          successes: 3
        max_breaker_sec: 300
    upstream_id: backend
upstreams:
  - id: backend
    type: roundrobin
    nodes:
      "backend:8080": 1

Troubleshooting

Symptom	Cause	Fix
Circuit never opens	unhealthy.http_statuses doesn't include the error code	Add the actual error codes your upstream returns
Circuit stays open too long	max_breaker_sec too high	Lower max_breaker_sec for faster recovery
Circuit flaps open/closed	Threshold too low with intermittent errors	Increase unhealthy.failures threshold
502 from APISIX (not circuit breaker)	Upstream truly unreachable (connection refused)	Connection errors also count toward unhealthy threshold
Recovery too slow	healthy.successes too high	Lower healthy.successes for faster recovery

---

This page is generated from a6-recipe-circuit-breaker/SKILL.md in the api7/a6 repository. Browse all skills on the AI Agent Skills page.