Configuration Status
The AISIX gateway reports whether its configuration took effect — and if not, why — through status endpoints and Prometheus metrics. These surfaces answer the operator question "is the gateway serving the configuration I intended?" They cover every resource source: a resources.yaml file, an etcd configuration store you manage, or the AISIX Cloud control plane.
These endpoints are served on the dedicated metrics listener, observability.metrics.prometheus.addr (default 0.0.0.0:9090), alongside GET /metrics. Like the metrics endpoint, they are unauthenticated by design; keep the listener private to your monitoring network.
| Endpoint | Purpose |
|---|---|
GET /status/config | Full configuration load state: derived state, source and applied hashes, resource counts, reload results, rejected entries, unknown-kind classifications, and partially compatible resources. |
GET /status/ready | Readiness gate: 503 until the first valid configuration is applied, 200 ok afterward. |
GET /status/models | Per-model runtime health: one row per configured model with its rotation status. |
GET /status/config
Returns a JSON document describing the last observed and last applied configuration:
curl -sS "http://127.0.0.1:9090/status/config"
{
"state": "synced",
"source": {
"type": "file",
"source_hash": "1dc0ee8d06edcde3ecbf23672858622a83f266910846f514ccb909cf41046653",
"observed_at": "YYYY-MM-DDTHH:MM:SSZ"
},
"applied": {
"config_hash": "1dc0ee8d06edcde3ecbf23672858622a83f266910846f514ccb909cf41046653",
"apply_seq": 1,
"applied_at": "YYYY-MM-DDTHH:MM:SSZ",
"resource_counts": {
"api_keys": 1,
"models": 1,
"provider_keys": 1
}
},
"last_reload": {
"successful": true,
"at": "YYYY-MM-DDTHH:MM:SSZ"
},
"last_failure": null,
"rejected": [],
"unknown_kinds": [],
"partially_compatible": []
}
Configuration States
state is derived by the gateway from the last observed and applied snapshots:
| State | Meaning |
|---|---|
synced | A non-empty configuration is serving, and there are no active entries in rejected. Entries classified in unknown_kinds leave this state unchanged, even when last_error identifies a placement problem. An unknown-kind key can also make source_hash and config_hash differ. Check unknown_kinds and partially_compatible for compatibility details. |
degraded | The gateway is serving, but the latest snapshot carried entries it rejected. Accepted resources and any last known good values remain in service; the rejected array explains each rejected entry. |
out_of_sync | The latest observed snapshot was rejected as a whole. The gateway keeps serving the last valid configuration. |
empty | A valid configuration was applied, but it holds zero resources. |
never_loaded | No valid configuration has been applied since the process started. |
A gateway configured from a resources file applies the file all or nothing, so a failed file reload reports out_of_sync rather than degraded. degraded occurs when the source delivers resources individually and only some of them are invalid.
Etcd-backed configuration reads are forward compatible. If a document contains an unrecognized field, the gateway serves the recognized fields and reports the ignored field in partially_compatible. This alone does not change state from synced. Resources-file validation remains strict, so an unknown field in a resources file rejects the whole reload.
Forward compatibility also covers the resource kind itself. When an etcd key names a resource kind that this gateway version does not know, the key is reported in unknown_kinds rather than rejected. It leaves state and last_reload.successful alone, and aisix_config_unknown_kind_resources counts it instead of aisix_config_rejected_resources. This usually means the control plane is projecting a resource kind introduced after the gateway was released; upgrading the gateway to a version that supports the kind clears the report.
The etcd loader uses the same classification when a known resource kind is written under the shared pricing-catalog prefix, which accepts only pricing. In that case, last_error says that the kind is not accepted under the global prefix. Fix the key at its source; upgrading the gateway does not correct it. A gateway running 1.3.0 or earlier does not separate either case from other rejections: it reports the key in rejected with last_error_kind unknown_kind, and holds aisix_config_last_reload_successful at 0 while the key is present.
never_loaded is the state an etcd-backed or AISIX Cloud gateway holds while it is still reaching its configuration source for the first time. It does not bind the proxy listener in that state, so this endpoint and GET /status/ready are where it is observed; see Startup and the First Configuration. A source that is reachable but holds no resources applies an empty configuration and reports empty, not never_loaded.
Response Fields
Top-level fields:
| Field | Type | Description |
|---|---|---|
state | string | Derived configuration state. One of synced, degraded, out_of_sync, empty, never_loaded. |
source | object | The latest snapshot observed from the configuration source. |
applied | object | The last configuration actually applied and served. Omitted while state is never_loaded. |
last_reload | object | Outcome of the most recent load. Omitted before the first load completes. |
last_failure | object or null | The most recent load failure since the process started. Sticky: it remains populated after a later successful reload, until restart. |
rejected | array | Active entries the gateway rejected. Empty when there are no active rejections; inspect unknown_kinds separately. |
unknown_kinds | array | Etcd keys classified as unknown_kind. This normally identifies a resource kind introduced after the gateway version, but can also identify a known kind written under the shared pricing-catalog prefix. Reported apart from rejected and empty when no key has this classification. |
partially_compatible | array | Served etcd resources that contain fields this gateway version does not recognize, aggregated by resource kind and field. Empty when every served document matches the gateway's schema. |
source fields:
| Field | Type | Description |
|---|---|---|
type | string | Where configuration is read from: file or etcd. |
connected | boolean | Whether the configuration store is reachable. Present only when type is etcd. |
observed_revision | number | Store revision of the latest observed snapshot. Present only when type is etcd. |
source_hash | string | SHA-256 hash of the latest observed snapshot. For a file source, this is the hash of the raw file bytes. |
observed_at | string | RFC 3339 UTC timestamp of the latest observation. |
applied fields:
| Field | Type | Description |
|---|---|---|
applied_revision | number | Store revision the applied configuration reflects. Present only when source.type is etcd. |
config_hash | string | SHA-256 hash of the accepted, served configuration. Equal to source_hash when the gateway served every observed entry. A rejected entry makes the two differ, and so does an entry in unknown_kinds, which is observed but never served. |
apply_seq | number | Counter that increments each time the applied configuration changes. Unchanged content does not advance it. |
applied_at | string | RFC 3339 UTC timestamp of the last applied change. |
resource_counts | object | Served resource count per kind, for example {"models": 2}. |
last_reload and last_failure fields:
| Field | Type | Description |
|---|---|---|
last_reload.successful | boolean | Whether the most recent load completed without rejections. Entries in unknown_kinds are not rejections and leave it true. |
last_reload.at | string | RFC 3339 UTC timestamp of the most recent load. |
last_failure.at | string | When the most recent failure occurred. |
last_failure.last_error_kind | string | Failure kind of the most recent failure. |
last_failure.last_error | string | Human-readable message of the most recent failure. |
Each entry in rejected:
| Field | Type | Description |
|---|---|---|
resource_kind | string | Plural resource kind, such as models or provider_keys. Empty when the source entry could not be attributed to a kind. |
resource_id | string | Resource ID. Empty when the source entry could not be parsed far enough to identify it. |
last_error_kind | string | Failure kind: bad_key, non_json, schema_failed, or parse_failed. |
last_error | string | Human-readable error. Schema messages mask credential values. |
first_seen_at | string | When this rejection was first observed since the process started. Stable across repeated reloads of the same bad entry. |
last_seen_at | string | When this rejection was most recently observed. |
serving_stale_since | string | RFC 3339 timestamp from which the gateway has kept serving the resource's last known good value. Omitted when no previous value remains in service. |
serving_stale_age_seconds | number | Seconds elapsed since serving_stale_since, recalculated when the status is read. Omitted together with serving_stale_since. |
Each entry in unknown_kinds:
| Field | Type | Description |
|---|---|---|
resource_kind | string | Plural resource kind as written in the key. Empty when the key could not be attributed to a kind. |
resource_id | string | Resource ID. Empty when the key could not be parsed far enough to identify it. |
last_error | string | Human-readable explanation from the load path. |
first_seen_at | string | RFC 3339 UTC timestamp when the key was first observed since the process started. Stable across repeated reloads of the same key. |
last_seen_at | string | RFC 3339 UTC timestamp when the key was most recently observed. |
A gateway whose latest snapshot carries a resource kind it does not know, with no entries in rejected, still reports synced and a successful reload:
{
"state": "synced",
"last_reload": {
"successful": true,
"at": "YYYY-MM-DDTHH:MM:SSZ"
},
"rejected": [],
"unknown_kinds": [
{
"resource_kind": "NEW_KIND",
"resource_id": "9a3f2c67-52b8-4b1e-9f4e-1f2f3a4b5c6d",
"last_error": "unknown kind \"NEW_KIND\"",
"first_seen_at": "YYYY-MM-DDTHH:MM:SSZ",
"last_seen_at": "YYYY-MM-DDTHH:MM:SSZ"
}
],
"partially_compatible": []
}
The gateway retains details for at most 256 unknown-kind keys across all resource kinds, using a separate budget from real rejections. If more than 256 affected keys remain, unknown_kinds and aisix_config_unknown_kind_resources are lower bounds even after a full configuration resynchronization. After an overflow, they can also remain incomplete when the total falls below 256, until a full resynchronization rebuilds the retained set. Unknown-kind volume cannot displace entries from rejected.
Each entry in partially_compatible:
| Field | Type | Description |
|---|---|---|
resource_kind | string | Plural resource kind, such as api_keys or models. |
field | string | Ignored field path. Array indexes are normalized to [], for example routing.targets[].priority. |
count | number | Served resources of this kind that contain the ignored field. |
Use config_hash to confirm a specific change landed: the hash is deterministic, so a deployment pipeline that knows what it shipped can compare hashes instead of diffing resources. For a file source, source_hash is the SHA-256 of the file bytes (sha256sum resources.yaml). For an etcd source, a matching hash does not mean every field is enforced; also require partially_compatible to be empty when exact schema compatibility matters.
GET /status/ready
A readiness gate for the configuration source only:
curl -sSi "http://127.0.0.1:9090/status/ready"
| Condition | Status | Body |
|---|---|---|
| No valid configuration applied yet | 503 Service Unavailable | no configuration available |
| A valid configuration has been applied | 200 OK | ok |
Use it as a startup or readiness probe so a gateway does not receive traffic before it can serve configured routes. It stays 200 once the first configuration is applied, including while a later reload fails and the gateway serves the last valid configuration.
The proxy listener's /livez and /readyz answer process-level questions — liveness and drain state — rather than repeating this one. On an etcd-backed or AISIX Cloud gateway that listener is not bound until the first configuration is applied, so while this endpoint reports 503 neither of them answers at all. See Health Checks.
GET /status/models
The per-model runtime health view — one row per configured model:
curl -sS "http://127.0.0.1:9090/status/models"
[
{
"id": "9a3f2c67-52b8-4b1e-9f4e-1f2f3a4b5c6d",
"display_name": "gpt-4o-prod",
"kind": "direct",
"status": "healthy"
},
{
"id": "5b17e9d2-8a44-4c05-b7a1-0c9d8e7f6a5b",
"display_name": "claude-prod",
"kind": "direct",
"status": "cooldown",
"status_reason": "upstream_auth_failure",
"cooldown_until": { "secs_since_epoch": 1784708130, "nanos_since_epoch": 0 }
}
]
| Status | Meaning |
|---|---|
healthy | The model is in rotation. |
cooldown | Recent upstream failures took the model out of rotation until cooldown_until; status_reason names the failure class, for example upstream_auth_failure or upstream_rate_limited. Only a model whose cooldown block sets enabled: true enters this state. |
unhealthy | Recent background model checks failed; routing avoids the model. last_check_status carries the HTTP status of the most recent check. |
not_applicable | The row is a multi-target or other virtual model; its availability derives from its target models' rows. |
Timestamps (cooldown_until, and last_checked_at on background-checked models) are seconds/nanoseconds epoch objects as shown above, not RFC 3339 strings. Like the other status endpoints, /status/models is unauthenticated and reflects the applied configuration, so it works in every gateway deployment.
Configuration Load Metrics
The GET /metrics endpoint on the same listener exposes the configuration load state as Prometheus series. Values refresh at scrape time from the same state that backs GET /status/config.
| Metric | Type | Labels | Description |
|---|---|---|---|
aisix_config_last_reload_successful | gauge | None | 1 when the most recent load completed without rejections, otherwise 0. Entries classified in unknown_kinds do not flip it to 0. |
aisix_config_last_reload_success_timestamp_seconds | gauge | None | Unix timestamp of the most recent successful load. |
aisix_config_reloads_total | counter | None | Configuration loads since the process started, counting boot loads, file reloads, and full synchronizations from the configuration store. |
aisix_config_reload_failures_total | counter | reason | Loads that did not fully succeed, bucketed by reason: fetch (source unreachable or unreadable), parse (source content could not be parsed), or validate (a resource failed schema, shape, or reference validation). |
aisix_config_rejected_resources | gauge | kind | Currently rejected entries per resource kind. 0 after the offending entries are fixed. |
aisix_config_partially_compatible_resources | gauge | kind | Served resources carrying at least one ignored field, grouped by resource kind. A resource with several ignored fields counts once. |
aisix_config_stale_served_resources | gauge | kind | Resources whose latest source value was rejected while the last known good value remains in service, grouped by resource kind. |
aisix_config_unknown_kind_resources | gauge | kind | Retained etcd keys classified as unknown_kind, grouped by resource kind. A series returns to 0 when its retained keys clear. aisix_config_rejected_resources does not count these retained keys. Values are lower bounds while more than 256 keys are affected; after an overflow, they can remain incomplete until a full configuration resynchronization. |
aisix_config_hash_info | gauge | hash | Info-style series: exactly one live sample with value 1, whose hash label is the applied config_hash. |
aisix_config_observed_revision | gauge | None | Store revision of the latest observed snapshot. Emitted only for an etcd source. |
aisix_config_applied_revision | gauge | None | Store revision of the applied configuration. Emitted only for an etcd source. |
aisix_config_source_connected | gauge | None | 1 when the configuration store is reachable. Emitted only for an etcd source. |
For the full metric catalog, see Metrics Reference.
Example Alerts
Alert when the gateway rejects any configured resource — the gateway keeps serving, but something an operator wrote is not in effect:
- alert: AisixConfigRejectedResources
expr: sum by (instance) (aisix_config_rejected_resources) > 0
for: 5m
labels:
severity: warning
annotations:
summary: "AISIX gateway is rejecting configured resources"
description: "Check GET /status/config on {{ $labels.instance }}: the rejected array names each entry and its error."
This alert excludes keys classified as unknown kinds, which aisix_config_unknown_kind_resources counts instead. On a gateway running 1.3.0 or earlier the separation does not exist. A resource kind introduced after that gateway version is counted as a rejection, so the alert fires and aisix_config_last_reload_successful stays at 0 until the gateway is upgraded. The 1.3.0 upgrade notes in the release notes describe the pricing-catalog case where this happens.
Alert when configuration reloads keep failing — the gateway is running on the last valid configuration and new changes are not taking effect:
- alert: AisixConfigReloadFailing
expr: aisix_config_last_reload_successful == 0
for: 10m
labels:
severity: warning
annotations:
summary: "AISIX gateway configuration reloads are failing"
description: "The last configuration load on {{ $labels.instance }} did not fully succeed. Check last_failure and rejected in GET /status/config."