Troubleshooting

Narrow an AISIX failure to startup, configuration, caller policy, the provider path, or AISIX Cloud projection before changing resources. Work through these checks in order until the failing layer is clear, then use the matching section to decide the next action.

Fast triage

Start with the runtime path before changing configuration.

Check listener health first:

curl -i "http://127.0.0.1:3000/livez"

When Prometheus metrics are enabled, check the latest observed and applied configuration on the private metrics/status listener:

curl -sS "http://127.0.0.1:9090/status/config"

Verify model discovery with the same caller API key the application uses:

AISIX_API_KEY="YOUR_CALLER_API_KEY"

curl -sS "http://127.0.0.1:3000/v1/models" \
  -H "Authorization: Bearer ${AISIX_API_KEY}"

Then send one real request to the endpoint that failed. For example, check the OpenAI-compatible chat path with the caller API key:

curl -sS "http://127.0.0.1:3000/v1/chat/completions" \
  -H "Authorization: Bearer ${AISIX_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4o-mini",
    "messages": [
      {
        "role": "user",
        "content": "Say hello."
      }
    ]
  }'

Use the response status, error type, and correlation header to choose the next check.

Signal	What to Check Next
Listener health fails.	Process, listener binding, startup config, and TLS.
Configuration status is degraded or out_of_sync.	Rejected resources, the last load failure, and the configuration source.
Configuration status does not reflect an expected direct etcd change.	etcd reachability, the watched prefix, and configuration source state.
Model discovery does not show the expected alias.	Caller API key access, model type, and configuration visibility.
Real proxy request fails before upstream dispatch.	Caller authentication, model access, guardrails, rate limits, or budgets.
Real proxy request reaches the provider and fails.	Provider key, base URL, upstream model ID, quota, outage, or outbound network path.

For request correlation, use x-aisix-request-id, which AISIX adds to proxy responses across endpoint families. Successful Chat Completions responses also include x-aisix-call-id. For exact header scope, see Headers and Error Codes.

Verify Configuration Visibility

Use this step when resources were created or updated recently, or when the proxy behaves as if it has older configuration.

Signal	Check	Action
Process fails during startup.	Dynamic-resource source, network reachability, TLS certificate paths, file permissions, and startup config syntax.	Fix the startup configuration or resource source, then restart the gateway.
Configuration status is never_loaded, degraded, or out_of_sync.	/status/config fields source, applied, rejected, and last_failure.	Restore the source or correct rejected configuration, then confirm that the next load is synced.
A direct etcd change does not appear in proxy traffic.	Applied gateway configuration snapshot, store connectivity, and the watched prefix.	Verify the final proxy path after the snapshot updates; see Configuration Propagation.
Model discovery omits a new alias.	Caller API key allow list, model type, and snapshot freshness.	Correct the model or caller API key, then query model discovery again.
Error mentions a missing provider key or unknown resource.	Provider key, model, and caller API key references.	Create or correct dependent resources in order, then send a real proxy request.

For a store-backed gateway, etcd provides the dynamic-resource source. If etcd becomes unavailable after a snapshot has loaded, the gateway can continue using that snapshot, but it cannot receive new or updated dynamic resources until the configuration connection recovers.

Verify Caller Access and Policy

Use this step when AISIX rejects the request before a provider call, or when model discovery differs between caller API keys.

Signal	Check	Action
Authentication error.	The application sends the plaintext caller API key, not the stored hash or upstream provider key.	Update the application secret or authorization header.
Permission or model-access error.	The requested model alias is allowed by the caller API key.	Add the alias to the caller API key or request an allowed model.
Content-policy error.	Enabled guardrails, the stage where each guardrail runs, and the triggering prompt or response content.	Adjust the prompt, guardrail rules, or fail-open behavior where appropriate.
Rate-limit or budget error.	Retry hint, API-key limit, model limit, shared policy, AISIX Cloud budget state, and replica-local counters.	Wait for the retry window, increase the limit, or adjust the matching policy.

For exact proxy error envelopes, status codes, and retry headers, see Proxy Errors and Retries.

When rate limits differ across gateway instances, identify the counter backend before changing a policy. Memory counters are local to each process. Redis shares counters, but a runtime Redis failure falls back to process-local enforcement. Outage-time counts are not reconciled into Redis after recovery, so restore Redis and let the active windows roll over before assuming the counters are aligned again.

Verify the Provider Path

Use this step after AISIX authenticates the caller, resolves the model alias, and starts dispatching to the configured provider.

Signal	Check	Action
502 or upstream_error.	Provider key secret, base URL, upstream model ID, provider quota, provider outage, and outbound network path.	Fix the provider path, then send the same proxy request again.
503 with provider unavailable.	Provider adapter availability and whether the resolved adapter supports the requested route.	Use a supported provider, adapter, endpoint, or model.
503 with all candidates unavailable.	Multi-target model health, cooldown state, and routing filters.	Restore a healthy target or adjust routing behavior.
Model health is degraded or down.	Recent upstream failure streak, provider outage, quota, outbound network path, and credential validity.	Restore provider reachability or route traffic to a healthy target.

Compare the failing route with the matching provider upstream guide when the problem is provider-specific.

Read a Transport Error

A transport error means the request never completed at the HTTP layer, so there is no upstream status code to interpret. Gateway logs record the cause chain alongside the message, which is what distinguishes the possible faults:

Cause in the log line	Meaning	Where to look
dns error: failed to lookup address information	The provider hostname did not resolve.	DNS configuration, the api_base hostname, and egress DNS policy.
tcp connect error: Connection refused	Nothing accepted the connection on that host and port.	The api_base port, and whether an intermediate proxy is listening.
tcp connect error: Connection timed out	The connection attempt was black-holed.	Firewall, security group, and egress rules on the path.
connection closed before message completed or a reset while sending	A pooled connection was closed by the far end before the request completed.	upstream.pool_idle_timeout_secs; see Tune the Upstream Connection Layer.
A TLS or certificate failure	The handshake with the provider or an intercepting proxy failed.	Trust roots, and whether a TLS-terminating proxy re-signs traffic.

Errors that appear only intermittently against a provider that is otherwise healthy usually point to connection reuse rather than the provider. Lower pool_idle_timeout_secs below the shortest idle timeout on the path and re-check.

The same failure has a mirror image on the inbound side. If a gateway or load balancer in front of AISIX reports intermittent resets or 502s against a healthy AISIX, check whether downstream.idle_timeout_secs is set below that node's own pool idle timeout — the gateway would then be closing connections the node in front still considers usable. Leaving it at 0 (the default) rules this out. See Tune the Downstream Connection Layer.

Attribute an Upstream Error to the Right Hop

When the recorded message is upstream returned HTTP <status>: <body>, the status and body came from whatever answered at api_base — which is the provider only when nothing sits in front of it. Response bodies that mention a proxy's own vocabulary indicate an intermediate hop generated the error rather than the model provider:

• upstream connect error or disconnect/reset before headers with a reset reason, or exceeded request buffer limit while retrying upstream, are emitted by Envoy-based proxies, service meshes, and API gateways.
• A generic HTML error page is emitted by a reverse proxy or load balancer, not by a provider's JSON API.

On modeled routes, the gateway records these bodies for diagnosis but does not return them to the caller. Upstream 5xx responses reach the application as a generic 502 envelope. Provider passthrough instead forwards the upstream provider's native status and body. Use the per-attempt records to identify the failing provider_key_id and target model, then compare them with access logs from the upstream endpoint configured in api_base.

Verify AISIX Cloud Projection

Use this step when control-plane state and live gateway behavior do not match, or when an AISIX gateway cannot receive projected configuration.

Signal	Check	Action
AISIX Cloud heartbeat fails.	Certificate identity, trust roots, runtime state, control-plane URL, and outbound network path.	Restore AISIX Cloud connectivity before investigating resource projection; see Connect an AISIX Gateway.
The control plane shows a resource but live traffic does not use it.	Environment scope and projection status for the gateway handling traffic.	Move the resource to the correct environment or wait for projection to complete.
AISIX Cloud budget checks fail or appear unavailable.	Control-plane connectivity, budget policy target, and budget-check response details.	Restore budget-check connectivity or correct the AISIX Cloud policy.
Playground succeeds but live traffic differs.	Live gateway, environment, model alias, caller key, and provider target.	Send the live request through the intended AISIX gateway and environment.

Use the relevant feature guide or reference page after identifying the failing layer. Re-run the caller-facing request after each correction to verify the complete path rather than only the component that failed.