Version: Dev

Metrics Reference

AISIX exposes operational metrics in Prometheus text format. A Prometheus server or another compatible collector can scrape these metrics for dashboards, alerts, and PromQL queries.

Prometheus metrics are enabled by default on a dedicated listener. The default startup configuration is:

config.yaml
observability:
  metrics:
    prometheus:
      enabled: true
      addr: 0.0.0.0:9090
      path: /metrics

Prometheus scrapes GET /metrics from this listener.

The metrics endpoint is unauthenticated by design. Keep its listener private to your monitoring network.

AISIX publishes configuration status whenever the endpoint is scraped. Other metric families are registered when AISIX first records the corresponding activity, so traffic metrics might not appear immediately after startup. Send a request through the proxy, then scrape again for the corresponding series to appear.

Metrics Catalog

Search metric names, descriptions, labels, and values, or filter the catalog by family and type. Expand an entry to review its details.

Metrics: 62
Families: 10

Query behavior

Metric Types

counter

A cumulative value that increases until the process restarts, such as a request or token total. Use rate() to calculate how quickly it changes.

gauge

A current value that can increase or decrease, such as active requests or remaining quota.

histogram

Observations counted in configurable buckets. The _bucket, _sum, and _count series can be aggregated before calculating a percentile.

summary

Observations with quantiles calculated by each gateway instance. Summaries also expose _sum and _count, but their quantiles cannot be aggregated across instances.

Search metrics

Showing 62 of 62 entries

Request Metrics

Track request outcomes and the work currently in progress at the proxy.

Metric entries: 8

Detailed Request Labels

Every counter in this family samples once per client request, and its status is the status the caller received. A request whose first target failed and whose fallback then succeeded is a single status="200" sample here; the failure it recovered from is not represented in this family at all. Count upstream attempts with the deployment metrics and read individual attempts in the usage log.

For the three detailed request counters, stream records whether the client requested streaming. is_fallback records whether a fallback target served the request and does not appear on latency metrics.

provider_key_name and user_name are human-readable companions to their IDs. Each name has a one-to-one relationship with its ID, so it does not add another series dimension. user_name is unknown until the control plane supplies it.

A failed request carries the same provider, upstream_model, and provider-key labels as a successful one, so a failure rate per provider or per provider key is computable from this family alone. The upstream labels report the last target the request selected: under retry or failover that is the attempt whose error the caller received. They report unknown only when the request never selected a target — an unknown model, an input guardrail block, a budget refusal, or a body refused before dispatch.

inbound_protocol is a bounded protocol family derived from the normalized endpoint. The Anthropic-protocol endpoints report anthropic; /mcp, /a2a, /v1/realtime, and /passthrough_route report mcp, a2a, realtime, and passthrough; and the remaining gateway endpoints report openai. The in-flight gauge uses the same values.

upstream_protocol is the other half of that pair: the protocol AISIX spoke to the upstream that served the request. It is resolved from the selected target's Provider Key by the same rule dispatch uses, so it reports the wire shape the request was actually converted to — openai, anthropic, bedrock, vertex, or azure-openai — and unknown when the request selected no upstream at all. Group by both labels to separate native traffic from cross-protocol conversion; see Separate Cross-Protocol Conversion Traffic.

Do not substitute provider for upstream_protocol. provider is an open vendor string, and the same vendor can be reached through different adapters, so a PromQL mapping from vendor names to protocols has to be maintained by hand and silently misreports every custom or OpenAI-compatible vendor.

endpoint is always a normalized route template, never a raw request path. Routes with a path parameter collapse to one series — /v1/batches/:id, /v1/videos/:id, and /mcp/{server} — while paths under /passthrough/ use /passthrough_route and an unrecognized path reports other.

aisix_requests_total . Description: Proxy request outcomes in the compatibility series with the broadest endpoint coverage. . Type: counter . Label count: 4 labels

Labels

provider, model, status, outcome

Values for `outcome`

success, client_error, rate_limited, upstream_error

success covers HTTP 200–399, client_error covers HTTP 400–499 except 429, rate_limited is HTTP 429, and all other statuses map to upstream_error.

Behavior

A2A agent calls use provider="a2a" and model="a2a".

PromQL example

sum(rate(aisix_requests_total[5m])) by (outcome)

aisix_llm_requests_total . Description: Model-inference request outcomes, including successful and failed requests. . Type: counter . Label count: 16 labels

Labels

endpoint, inbound_protocol, upstream_protocol, provider, model, upstream_model, provider_key_id, provider_key_name, api_key_id, team_id, user_id, user_name, stream, is_fallback, status, outcome

Values for `inbound_protocol`

openai, anthropic, realtime

Values for `upstream_protocol`

openai, anthropic, bedrock, vertex, azure-openai, unknown

The protocol the gateway spoke to the upstream, which is independent of inbound_protocol. unknown means the request selected no upstream — a rejection before dispatch, or a route that calls no model.

Values for `stream`

false, true

Values for `is_fallback`

false, true

Values for `outcome`

success, client_error, rate_limited, upstream_error

success covers HTTP 200–399, client_error covers HTTP 400–499 except 429, rate_limited is HTTP 429, and all other statuses map to upstream_error.

Behavior

Covers the endpoints that call a model: chat-completions, completions, messages, count-tokens, responses, embeddings, rerank, audio, image generation and editing, videos, and realtime sessions. Requests that reach no model are counted in aisix_proxy_requests_total only — MCP tool calls, A2A agent calls, the provider passthrough, and the file, batch, and fine-tuning management routes.

A request refused before dispatch, such as an oversized body, is counted against the endpoint it targeted, so a success rate over an endpoint includes those failures in its denominator.

aisix_proxy_requests_total . Description: Detailed request outcomes for all proxied traffic, model-inference and otherwise. . Type: counter . Label count: 16 labels

Labels

Values for `inbound_protocol`

openai, anthropic, mcp, a2a, realtime, passthrough

Values for `upstream_protocol`

openai, anthropic, bedrock, vertex, azure-openai, unknown

Values for `stream`

false, true

Values for `is_fallback`

false, true

Values for `outcome`

success, client_error, rate_limited, upstream_error

success covers HTTP 200–399, client_error covers HTTP 400–499 except 429, rate_limited is HTTP 429, and all other statuses map to upstream_error.

Behavior

One sample per client request, carrying the status the caller received. Retries and failovers within a request do not add samples, so this counter answers "how many requests did callers make, and how did they end" — not "how many calls did the gateway place upstream".

aisix_proxy_failed_requests_total . Description: The subset of proxy requests whose outcome is not success. . Type: counter . Label count: 16 labels

Labels

Values for `inbound_protocol`

openai, anthropic, mcp, a2a, realtime, passthrough

Values for `upstream_protocol`

openai, anthropic, bedrock, vertex, azure-openai, unknown

Values for `stream`

false, true

Values for `is_fallback`

false, true

Values for `outcome`

client_error, rate_limited, upstream_error

client_error covers HTTP 400–499 except 429, rate_limited is HTTP 429, and all other statuses map to upstream_error.

aisix_proxy_in_flight_requests . Description: Requests currently being handled by the proxy, grouped by normalized endpoint and inbound protocol. . Type: gauge . Label count: 2 labels

Labels

endpoint, inbound_protocol

Values for `inbound_protocol`

openai, anthropic, mcp, a2a, realtime, passthrough

Behavior

MCP requests use inbound_protocol="mcp", with endpoint="/mcp" for the aggregated gateway and endpoint="/mcp/{server}" for the per-server endpoint. A2A calls use endpoint="/a2a" and inbound_protocol="a2a".

Requests normalized to endpoint="/passthrough_route" use inbound_protocol="passthrough".

PromQL example

sum(aisix_proxy_in_flight_requests) by (endpoint, inbound_protocol)

aisix_proxy_client_cancelled_requests_total . Description: Requests whose caller disconnected before the gateway sent response headers. . Type: counter . Label count: 4 labels

Labels

endpoint, model, provider_key_id, provider_key_name

Behavior

These requests never reach a normal outcome, so they do not appear in the other request counters. The gateway records them here and in the access log with status 499.

A rising rate usually means callers give up while waiting for the first token. Break the series down by model and compare it against aisix_llm_time_to_first_token_seconds for the same model.

model is the model the caller addressed, and the provider-key labels identify the target the request was waiting on. A caller that disconnects before the gateway resolves them — during the request body upload, for example — reports unknown for all three.

The caller identity, status, and outcome labels of the other request counters are deliberately absent. A cancelled request has no status and often no team or user, so those dimensions would be unknown on every sample.

A caller that disconnects after response headers were sent is not counted here. That request already has a normal outcome and a usage event.

PromQL example

sum(rate(aisix_proxy_client_cancelled_requests_total[5m])) by (endpoint, model)

aisix_proxy_request_body_limit_rejections_total . Description: Requests refused for exceeding proxy.request_body_limit_bytes, grouped by how the gateway finished reading the refused body. . Type: counter . Label count: 3 labels

Labels

endpoint, inbound_protocol, outcome

Values for `inbound_protocol`

openai, anthropic, mcp, a2a, realtime, passthrough

Values for `outcome`

completed, cap_reached, timeout, client_read_error

completed means the caller sent the whole body it declared, so it could read the 413. cap_reached and timeout mean the gateway stopped absorbing the body first, and client_read_error means the caller went away mid-body — in all three the caller usually sees a closed connection instead of the response.

Behavior

The gateway reads and discards the body of a refused request so the caller can receive the 413 on the same connection. That read is bounded, and outcome reports how it ended.

A rising share of any outcome other than completed means callers are seeing closed connections instead of 413 responses. The matching aisix::body_limit log entry carries the declared size, the configured limit, and the bytes read for the same request_id.

Only requests that declare a Content-Length over the limit are counted here. A chunked body over the limit is rejected while it is being read, has no comparable outcome, and appears in aisix_requests_total with status 413.

PromQL example

sum(rate(aisix_proxy_request_body_limit_rejections_total[5m])) by (endpoint, inbound_protocol, outcome)

aisix_auth_decisions_total . Description: Caller authentication decisions across API key, JWT, and missing-credential paths. . Type: counter . Label count: 3 labels

Labels

method, result, reason

Values for `method`

api_key, jwt, none

Values for `result`

allowed, denied

Behavior

reason is none for an allowed request. Denied requests use a bounded reason such as missing_credentials, unknown_key, key_expired, jwt_bad_signature, jwt_untrusted_issuer, or jwt_identity_unmapped.

PromQL example

sum(rate(aisix_auth_decisions_total{result="denied"}[5m])) by (method, reason)

Latency Metrics

Inspect latency on one gateway or aggregate histogram buckets across gateway instances.

Metric entries: 6

Latency Aggregation and Labels

Use histograms for service-level dashboards and alerts across gateway instances because their _bucket, _sum, and _count series can be aggregated before using histogram_quantile(). Use summaries to inspect precomputed quantiles from one gateway instance. Summary quantiles cannot be aggregated across instances, so do not average them.

Each histogram has its own bucket boundaries, because the two distributions differ: end-to-end latency starts in the milliseconds, while time to first token cannot be faster than the upstream that produces the token. Both sets are configurable. env_id identifies the environment served by an AISIX gateway connected to AISIX Cloud and is unknown when the AISIX gateway is not connected to AISIX Cloud. status_class is one of 2xx, 3xx, 4xx, 5xx, or other. Per-key and per-user labels are excluded to keep the number of bucket series manageable; use usage analytics for those dimensions.

aisix_request_duration_seconds . Description: Request duration across proxy endpoints in the compatibility series. HTTP streams record time until the response starts; realtime records the full WebSocket session until it closes. . Type: summary . Label count: 3 labels

Labels

provider, model, status

Behavior

Summary quantiles are calculated by each AISIX instance and cannot be aggregated across instances.

aisix_llm_request_duration_seconds . Description: Detailed request duration for model-inference endpoints. HTTP streams record time until the response starts; realtime records the full WebSocket session until it closes. . Type: summary . Label count: 15 labels

Labels

Values for `inbound_protocol`

openai, anthropic, realtime

Values for `upstream_protocol`

openai, anthropic, bedrock, vertex, azure-openai, unknown

Values for `stream`

false, true

Values for `outcome`

success, client_error, rate_limited, upstream_error

success covers HTTP 200–399, client_error covers HTTP 400–499 except 429, rate_limited is HTTP 429, and all other statuses map to upstream_error.

aisix_proxy_request_duration_seconds . Description: Detailed request duration for all proxied traffic. HTTP streams record time until the response starts; realtime records the full WebSocket session until it closes. . Type: summary . Label count: 15 labels

Labels

Values for `inbound_protocol`

openai, anthropic, mcp, a2a, realtime, passthrough

Values for `upstream_protocol`

openai, anthropic, bedrock, vertex, azure-openai, unknown

Values for `stream`

false, true

Values for `outcome`

success, client_error, rate_limited, upstream_error

success covers HTTP 200–399, client_error covers HTTP 400–499 except 429, rate_limited is HTTP 429, and all other statuses map to upstream_error.

aisix_llm_time_to_first_token_seconds . Description: Time from the upstream attempt start to the first streamed frame for streaming chat-completions, messages, and responses requests. . Type: summary . Label count: 12 labels

Labels

endpoint, inbound_protocol, upstream_protocol, provider, model, upstream_model, provider_key_id, provider_key_name, api_key_id, team_id, user_id, user_name

Values for `inbound_protocol`

openai, anthropic

Values for `upstream_protocol`

openai, anthropic, bedrock, vertex, azure-openai, unknown

aisix_request_e2e_latency_seconds . Description: Client-perceived latency for chat completions, messages, and responses, plus agent-call latency for A2A, including the full duration of streams. . Type: histogram . Label count: 6 labels

Labels

env_id, endpoint, model, provider, status_class, streaming

Values for `status_class`

2xx, 3xx, 4xx, 5xx, other

Values for `streaming`

false, true

Behavior

Default buckets range from 5 milliseconds to 600 seconds, and are configurable with observability.metrics.buckets.request_e2e_latency. The low boundaries record fast responses such as cache hits and requests rejected before dispatch. Aggregate _bucket series before calculating percentiles across gateway instances.

Each covered model-inference request is observed once. Non-streaming requests and failures are recorded when the handler returns. Streaming requests are recorded when the stream finishes, including client cancellation; a canceled model stream retains its committed status and records the duration up to cancellation.

A2A calls use endpoint="/a2a". Dispatched calls record the agent-call lifetime, including the complete stream. A pre-dispatch rejection that reaches A2A accounting records zero duration, and an abandoned A2A stream records 499 in the 4xx status class. Calls rejected before A2A accounting are absent from this histogram.

PromQL example

histogram_quantile(
  0.90,
  sum by (le) (rate(aisix_request_e2e_latency_seconds_bucket[5m]))
)

aisix_request_ttft_seconds . Description: Time from the upstream attempt start to the first streamed frame for streaming chat-completions, messages, and responses requests. . Type: histogram . Label count: 6 labels

Labels

env_id, endpoint, model, provider, status_class, streaming

Values for `status_class`

2xx, 3xx, 4xx, 5xx, other

Values for `streaming`

true

Behavior

The first frame stops the timer even when it carries no visible generated output, such as a role-only opener or an Anthropic message_start event. Content, reasoning, and tool-call frames also stop it.

Default buckets range from 50 milliseconds to 300 seconds, and are configurable with observability.metrics.buckets.request_ttft. Lower the first boundaries when a nearby model server can produce output in under 50 milliseconds. Deployments that use only hosted providers can raise the floor to remove consistently empty buckets.

PromQL example

histogram_quantile(
  0.90,
  sum by (le) (rate(aisix_request_ttft_seconds_bucket[5m]))
)

Usage and Cost Metrics

Measure token volume, estimated spend, and normalized client usage.

Metric entries: 9

Which Endpoints Report Tokens

Every endpoint that receives token counts from the upstream records them here: chat-completions, completions, messages, responses, embeddings, rerank, the audio transcription routes, image generation and editing, and realtime sessions.

Two model-inference endpoints report no tokens because they are not billed that way — /v1/audio/speech is billed per input character and /v1/videos per video. Both still count as requests, so divide token totals by requests only within one endpoint, never across all of them.

The per-request token and spend series — the three aisix_llm_*_tokens_total counters and aisix_llm_spend_micro_usd_total — carry the same labels as the detailed request counters, so a query can join token volume to request outcomes on endpoint, model, provider, and the caller identity labels. The other two do not: aisix_tokens_consumed_total is labeled by provider and model only, and aisix_llm_tokens_by_client_total by client_type, model, and token_type.

Three further counters break out the tokens the upstream provider served from its own prompt cache. They carry the same labels as the token counters above, and each appears only once its value is non-zero, so a provider that reports no cache detail creates no series. They describe the **provider's** cache, not the AISIX response cache — that one is aisix_cache_requests_total under Cache Metrics, and the two are unrelated.

Providers report cache reads under two different accounting rules, which is why there are two read counters rather than one. aisix_llm_cached_input_tokens_total is the OpenAI shape, where the cached tokens are part of the reported prompt tokens and are therefore already inside aisix_llm_input_tokens_total. aisix_llm_cache_read_input_tokens_total and aisix_llm_cache_creation_input_tokens_total are the Anthropic shape, reported beside the input tokens rather than within them, so they are outside aisix_llm_input_tokens_total and inside aisix_llm_total_tokens_total.

Keeping them apart is what makes a cross-protocol query correct: the input a request really consumed is aisix_llm_input_tokens_total + aisix_llm_cache_read_input_tokens_total + aisix_llm_cache_creation_input_tokens_total, and the part of it served from cache is aisix_llm_cached_input_tokens_total + aisix_llm_cache_read_input_tokens_total. Both expressions hold for either provider shape. See Calculate Prompt Cache Hit Rate.

aisix_tokens_consumed_total . Description: The sum of total tokens across every endpoint that reports token usage, in the compatibility series with the broadest coverage. . Type: counter . Label count: 2 labels

Labels

provider, model

aisix_llm_input_tokens_total . Description: Input tokens reported by the upstream, across every endpoint that reports token usage. . Type: counter . Label count: 12 labels

Labels

endpoint, inbound_protocol, upstream_protocol, provider, model, upstream_model, provider_key_id, provider_key_name, api_key_id, team_id, user_id, user_name

Values for `inbound_protocol`

openai, anthropic, realtime

Values for `upstream_protocol`

openai, anthropic, bedrock, vertex, azure-openai, unknown

aisix_llm_output_tokens_total . Description: Output tokens reported by the upstream, across every endpoint that reports token usage. . Type: counter . Label count: 12 labels

Labels

endpoint, inbound_protocol, upstream_protocol, provider, model, upstream_model, provider_key_id, provider_key_name, api_key_id, team_id, user_id, user_name

Values for `inbound_protocol`

openai, anthropic, realtime

Values for `upstream_protocol`

openai, anthropic, bedrock, vertex, azure-openai, unknown

aisix_llm_total_tokens_total . Description: Total tokens reported by the upstream, across every endpoint that reports token usage. . Type: counter . Label count: 12 labels

Labels

endpoint, inbound_protocol, upstream_protocol, provider, model, upstream_model, provider_key_id, provider_key_name, api_key_id, team_id, user_id, user_name

Values for `inbound_protocol`

openai, anthropic, realtime

Values for `upstream_protocol`

openai, anthropic, bedrock, vertex, azure-openai, unknown

aisix_llm_cached_input_tokens_total . Description: Input tokens the upstream served from its prompt cache and reported as part of its prompt token count. . Type: counter . Label count: 12 labels

Labels

endpoint, inbound_protocol, upstream_protocol, provider, model, upstream_model, provider_key_id, provider_key_name, api_key_id, team_id, user_id, user_name

Values for `inbound_protocol`

openai, anthropic, realtime

Values for `upstream_protocol`

openai, anthropic, bedrock, vertex, azure-openai, unknown

Behavior

The OpenAI accounting shape: prompt_tokens_details.cached_tokens for OpenAI and OpenAI-compatible providers, prompt_cache_hit_tokens for DeepSeek, and cachedContentTokenCount for Gemini on Vertex AI. These tokens are already counted in aisix_llm_input_tokens_total, so adding the two together double-counts them.

Recorded only when the upstream reports the field. AISIX never infers a cache hit from response timing or prompt similarity.

aisix_llm_cache_read_input_tokens_total . Description: Input tokens the upstream served from its prompt cache and reported separately from its input token count. . Type: counter . Label count: 12 labels

Labels

endpoint, inbound_protocol, upstream_protocol, provider, model, upstream_model, provider_key_id, provider_key_name, api_key_id, team_id, user_id, user_name

Values for `inbound_protocol`

openai, anthropic, realtime

Values for `upstream_protocol`

openai, anthropic, bedrock, vertex, azure-openai, unknown

Behavior

The Anthropic accounting shape: cache_read_input_tokens for Anthropic and cacheReadInputTokens for Amazon Bedrock. These tokens are **not** in aisix_llm_input_tokens_total, and they are in aisix_llm_total_tokens_total.

aisix_llm_cache_creation_input_tokens_total . Description: Input tokens the upstream wrote into its prompt cache. . Type: counter . Label count: 12 labels

Labels

endpoint, inbound_protocol, upstream_protocol, provider, model, upstream_model, provider_key_id, provider_key_name, api_key_id, team_id, user_id, user_name

Values for `inbound_protocol`

openai, anthropic, realtime

Values for `upstream_protocol`

openai, anthropic, bedrock, vertex, azure-openai, unknown

Behavior

cache_creation_input_tokens for Anthropic and cacheWriteInputTokens for Amazon Bedrock. Like cache reads in the same shape, these tokens are outside aisix_llm_input_tokens_total and inside aisix_llm_total_tokens_total.

Providers bill cache writes above the standard input rate and cache reads well below it, so a workload that writes far more than it reads costs more than the same workload with caching switched off. Compare this counter with aisix_llm_cache_read_input_tokens_total to catch it.

aisix_llm_spend_micro_usd_total . Description: Estimated spend in micro-USD, where 1 USD equals 1,000,000 micro-USD. Recorded wherever the gateway resolves a price for the request. . Type: counter . Label count: 12 labels

Labels

endpoint, inbound_protocol, upstream_protocol, provider, model, upstream_model, provider_key_id, provider_key_name, api_key_id, team_id, user_id, user_name

Values for `inbound_protocol`

openai, anthropic, realtime

Values for `upstream_protocol`

openai, anthropic, bedrock, vertex, azure-openai, unknown

aisix_llm_tokens_by_client_total . Description: Token volume across every endpoint that reports token usage, grouped by normalized client, requested model, and token type. . Type: counter . Label count: 3 labels

Labels

client_type, model, token_type

Values for `client_type`

openai-python, openai-node, anthropic-python, anthropic-typescript, claude-code, codex, cline, roo-code, kilocode, zoo-code, github-copilot, cursor, opencode, qwen-code, gemini-cli, crush, zed, aider, vercel-ai-sdk, langchain, llamaindex, litellm, curl, python-requests, httpx, aiohttp, okhttp, go-http-client, node, postman, browser, other, unknown

An unrecognized User-Agent maps to other; a missing User-Agent maps to unknown. Deployments can extend this set with admin-defined mapping rules (observability.metrics.client_type_rules). Full user-agent strings and versions remain in request logs and usage analytics.

Values for `token_type`

input, output, total

total includes input, output, and Anthropic cache-creation and cache-read tokens.

Behavior

model is the caller-requested model name, matching the model label on the aisix_llm_* token series. Routing, semantic, ensemble, and fallback dispatch retain this alias instead of reporting the selected direct model.

The bounded client_type allowlist prevents a client-controlled user-agent from creating unbounded Prometheus cardinality. Concrete model aliases are bounded by the configured model set, but a wildcard alias records each concrete model name requested through that pattern. Restrict wildcard access and monitor label growth when cardinality matters.

Admin-defined mapping rules (observability.metrics.client_type_rules) can classify additional clients. Rules are matched before the built-in allowlist and emit a fixed, validated label value, so the label set stays bounded.

Aggregated across all labels, the cache-inclusive total aligns with aisix_llm_total_tokens_total for the same endpoint population. Individual series do not align because the two metric families use different label sets; the dedicated client-type series avoids multiplying the per-key token series by another label dimension.

Anthropic reports cache tokens separately from input tokens, so total can exceed input plus output.

PromQL example

sum by (client_type, model, token_type) (
  rate(aisix_llm_tokens_by_client_total[5m])
)

Rate Limit and Budget Metrics

Monitor rate-limit rejections and the latest quota or budget state for each label set.

Metric entries: 8

aisix_ratelimit_rejections_total . Description: Requests rejected by the shared rate-limit gate across proxy endpoints. . Type: counter . Label count: 3 labels

Labels

scope, layer, policy_id

Values for `scope`

requests, tokens

Concurrency-limit rejections use requests; the runtime does not emit a separate concurrency value.

Values for `layer`

api_key, model, mcp, policy

Behavior

policy_id is empty outside layer="policy"; on the policy layer it carries the configured policy ID.

aisix_ratelimit_remaining_requests . Description: Remaining request quota reported while processing a chat-completions request, grouped by API key and model. . Type: gauge . Label count: 2 labels

Labels

api_key_id, model

Behavior

model is the configured model the request resolved to — a wildcard alias reports the row, such as openai/*, not the concrete name the caller sent.

Retired to NaN within seconds of the API key being deleted, or rebound to another team or member. NaN rather than 0 because 0 is a meaningful reading on this gauge; every comparison against it is false, so an alert on a retired series stops firing. An aggregation over a family containing one reads NaN too.

aisix_ratelimit_remaining_tokens . Description: Remaining token quota reported while processing a chat-completions request, grouped by API key and model. . Type: gauge . Label count: 2 labels

Labels

api_key_id, model

Behavior

model is the configured model the request resolved to — a wildcard alias reports the row, such as openai/*, not the concrete name the caller sent.

aisix_budget_limit_usd . Description: Budget limit in USD. . Type: gauge . Label count: 4 labels

Labels

api_key_id, team_id, user_id, user_name

Behavior

Removing a budget from a key does not zero this gauge — it holds the last budgeted value, which is why queries guard on aisix_budget_details_present == 1.

aisix_budget_spent_usd . Description: Budget spent in USD. . Type: gauge . Label count: 4 labels

Labels

api_key_id, team_id, user_id, user_name

Behavior

Removing a budget from a key does not zero this gauge — it holds the last budgeted value, which is why queries guard on aisix_budget_details_present == 1.

aisix_budget_remaining_usd . Description: Budget remaining in USD. . Type: gauge . Label count: 4 labels

Labels

api_key_id, team_id, user_id, user_name

Behavior

Removing a budget from a key does not zero this gauge — it holds the last budgeted value, which is why queries guard on aisix_budget_details_present == 1.

aisix_budget_reset_seconds . Description: Seconds until the budget period resets. . Type: gauge . Label count: 4 labels

Labels

api_key_id, team_id, user_id, user_name

Behavior

Removing a budget from a key does not zero this gauge — it holds the last budgeted value, which is why queries guard on aisix_budget_details_present == 1.

aisix_budget_details_present . Description: Whether budget details are populated. . Type: gauge . Label count: 4 labels

Labels

api_key_id, team_id, user_id, user_name

Values for `metric value`

0, 1

1 means budget details are present; 0 means they have been cleared.

Cache Metrics

Measure response-cache effectiveness per policy and watch the semantic layer’s embedding and store health.

Metric entries: 4

Cache Hit Rate

Every request covered by an enabled cache policy with an available backend records one aisix_cache_requests_total observation. Requests with no matching policy or an unavailable backend are not counted — the gate never opened — so the series measures policy effectiveness, not total traffic.

Hit rate per policy: sum by (policy) (rate(aisix_cache_requests_total{outcome=~"hit_exact|hit_semantic"}[5m])) / sum by (policy) (rate(aisix_cache_requests_total[5m])). Split by outcome to see how much the semantic layer adds over exact matching.

Semantic-layer failures degrade to ordinary misses, which makes a broken embedding model or store indistinguishable from a healthy low hit rate in the outcome counter alone. Alert on aisix_cache_semantic_embedding_failures_total and aisix_cache_semantic_store_failures_total to separate the two.

aisix_cache_requests_total . Description: Cache-eligible requests by policy name and outcome, counted once per request when a matching enabled policy with an available backend opened the cache gate. . Type: counter . Label count: 2 labels

Labels

policy, outcome

Values for `outcome`

hit_exact, hit_semantic, miss, bypass

bypass means the caller sent Cache-Control: no-cache, which skips the read path. no-store keeps the read path active, so it records an ordinary hit or miss and only suppresses a write after a miss.

aisix_cache_semantic_embedding_seconds . Description: Latency of the semantic layer’s embedding calls, per policy. Summary series without fixed buckets. . Type: summary . Label count: 1 label

Labels

policy

aisix_cache_semantic_embedding_failures_total . Description: Embedding failures on the semantic layer. Failed requests proceed to the upstream uncached. . Type: counter . Label count: 2 labels

Labels

policy, cause

Values for `cause`

resolve, embed

resolve (embedding model missing or not an embedding model) is counted once per eligible request, including requests that then hit the exact layer; embed (provider call failed or timed out) is counted per embedding call.

aisix_cache_semantic_store_failures_total . Description: Semantic-store operation failures by operation. The in-process store cannot fail; shared (Redis) stores can. Failures degrade to an ordinary miss. . Type: counter . Label count: 2 labels

Labels

policy, op

Values for `op`

lookup, store

Deployment Metrics

Monitor how each target model behaves: upstream attempts and their outcomes, fallbacks between targets, and whether a target remains in rotation.

Metric entries: 7

Attempts, Not Requests

These counters sample once per upstream **attempt**, and each sample is filed under the target that was attempted rather than the Model Group the caller named. One client request that failed over across three targets is three samples here and one sample in the request metrics. Use this family to ask which target is failing, and the request family to ask what callers experienced.

Only attempts that reached the upstream are counted. An attempt the gateway refused on its own — the target was over its own rate limit, or its credentials or endpoint failed validation before anything was sent — produced no upstream response, so it is absent here while still appearing in the usage log. That keeps a misconfiguration from reading as an unhealthy target.

Emitted by the endpoints that dispatch through a Model Group: /v1/chat/completions, /v1/messages, and /v1/responses. The other endpoints call one model per request and are fully described by the request metrics.

aisix_deployment_requests_total . Description: Upstream attempts dispatched to a target model, whatever their outcome. . Type: counter . Label count: 4 labels

Labels

provider, model, upstream_model, provider_key_id

PromQL example

sum(rate(aisix_deployment_requests_total[5m])) by (model)

aisix_deployment_success_responses_total . Description: Upstream attempts a target model answered successfully. . Type: counter . Label count: 4 labels

Labels

provider, model, upstream_model, provider_key_id

Behavior

For a streamed response, an attempt counts as successful once the upstream stream is established. A stream that breaks after that point is not reclassified here.

aisix_deployment_failure_responses_total . Description: Upstream attempts that failed at the target model, including the failures a fallback went on to rescue. . Type: counter . Label count: 4 labels

Labels

provider, model, upstream_model, provider_key_id

PromQL example

sum(rate(aisix_deployment_failure_responses_total[5m])) by (model)
  / sum(rate(aisix_deployment_requests_total[5m])) by (model)

aisix_routing_successful_fallbacks_total . Description: Fallback attempts that served the request after an earlier target failed. . Type: counter . Label count: 2 labels

Labels

model, fallback_model

Behavior

model is what the caller asked for — the Model Group name. fallback_model is the target the gateway moved to.

aisix_routing_failed_fallbacks_total . Description: Fallback attempts that failed in turn. . Type: counter . Label count: 2 labels

Labels

model, fallback_model

Behavior

A request rescued by its second fallback contributes one sample here and one to the successful family.

aisix_deployment_state . Description: Whether a target model is in rotation. . Type: gauge . Label count: 4 labels

Labels

provider, model, upstream_model, provider_key_id

Values for `metric value`

0, 2

0 is healthy. 2 is out of rotation because the target is cooling down or failing a background health check.

aisix_deployment_cooled_down_total . Description: Number of times a target model entered cooldown. . Type: counter . Label count: 4 labels

Labels

provider, model, upstream_model, provider_key_id

Guardrail Metrics

Track per-guardrail execution latency and outcomes, plus aggregate blocks and fail-open bypasses, across all endpoints.

Metric entries: 3

Guardrail Execution Latency

Every timed guardrail execution records an aisix_guardrail_latency_seconds observation. A guardrail normally runs once per applicable phase, while streaming window scans can record several output executions for one response. The histogram uses configurable buckets that default to 1 ms through 30 s, so histogram_quantile yields P50/P95/P99 per guardrail.

The kind label identifies the configured guardrail implementation, and the result label separates fail-open bypasses (bypassed, with the failure tag in error_type) from policy decisions. A fail-closed evaluation failure surfaces as blocked; filter on error_type to distinguish it from a policy match.

The _count series doubles as a per-guardrail execution counter: sum by (guardrail, result) (rate(aisix_guardrail_latency_seconds_count[5m])) gives execution and block rates without a separate counter.

The aggregate counters cover guarded endpoints but count different units. aisix_guardrail_blocks_total counts rejected requests, while aisix_guardrail_bypasses_total counts bypass events, including proxy-raised passes before a guardrail member executes. They omit guardrail, kind, and phase labels; use the histogram _count series for timed execution dimensions.

aisix_guardrail_latency_seconds . Description: Wall-clock duration of one timed guardrail execution across all endpoints. guardrail is the configured guardrail name. . Type: histogram . Label count: 6 labels

Labels

env_id, guardrail, kind, phase, result, error_type

Values for `kind`

keyword, pii, aliyun_ai_guardrail, aliyun_text_moderation, azure_content_safety, azure_content_safety_text_moderation, bedrock, lakera, openai_moderation, presidio, semantic, custom

keyword and pii run in-process. semantic calls a configured embedding model, custom runs the configured script, and the remaining kinds call moderation services.

Values for `phase`

input, output

Values for `result`

allowed, blocked, masked, bypassed, would_block, would_mask

bypassed is an evaluation failure that failed open; a fail-closed evaluation failure records as blocked. would_block / would_mask come from enforcement_mode: monitor guardrails.

Values for `error_type`

none, aliyun_5xx, aliyun_config_error, aliyun_bad_response, aliyun_throttled, aliyun_timeout, azure_cs_5xx, azure_cs_config_error, azure_cs_throttled, azure_cs_timeout, bedrock_5xx, bedrock_throttled, bedrock_timeout, bedrock_too_large, lakera_5xx, lakera_config_error, lakera_throttled, lakera_timeout, lakera_too_large, openai_moderation_5xx, openai_moderation_config_error, openai_moderation_throttled, openai_moderation_timeout, openai_moderation_too_large, presidio_5xx, presidio_config_error, presidio_throttled, presidio_timeout, presidio_too_large, custom_timeout, custom_script_error, custom_engine_error, custom_no_verdict, custom_bad_verdict, custom_unknown_action, semantic_embed_unresolved, semantic_embed_timeout, semantic_embed_upstream

Set to the bounded failure tag when an evaluation failure produces result="blocked", result="bypassed", or a monitor-mode result="would_block"; otherwise none. The bypassed values are also used by aisix_guardrail_bypasses_total{reason}. That counter additionally uses unscannable_body for proxy-raised passes that do not create a timed histogram observation.

Behavior

Default buckets: 0.001, 0.0025, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10, and 30 seconds, configurable with observability.metrics.buckets.guardrail_latency.

Monitor-mode executions record would_block / would_mask while the request proceeds, so a staged policy can be sized before enforcing it.

PII and keyword masking performed by synchronous per-field operations is not timed by this series; its in-process cost is measured in microseconds.

aisix_guardrail_blocks_total . Description: Requests rejected by input or output guardrail enforcement across all endpoints, including fail-closed paths such as a streamed-output buffer overflow before a guardrail member executes. . Type: counter . Label count: no labels

Labels

None.

aisix_guardrail_bypasses_total . Description: Guardrail evaluation failures or proxy-raised unreadable-content passes where a fail-open policy let the request or response continue, across all guarded endpoints. . Type: counter . Label count: 1 label

Labels

reason

Values for `reason`

aliyun_5xx, aliyun_config_error, aliyun_bad_response, aliyun_throttled, aliyun_timeout, azure_cs_5xx, azure_cs_config_error, azure_cs_throttled, azure_cs_timeout, bedrock_5xx, bedrock_throttled, bedrock_timeout, bedrock_too_large, lakera_5xx, lakera_config_error, lakera_throttled, lakera_timeout, lakera_too_large, openai_moderation_5xx, openai_moderation_config_error, openai_moderation_throttled, openai_moderation_timeout, openai_moderation_too_large, presidio_5xx, presidio_config_error, presidio_throttled, presidio_timeout, presidio_too_large, custom_timeout, custom_script_error, custom_engine_error, custom_no_verdict, custom_bad_verdict, custom_unknown_action, semantic_embed_unresolved, semantic_embed_timeout, semantic_embed_upstream, unscannable_body

A2A Metrics

Measure Agent-to-Agent traffic by the agent that was reached and the operation that was invoked.

Metric entries: 4

Agent and Operation Labels

agent is a registered A2A agent, and operation is the canonical operation a caller invoked. A2A 0.3 and 1.0 spell one operation two ways, so AISIX records the canonical form and a deployment fronting both versions still aggregates as one series. An unrecognized method becomes unknown.

Task ids, context ids, and JSON-RPC request ids are deliberately absent. They are what makes an individual call traceable, which is exactly what makes them unusable as label values; find them in usage events and traces instead.

aisix_a2a_requests_total does not agree with aisix_proxy_requests_total{endpoint="/a2a"}, by design. A call refused before its agent is resolved has no agent to file under and is counted only in the proxy family, and a stream the caller abandons is a 4xx here but a 2xx there, because the response really did begin as a 200. Read this family for agent health and the proxy family for route traffic.

For streaming A2A, aisix_proxy_request_duration_seconds records time until the response starts. For dispatched calls, aisix_request_e2e_latency_seconds{endpoint="/a2a"} records the agent-call lifetime, including the complete stream.

aisix_a2a_requests_total . Description: A2A calls by the agent reached, the canonical operation invoked, and the status class. . Type: counter . Label count: 3 labels

Labels

agent, operation, status

Behavior

Counted at the same point the usage event is emitted, so a call that is accounted for is also metered.

PromQL example

sum by (agent, operation) (
  rate(aisix_a2a_requests_total{status!="2xx"}[5m])
)

aisix_a2a_ttfb_seconds . Description: Time to the upstream agent's first streamed event on a streaming A2A call. . Type: histogram . Label count: 2 labels

Labels

agent, operation

Behavior

Named for the event rather than a token because an agent stream carries task updates, not tokens. Default buckets match aisix_request_ttft_seconds, and are configurable with observability.metrics.buckets.a2a_ttfb.

Observed only on streaming operations that produced at least one event.

PromQL example

histogram_quantile(
  0.90,
  sum by (le, agent) (rate(aisix_a2a_ttfb_seconds_bucket[5m]))
)

aisix_a2a_stream_events_total . Description: Events relayed downstream on streaming A2A calls. . Type: counter . Label count: 2 labels

Labels

agent, operation

Behavior

Divided by aisix_a2a_requests_total over the streaming operations, it gives events per call — how chatty an agent is, and whether that changed. Both message/stream and tasks/resubscribe stream, so filter the denominator to both or the ratio is inflated.

PromQL example

sum by (agent) (rate(aisix_a2a_stream_events_total[5m]))
/
sum by (agent) (
  rate(aisix_a2a_requests_total{operation=~"message/stream|tasks/resubscribe"}[5m])
)

aisix_a2a_task_state_total . Description: A2A calls by the task state the agent last reported, normalized to the specification's set plus unknown. . Type: counter . Label count: 2 labels

Labels

agent, state

Behavior

A cumulative counter of the state each call ENDED on, so read it as a rate rather than as a live backlog: a task that later moves on does not decrement an earlier sample, and a client polling one task with tasks/get reports its state on every poll. A call whose upstream never answered reports no state and is not counted here.

PromQL example

sum by (agent, state) (rate(aisix_a2a_task_state_total[5m]))

Usage Event Metrics

Distinguish usage-event emission attempts from events the delivery queue did not accept.

Metric entries: 2

Usage Event Delivery

For each emission attempt, the event is either accepted by the queue or counted as a drop. Subtract the drop rate from the emission-attempt rate to calculate the accepted rate.

Both counters carry the same model, provider_key_id, and provider_key_name labels, so that subtraction also holds per model and per provider key. Use it to identify whose queue handoffs were rejected. A rising drop rate concentrated on one model or one provider key points at traffic whose usage records never reached the worker.

Queue acceptance is not delivery to storage. After the worker accepts an event, control-plane shipping can still fail (telemetry batch failed (events dropped)) and exporter pipelines can still drop after retries (sink delivery dropped after retries). Neither increments aisix_usage_event_drops_total.

The status_code label is one of 2xx, 3xx, 4xx, 5xx, or other, while status carries the raw HTTP code alongside it — status names one failure mode, status_code rolls a family up, and both describe the same event. Example handler values include chat, embeddings, messages, and mcp.

user_id is the organization member who owned the API key the request authenticated with, and user_name is that member's display name. Both are unknown when the key belongs to no member. They appear on both counters, so emitted = delivered + dropped still holds per member. Because a JWT-authenticated request runs as the key its identity resolves to, one member label covers every credential that member calls through. The name has a one-to-one relationship with the ID, so it does not add another series dimension. It is the name the control plane stamped when it last projected that API key, not a live lookup: a member renamed afterwards keeps their old name on these metrics until that key is written again for some other reason.

model is the model the caller addressed, collapsed to a configured model name. A request that carried no resolvable model reports unresolved, and one with no model at all — an MCP tool call, an A2A agent call, a passthrough route — reports unknown, as do both provider-key labels wherever no upstream key was resolved.

MCP tool calls use handler="mcp" and inbound_protocol="mcp". Their usage-event payloads identify the MCP server and tool; token and cost fields are zero.

A2A agent calls use handler="a2a". The metric maps their bounded inbound_protocol label to other, while the delivered event uses inbound_protocol="a2a" and identifies the agent name and JSON-RPC method. The gateway estimates prompt and completion tokens from message text and sets usage_estimated: true; cost remains zero.

aisix_usage_events_emitted_total . Description: Usage-event emission attempts, counted before AISIX tries to enqueue the event, including events the queue accepted and events it rejected. . Type: counter . Label count: 10 labels

Labels

handler, status_code, status, inbound_protocol, upstream_protocol, model, provider_key_id, provider_key_name, user_id, user_name

Values for `handler`

a2a, audio, batch, batches, chat, completions, count_tokens, embeddings, files, fine_tuning, images, mcp, messages, passthrough_route, realtime, rerank, responses, videos

Values for `status_code`

2xx, 3xx, 4xx, 5xx, other

Values for `inbound_protocol`

openai, anthropic, mcp, other

other includes A2A, realtime, passthrough, and every other value outside the three named protocol buckets.

aisix_usage_event_drops_total . Description: Usage events that were not accepted by the queue. . Type: counter . Label count: 7 labels

Labels

reason, model, provider_key_id, provider_key_name, upstream_protocol, user_id, user_name

Values for `reason`

sink_disabled, sink_full, sink_closed

PromQL example

sum(rate(aisix_usage_event_drops_total[5m])) by (model, provider_key_name, reason)

Configuration Metrics

Monitor whether AISIX can load and apply changes from its configuration source.

Metric entries: 11

Configuration State

AISIX reflects the same live configuration state in these metrics and at GET /status/config on the metrics and status listener.

Reload metrics apply to both file and etcd configuration sources. Revision and source-connection metrics are emitted only when AISIX loads configuration from etcd.

Compare the observed and applied revisions to determine whether the gateway is serving the latest etcd configuration.

aisix_config_last_reload_successful . Description: Whether the latest configuration load succeeded. . Type: gauge . Label count: no labels

Labels

None.

Values for `metric value`

0, 1

1 indicates success and 0 indicates failure.

aisix_config_last_reload_success_timestamp_seconds . Description: Unix timestamp in seconds of the last successful configuration load. . Type: gauge . Label count: no labels

Labels

None.

Behavior

The series is not emitted until a configuration load succeeds.

aisix_config_reloads_total . Description: Full configuration reload attempts, including source fetch failures. . Type: counter . Label count: no labels

Labels

None.

Behavior

Incremental etcd watch events do not increment this counter.

aisix_config_reload_failures_total . Description: Configuration reload failures grouped by reason. . Type: counter . Label count: 1 label

Labels

reason

Values for `reason`

fetch, parse, validate

fetch indicates that the source could not be read, parse indicates invalid source data, and validate indicates an invalid resource.

aisix_config_rejected_resources . Description: Current number of rejected resources, grouped by resource kind. . Type: gauge . Label count: 1 label

Labels

kind

Behavior

When all rejections for a resource kind clear, AISIX sets its existing series to 0.

aisix_config_partially_compatible_resources . Description: Served resources carrying at least one field this gateway version does not recognize, grouped by resource kind. . Type: gauge . Label count: 1 label

Labels

kind

Behavior

A resource with several ignored fields counts once. Inspect partially_compatible at GET /status/config for the field paths and per-field counts.

When all partially compatible resources for a kind clear, AISIX sets its existing series to 0.

aisix_config_stale_served_resources . Description: Resources whose latest source value was rejected while their last known good value remains in service, grouped by resource kind. . Type: gauge . Label count: 1 label

Labels

kind

Behavior

Inspect rejected at GET /status/config to identify each resource and when stale serving began.

When no stale value remains in service for a kind, AISIX sets its existing series to 0.

aisix_config_observed_revision . Description: Latest etcd revision observed by the gateway. . Type: gauge . Label count: no labels

Labels

None.

aisix_config_applied_revision . Description: The etcd revision represented by the configuration currently served by the gateway. . Type: gauge . Label count: no labels

Labels

None.

aisix_config_hash_info . Description: Applied configuration hash. . Type: gauge . Label count: 1 label

Labels

hash

Values for `metric value`

0, 1

Filter for value 1 to select the current hash. AISIX retains earlier hash labels with value 0 after the applied configuration changes.

aisix_config_source_connected . Description: Whether the gateway is connected to its etcd configuration source. . Type: gauge . Label count: no labels

Labels

None.

Values for `metric value`

0, 1

1 indicates connected and 0 indicates disconnected.

Analyze Metrics with PromQL

Use these PromQL examples in the Prometheus expression browser or another compatible monitoring interface after configuring it to scrape the AISIX metrics endpoint. Adjust the time window, label filters, and grouping dimensions to match the traffic and gateway instances you want to inspect.

Calculate Success Rate

Calculate the success rate by dividing the successful request rate by the total request rate. The following query combines all model-inference traffic over a five-minute window:

sum(rate(aisix_llm_requests_total{outcome="success"}[5m]))
  /
sum(rate(aisix_llm_requests_total[5m]))

Add an endpoint filter or group by endpoint when you want to analyze one API separately.

To include traffic that is not counted as model inference — MCP tool calls, A2A agent calls, passthrough routes (endpoint label /passthrough_route), and the file, batch, and fine-tuning routes — run the same query against aisix_proxy_requests_total, which carries every proxied request.

To measure only the primary routing path, restrict the numerator and denominator to requests that were not served by a fallback target:

sum(rate(aisix_llm_requests_total{outcome="success", is_fallback="false"}[5m]))
  /
sum(rate(aisix_llm_requests_total{is_fallback="false"}[5m]))

Whether rate-limited requests belong in the population is an operational policy decision. To exclude clients that reached their quota from the denominator, use:

sum(rate(aisix_llm_requests_total{outcome="success"}[5m]))
  /
sum(rate(aisix_llm_requests_total{outcome!="rate_limited"}[5m]))

Separate Cross-Protocol Conversion Traffic

Every detailed request, latency, and token series carries two protocol labels: inbound_protocol, the protocol the caller spoke to AISIX, and upstream_protocol, the protocol AISIX spoke to the upstream that served the request. Group by both to get a conversion matrix, where the diagonal is native traffic and every other cell is traffic AISIX translated:

sum by (inbound_protocol, upstream_protocol) (rate(aisix_llm_requests_total[5m]))

Conversion has a cost in fidelity — features that exist on one side of a translation have no counterpart on the other — so it is worth knowing how much of the fleet depends on it. To isolate one direction, name both ends:

# Anthropic-protocol callers served by an OpenAI-shape upstream
sum(rate(aisix_llm_requests_total{inbound_protocol="anthropic", upstream_protocol="openai"}[5m]))

Compare reliability across upstream protocols to tell a provider problem apart from a translation problem. If one protocol's success rate diverges while the callers are the same, the difference is on the upstream side:

sum by (upstream_protocol) (rate(aisix_llm_requests_total{outcome="success"}[5m]))
  /
sum by (upstream_protocol) (rate(aisix_llm_requests_total[5m]))

upstream_protocol reports unknown when a request selected no upstream at all — an unknown model, an input guardrail block, an AISIX Cloud budget refusal, or a body refused before dispatch. Those requests are real failures, so keep them in the denominator of a success rate and exclude them only when you are specifically comparing upstreams.

Do not derive the upstream protocol from provider instead. provider is an open vendor string, the same vendor can be reached through more than one adapter, and a custom vendor served over an OpenAI-compatible endpoint carries whatever name the operator chose. A hand-maintained vendor-to-protocol mapping in PromQL misreports exactly the traffic this label exists to describe.

Calculate Aggregate Latency Percentiles

Combine histogram buckets across gateway instances before calculating a percentile. P90 is the value at or below which 90% of observations fall. Keep le in the sum by grouping, and add labels such as model or provider when you need a breakdown:

# P90 end-to-end latency across all matched gateway instances
histogram_quantile(
  0.90,
  sum by (le) (rate(aisix_request_e2e_latency_seconds_bucket{status_class="2xx"}[5m]))
)

# P90 end-to-end latency per model
histogram_quantile(
  0.90,
  sum by (le, model) (rate(aisix_request_e2e_latency_seconds_bucket{status_class="2xx"}[5m]))
)

# P90 time to first streamed frame per provider
histogram_quantile(
  0.90,
  sum by (le, provider) (rate(aisix_request_ttft_seconds_bucket[5m]))
)

Streaming end-to-end time covers the full generation, so streaming and non-streaming requests have different latency distributions. Use the streaming label to analyze them separately:

# P90 end-to-end latency for successful streaming requests
histogram_quantile(
  0.90,
  sum by (le) (rate(aisix_request_e2e_latency_seconds_bucket{streaming="true", status_class="2xx"}[5m]))
)

Compare Single-Instance Streaming Latency

Summary series expose precomputed quantile labels for each gateway instance. Select one scrape target and the model or provider you want to inspect:

# P90 time to first streamed frame for streaming chat completions
aisix_llm_time_to_first_token_seconds{endpoint="/v1/chat/completions", quantile="0.9"}

# P90 time to response start for that same traffic
aisix_llm_request_duration_seconds{endpoint="/v1/chat/completions", stream="true", quantile="0.9"}

Both queries pin endpoint because the two series do not cover the same traffic by default. Time to first streamed frame is recorded for streaming /v1/chat/completions, /v1/messages, and /v1/responses, while the duration series covers every model-inference endpoint. Without the filter, traffic on the other model-inference endpoints can move the duration P90 without contributing to the TTFT P90.

aisix_llm_request_duration_seconds is recorded when the gateway hands the response to the client. On a streamed request that happens before any frame is read, so the value covers the work up to response start and excludes the whole generation. It is not an end-to-end figure for streaming traffic and does not become one by filtering on stream="true". The same applies to aisix_request_duration_seconds and aisix_proxy_request_duration_seconds, which are recorded at the same moment. On non-streamed traffic all three do cover the full request.

For a streamed request's end-to-end latency use aisix_request_e2e_latency_seconds, which is recorded at stream completion. It is a histogram rather than a summary, so read it with the quantile queries above rather than per-instance.

For dispatched A2A calls, the same histogram uses endpoint="/a2a" and records the agent-call lifetime. A pre-dispatch rejection that reaches A2A accounting records zero duration. For a streaming call, aisix_proxy_request_duration_seconds stops when the response starts, while the histogram continues until the stream finishes; an abandoned stream records 499 in the 4xx status class.

Do not average summary quantiles across instances. Use the histogram queries above to calculate percentiles across gateway instances.

Measure Guardrail Latency and Outcomes

aisix_guardrail_latency_seconds records each timed guardrail execution, labeled with the guardrail name, kind, phase, and result. A guardrail normally runs once per applicable phase. Streaming window scans can run the same output guardrail several times for one response. Synchronous per-field PII and keyword masking operations are not included. Calculate P95 execution latency per guardrail to verify a moderation budget:

histogram_quantile(
  0.95,
  sum by (le, guardrail) (rate(aisix_guardrail_latency_seconds_bucket[5m]))
)

Because guardrails run sequentially, their total contribution to request latency is the sum of their execution times for that request. Calculate the mean execution time for each guardrail and phase, and compare it with your moderation budget:

sum by (guardrail, phase) (rate(aisix_guardrail_latency_seconds_sum[5m]))
/
sum by (guardrail, phase) (rate(aisix_guardrail_latency_seconds_count[5m]))

Compare local detection with remote moderation services using the kind label. keyword and pii run in-process; every other kind invokes a remote service:

histogram_quantile(
  0.95,
  sum by (le, kind) (rate(aisix_guardrail_latency_seconds_bucket[5m]))
)

The _count series doubles as an execution counter. Track block and fail-open bypass rates per guardrail, and alert when a remote guardrail starts failing open:

sum by (guardrail, result) (rate(aisix_guardrail_latency_seconds_count[5m]))

# Fail-open bypasses by failure cause
sum by (guardrail, error_type) (rate(aisix_guardrail_latency_seconds_count{result="bypassed"}[5m]))

Calculate Token Volume by Client

Use token_type="total" to calculate cache-inclusive token volume by normalized client type:

sum by (client_type) (rate(aisix_llm_tokens_by_client_total{token_type="total"}[5m]))

To compare input and output volume, select both token types and include token_type in the grouping:

sum by (client_type, token_type) (rate(aisix_llm_tokens_by_client_total{token_type=~"input|output"}[5m]))

Break Down Token Volume by Client Type and Model

The model label records the model name the client requested, not the direct model AISIX selected. Group by client_type and model to see how each normalized client type distributes token volume across models:

sum by (client_type, model) (rate(aisix_llm_tokens_by_client_total{token_type="total"}[5m]))

To inspect one client type, filter on client_type and group by model only:

sum by (model) (rate(aisix_llm_tokens_by_client_total{client_type="claude-code", token_type="total"}[5m]))

Calculate Prompt Cache Hit Rate

Upstream providers cache prompt prefixes and bill cached input well below the standard rate, so the share of input served from that cache is the single number that says whether prompt caching is paying off. AISIX reports it in three counters, and the split between them is what makes the query correct across providers:

Counter	Reported by the upstream as	Relationship to `aisix_llm_input_tokens_total`
`aisix_llm_cached_input_tokens_total`	part of the prompt token count	already inside it
`aisix_llm_cache_read_input_tokens_total`	a counter beside the input tokens	outside it
`aisix_llm_cache_creation_input_tokens_total`	a counter beside the input tokens	outside it

So the input a request really consumed is input + cache_read + cache_creation, and the part of it that came from cache is cached + cache_read. Both expressions hold whichever shape the provider reports, which makes this hit rate comparable across a mixed fleet:

(
  sum(rate(aisix_llm_cached_input_tokens_total[5m]))
  + sum(rate(aisix_llm_cache_read_input_tokens_total[5m]))
)
  /
(
  sum(rate(aisix_llm_input_tokens_total[5m]))
  + sum(rate(aisix_llm_cache_read_input_tokens_total[5m]))
  + sum(rate(aisix_llm_cache_creation_input_tokens_total[5m]))
)

Add by (model) to every sum to find which workloads benefit. A prompt with a stable prefix — a long system prompt, a fixed tool schema, a pinned document — should settle at a high rate; one that rewrites its opening on every call stays near zero and is the candidate to restructure.

A cache write costs more than an ordinary input token while a cache read costs much less, so a workload that writes far more than it reads is paying a premium for a cache nobody hits. Watch the ratio for the providers that report both:

sum by (model) (rate(aisix_llm_cache_creation_input_tokens_total{upstream_protocol=~"anthropic|bedrock"}[5m]))
  /
sum by (model) (rate(aisix_llm_cache_read_input_tokens_total{upstream_protocol=~"anthropic|bedrock"}[5m]))

A ratio well above 1 sustained over a long window means the cache entries expire before they are reused. Shorten the interval between calls that share a prefix, or stop caching that prompt.

Each counter is created only once its value is non-zero, so a provider that reports no cache detail contributes no series and sum() over it returns nothing rather than zero. Wrap a query in or vector(0) when an alert must evaluate before any cached traffic has been seen.

These counters describe the provider's prompt cache. The AISIX response cache is a separate mechanism measured by aisix_cache_requests_total — see Measure Response Cache Effectiveness.

Calculate Spend and Cost per Request

aisix_llm_spend_micro_usd_total counts micro-USD, where 1 USD is 1,000,000 micro-USD. Divide by 1e6 for a figure in dollars:

# USD per hour, by team
sum by (team_id) (rate(aisix_llm_spend_micro_usd_total[5m])) * 3600 / 1e6

# Projected 30-day spend at the last day's rate
sum(rate(aisix_llm_spend_micro_usd_total[24h])) * 86400 * 30 / 1e6

Because the spend counter shares its labels with the request counter, average cost per request is a direct division. Group both sides by the same labels:

sum by (model) (rate(aisix_llm_spend_micro_usd_total[5m])) / 1e6
  /
sum by (model) (rate(aisix_llm_requests_total[5m]))

Spend is recorded only where AISIX resolves a price for the request, while the request counter counts every request. On a deployment where some models carry no configured cost, the denominator includes their traffic and the average reads low. Filter both sides to the priced models, or compare each model separately, rather than reading one fleet-wide number.

Calculate Tokens per Request

sum by (endpoint, model) (rate(aisix_llm_total_tokens_total[5m]))
  /
sum by (endpoint, model) (rate(aisix_llm_requests_total[5m]))

Keep endpoint in the grouping. /v1/audio/speech is billed per input character and /v1/videos per video, so both count as requests and report no tokens; summing across endpoints pulls the average down by however much of that traffic there is.

aisix_llm_total_tokens_total is cache-inclusive: it contains input, output, and the cache-read and cache-creation tokens that providers report beside their input count. That makes it the right numerator for a per-request consumption figure and the wrong one for a cache hit rate — use the query in Calculate Prompt Cache Hit Rate for that.

Rank the Heaviest Consumers

The token and spend counters carry the full caller identity, so topk answers who to talk to about a bill:

# Ten API keys with the highest token rate
topk(10, sum by (api_key_id) (rate(aisix_llm_total_tokens_total[1h])))

# Ten users with the highest spend rate, in USD per hour
topk(10, sum by (user_id, user_name) (rate(aisix_llm_spend_micro_usd_total[1h])) * 3600 / 1e6)

# Which models one team spends on
sum by (model) (rate(aisix_llm_spend_micro_usd_total{team_id="team-platform"}[1h])) * 3600 / 1e6

user_name and provider_key_name are one-to-one with their IDs, so including a name in the grouping adds no series. user_name reports unknown until the control plane supplies it.

Measure Response Cache Effectiveness

aisix_cache_requests_total counts requests that reached an enabled cache policy with an available backend. Requests with no matching policy are never counted, so this measures how well the configured policies work, not what share of all traffic is cached:

sum by (policy) (rate(aisix_cache_requests_total{outcome=~"hit_exact|hit_semantic"}[5m]))
  /
sum by (policy) (rate(aisix_cache_requests_total[5m]))

Split by outcome to see how much the semantic layer adds over exact matching, which is what justifies the embedding call it costs:

sum by (policy, outcome) (rate(aisix_cache_requests_total[5m]))

A semantic-layer failure degrades to an ordinary miss, so a broken embedding model or store looks exactly like a healthy low hit rate in the outcome counter. Alert on the failure counters to tell them apart:

sum by (policy, cause) (rate(aisix_cache_semantic_embedding_failures_total[5m])) > 0
sum by (policy, op) (rate(aisix_cache_semantic_store_failures_total[5m])) > 0

This cache and the upstream provider's prompt cache are independent, and they interact in one direction worth knowing: a request served from the AISIX response cache never reaches the upstream, so it contributes nothing to the provider cache counters. A rising response-cache hit rate therefore lowers the absolute prompt-cache token rates without meaning that prompt caching got worse.

Measure Target Health and Fallback

Request metrics say what callers experienced; deployment metrics say which target was responsible. A Model Group whose success rate looks healthy can be hiding one consistently failing target that fallback keeps rescuing. Compare the two:

# Failure rate per attempted target
sum by (model, upstream_model, provider_key_id) (rate(aisix_deployment_failure_responses_total[5m]))
  /
sum by (model, upstream_model, provider_key_id) (rate(aisix_deployment_requests_total[5m]))

# Share of client requests that a fallback target served
sum(rate(aisix_llm_requests_total{is_fallback="true"}[5m]))
  /
sum(rate(aisix_llm_requests_total[5m]))

A rising fallback share with a flat success rate is the signal to act on: callers are still being served, and the primary target is degrading behind that.

# Targets currently out of rotation
count by (model) (aisix_deployment_state == 2)

# How often targets enter cooldown
sum by (model, upstream_model) (rate(aisix_deployment_cooled_down_total[15m]))

Deployment counters sample once per upstream attempt, so one client request that failed over across three targets is three samples here and one in the request family. Do not divide one family by the other.

Monitor Rate Limits and Diagnose Budget Denials

The rate-limit metrics below apply to every AISIX gateway. AISIX Cloud budget gauges describe budget denials; they are not a continuous feed of current spend.

The share of requests a rate limit rejected comes from the request family's outcome label, which covers every endpoint:

sum(rate(aisix_llm_requests_total{outcome="rate_limited"}[5m]))
  /
sum(rate(aisix_llm_requests_total[5m]))

To tell a request-count limit from a token limit, use the dedicated counter's scope label:

sum by (scope) (rate(aisix_ratelimit_rejections_total[5m]))

When AISIX Cloud rejects a request because a blocking budget was exceeded, the budget gauges record the denial details. They carry the same api_key_id, team_id, user_id, and user_name labels, so you can inspect the reported spend against the limit:

# Reported spend relative to the limit that blocked the request
(aisix_budget_spent_usd / aisix_budget_limit_usd)
  and on (api_key_id, team_id, user_id) (aisix_budget_details_present == 1)

Guard on aisix_budget_details_present == 1. A later decision without budget details does not zero the other gauges — they stay at the amounts from the last denial, so an unguarded ratio can report stale values. The flag is what tells the two apart.

These gauges cannot warn before AISIX Cloud blocks a request because the denial is what supplies their values. Configure Budget Alerts and Notifications for threshold notifications before a blocking budget starts rejecting traffic.

When a Key Goes Away

aisix_budget_* and aisix_ratelimit_remaining_* are written only while a request is being served, and a Prometheus series is never dropped once it exists. AISIX therefore retires them: within seconds of an API key being deleted — or rebound to another team or member, which starts a new series and strands the old one — the stale sample is rewritten as NaN, and aisix_budget_details_present as 0.

NaN rather than 0 is deliberate. aisix_ratelimit_remaining_requests 0 means the caller is out of quota and aisix_budget_remaining_usd 0 means the budget is spent, so zeroing a retired series would replace a stale reading with a false one. Every comparison against NaN is false, which is what makes an alert on a retired series stop firing rather than latch. Two consequences to plan for:

The series still exists, so absent() does not report it missing.
An aggregation over a family that contains a retired series reads NaN — sum(aisix_budget_spent_usd) returns NaN, not the live total. Group by an identifying label, or filter on aisix_budget_details_present == 1 first. The per-series queries above are unaffected.

aisix_deployment_state is not retired. It is written only when a deployment's health changes, so blanking it would leave a live model reporting no state at all until its next transition. A model deleted while it was cooling down therefore stays counted by count by (model) (aisix_deployment_state == 2) until the gateway restarts.

Measure Concurrency and Abandoned Requests

aisix_proxy_in_flight_requests is the live concurrency at the proxy, which is what sizes a gateway rather than the request rate:

sum by (endpoint, inbound_protocol) (aisix_proxy_in_flight_requests)

A caller that disconnects before response headers were sent never reaches a normal outcome and is absent from the request counters, so it has to be added back into the denominator to get an abandonment rate:

sum(rate(aisix_proxy_client_cancelled_requests_total[5m]))
  /
(
  sum(rate(aisix_proxy_requests_total[5m]))
  + sum(rate(aisix_proxy_client_cancelled_requests_total[5m]))
)

Abandonment usually means callers give up while waiting for the first token. Break it down by model and compare against time to first token for that same model:

sum by (model) (rate(aisix_proxy_client_cancelled_requests_total[5m]))

Denied authentication is the other silent failure mode — a rotated key or a misconfigured issuer shows up here long before anyone reports it:

sum by (method, reason) (rate(aisix_auth_decisions_total{result="denied"}[5m]))

Verify Usage Event Delivery

Usage events feed billing and analytics, so a query that silently loses them is worth an alert of its own. Every emission attempt is either accepted by the delivery queue or counted as a drop:

sum(rate(aisix_usage_event_drops_total[5m]))
  /
sum(rate(aisix_usage_events_emitted_total[5m]))

Both counters carry model, provider_key_id, and provider_key_name, so the same division holds per model and per provider key. A drop rate concentrated on one of them points at the traffic whose usage records went missing:

sum by (model, provider_key_name, reason) (rate(aisix_usage_event_drops_total[5m]))

Queue acceptance is not delivery to storage. Control-plane shipping and exporter pipelines can still fail after the worker accepts an event, and neither increments the drop counter — those failures appear in the gateway log.

Alert on Configuration Problems

A gateway that cannot load its configuration keeps serving the last good one, which means the symptom is silence rather than an error. Alert on the state directly:

# The most recent load failed
aisix_config_last_reload_successful == 0

# Nothing has loaded successfully for five minutes
time() - aisix_config_last_reload_success_timestamp_seconds > 300

# Resources the gateway refused, by kind
sum by (kind) (aisix_config_rejected_resources) > 0

# Serving an older etcd revision than the one observed
aisix_config_observed_revision - aisix_config_applied_revision > 0

aisix_config_partially_compatible_resources is the upgrade-order signal: it counts resources carrying fields this gateway version does not recognize, which is what a control plane one release ahead of its data planes produces. It is expected during an upgrade window and should return to zero once the data planes are upgraded:

sum by (kind) (aisix_config_partially_compatible_resources) > 0

Inspect GET /status/config on the metrics listener for the specific resources and field paths behind any of these.

Configure Metrics

Configure custom client classification and histogram bucket boundaries at startup. Changes take effect after a gateway restart and can alter label values or bucket series, so coordinate them with dashboards, alerts, and recording rules.

Map Custom Clients to a Client Type

AISIX recognizes common AI coding clients and SDKs out of the box. To classify in-house tools — or re-bucket a client that the built-in rules place in a generic bucket such as node — define mapping rules in the gateway configuration:

config.yaml
observability:
  metrics:
    client_type_rules:
      - pattern: "^billing-batcher/"
        client: billing-batcher
      - pattern: "internal-eval-harness"
        client: eval-harness

Each rule maps a regular expression to a fixed client value, which AISIX emits as the client_type label. Rules are evaluated in order against the raw User-Agent header, before the built-in rules, and the first match wins. Matching is case-insensitive and unanchored, so anchor the pattern with ^ when you need a prefix match.

The client value — never the request's User-Agent — becomes the label, so the label set stays bounded regardless of what clients send. Configuration limits keep it that way: at most 64 rules, patterns up to 512 bytes, and client values of up to 64 characters matching [a-z0-9][a-z0-9._-]*. AISIX validates the rules at startup and refuses to start on an invalid rule; changes take effect after a restart.

Requests with an empty User-Agent always report unknown, and requests that match no rule fall through to the built-in classification.

Customize Histogram Buckets

Four metrics are true Prometheus histograms with le bucket boundaries, and each one ships its own default boundaries because the distributions differ:

Metric	Default boundaries (seconds)
`aisix_request_e2e_latency_seconds`	0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2, 5, 10, 30, 60, 120, 300, 420, 600
`aisix_request_ttft_seconds`	0.05, 0.1, 0.25, 0.5, 1, 2, 5, 10, 30, 60, 120, 300
`aisix_guardrail_latency_seconds`	0.001, 0.0025, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10, 30
`aisix_a2a_ttfb_seconds`	0.05, 0.1, 0.25, 0.5, 1, 2, 5, 10, 30, 60, 120, 300

End-to-end latency includes cache hits and requests rejected before dispatch, so it needs millisecond boundaries. Time to first token (TTFT) is recorded at the first frame an upstream streams, whatever it carries; against a hosted provider, sub-50 ms boundaries are usually empty. Guardrail metrics need low boundaries for fast in-process checks and high boundaries for remote moderation services. An A2A agent's time to its first streamed event starts from the same defaults as TTFT. It takes its own a2a_ttfb override, because an agent task can think for minutes before it says anything.

Override the boundaries per metric when your traffic has a different distribution. For example, a vLLM or Ollama model server on the same node can produce its first output in milliseconds:

config.yaml
observability:
  metrics:
    buckets:
      request_ttft: [0.005, 0.01, 0.025, 0.05, 0.1, 0.5, 1, 5, 30]

Each field is optional and replaces only the metric it names; omitted metrics keep their defaults. A supplied list must contain 1–64 finite, positive, strictly increasing boundaries. Do not list +Inf — AISIX appends that bucket itself. AISIX validates the configuration at startup and refuses to start on an invalid list; changes take effect after a restart.

Every boundary adds one _bucket time series for each label combination, so a longer list increases the number of series a collector stores.

For a gateway deployed with the AISIX Helm chart, provide the comma-separated list through extraEnvVars:

extraEnvVars:
  - name: AISIX_OBSERVABILITY__METRICS__BUCKETS__REQUEST_TTFT
    value: "0.005,0.01,0.025,0.05,0.1,0.5,1,5,30"

caution

Changing boundaries changes the emitted _bucket series. A dashboard or recording rule that selects a removed le value stops matching. Do not compare bucket-derived quantiles from before and after the change.

See Set Any Other Gateway Configuration for the Helm configuration pattern.

Metrics Catalog

Metric Types

Request Metrics

Labels

Values for outcome

Behavior

PromQL example

Labels

Values for inbound_protocol

Values for upstream_protocol

Values for stream

Values for is_fallback

Values for outcome

Behavior

Labels

Values for inbound_protocol

Values for upstream_protocol

Values for stream

Values for is_fallback

Values for outcome

Behavior

Labels

Values for inbound_protocol

Values for upstream_protocol

Values for stream

Values for is_fallback

Values for outcome

Labels

Values for inbound_protocol

Behavior

PromQL example

Labels

Behavior

PromQL example

Labels

Values for inbound_protocol

Values for outcome

Behavior

PromQL example

Labels

Values for method

Values for result

Behavior

PromQL example

Latency Metrics

Labels

Behavior

Labels

Values for inbound_protocol

Values for upstream_protocol

Values for stream

Values for outcome

Labels

Values for inbound_protocol

Values for upstream_protocol

Values for stream

Values for outcome

Labels

Values for inbound_protocol

Values for upstream_protocol

Labels

Values for status_class

Values for streaming

Behavior

PromQL example

Labels

Values for status_class

Values for streaming

Behavior

PromQL example

Usage and Cost Metrics

Labels

Labels

Values for inbound_protocol

Values for upstream_protocol

Labels

Values for inbound_protocol

Values for upstream_protocol

Labels

Values for inbound_protocol

Values for `outcome`

Values for `inbound_protocol`

Values for `upstream_protocol`

Values for `stream`

Values for `is_fallback`

Values for `outcome`

Values for `inbound_protocol`

Values for `upstream_protocol`

Values for `stream`

Values for `is_fallback`

Values for `outcome`

Values for `inbound_protocol`

Values for `upstream_protocol`

Values for `stream`

Values for `is_fallback`

Values for `outcome`

Values for `inbound_protocol`

Values for `inbound_protocol`

Values for `outcome`

Values for `method`

Values for `result`

Values for `inbound_protocol`

Values for `upstream_protocol`

Values for `stream`

Values for `outcome`

Values for `inbound_protocol`

Values for `upstream_protocol`

Values for `stream`

Values for `outcome`

Values for `inbound_protocol`

Values for `upstream_protocol`

Values for `status_class`

Values for `streaming`

Values for `status_class`

Values for `streaming`

Values for `inbound_protocol`

Values for `upstream_protocol`

Values for `inbound_protocol`

Values for `upstream_protocol`

Values for `inbound_protocol`

Values for `upstream_protocol`

Values for `inbound_protocol`

Values for `upstream_protocol`

Values for `inbound_protocol`

Values for `upstream_protocol`

Values for `inbound_protocol`

Values for `upstream_protocol`

Values for `inbound_protocol`

Values for `upstream_protocol`

Values for `client_type`

Values for `token_type`

Values for `scope`

Values for `layer`

Values for `metric value`

Values for `outcome`

Values for `cause`

Values for `op`

Values for `metric value`

Values for `kind`

Values for `phase`

Values for `result`

Values for `error_type`