Metrics Reference

AISIX exposes operational metrics in Prometheus text format. A Prometheus server or another compatible collector can scrape these metrics for dashboards, alerts, and PromQL queries.

Prometheus metrics are enabled by default on a dedicated listener. The default startup configuration is:

config.yaml
observability:
  metrics:
    prometheus:
      enabled: true
      addr: 0.0.0.0:9090
      path: /metrics

Prometheus scrapes GET /metrics from this listener.

The metrics endpoint is unauthenticated by design. Keep its listener private to your monitoring network.

AISIX publishes configuration status whenever the endpoint is scraped. Other metric families are registered when AISIX first records the corresponding activity, so traffic metrics might not appear immediately after startup. Send a request through the proxy, then scrape again for the corresponding series to appear.

Metrics Catalog

Search metric names, descriptions, labels, and values, or filter the catalog by family and type. Expand an entry to review its details.

Metrics: 59
Families: 10

Query behavior

Metric Types

counter

A cumulative value that increases until the process restarts, such as a request or token total. Use rate() to calculate how quickly it changes.

gauge

A current value that can increase or decrease, such as active requests or remaining quota.

histogram

Observations counted in configurable buckets. The _bucket, _sum, and _count series can be aggregated before calculating a percentile.

summary

Observations with quantiles calculated by each gateway instance. Summaries also expose _sum and _count, but their quantiles cannot be aggregated across instances.

Search metrics

Showing 59 of 59 entries

Request Metrics

Track request outcomes and the work currently in progress at the proxy.

Metric entries: 8

Detailed Request Labels

Every counter in this family samples once per client request, and its status is the status the caller received. A request whose first target failed and whose fallback then succeeded is a single status="200" sample here; the failure it recovered from is not represented in this family at all. Count upstream attempts with the [deployment metrics](#deployment-metrics) and read individual attempts in the usage log.

For the three detailed request counters, stream records whether the client requested streaming. is_fallback records whether a fallback target served the request and does not appear on latency metrics.

provider_key_name and user_name are human-readable companions to their IDs. Each name has a one-to-one relationship with its ID, so it does not add another series dimension. user_name is unknown until the control plane supplies it.

inbound_protocol is a bounded protocol family. It is derived from the endpoint: the Anthropic-protocol routes report anthropic, /mcp, /a2a, and /v1/realtime report mcp, a2a, and realtime, and everything else reports openai. The in-flight gauge uses the same values.

endpoint is always a normalized route template, never a raw request path. Routes with a path parameter collapse to one series — /v1/batches/:id, /v1/videos/:id, /mcp/{server}, /passthrough/:provider/*rest — and an unrecognized path reports other.

aisix_requests_total . Description: Proxy request outcomes in the compatibility series with the broadest endpoint coverage. . Type: counter . Label count: 4 labels

Labels

provider, model, status, outcome

Values for `outcome`

success, client_error, rate_limited, upstream_error

success covers HTTP 200–399, client_error covers HTTP 400–499 except 429, rate_limited is HTTP 429, and all other statuses map to upstream_error.

Behavior

A2A agent calls use provider="a2a" and model="a2a".

PromQL example

sum(rate(aisix_requests_total[5m])) by (outcome)

aisix_llm_requests_total . Description: Model-inference request outcomes, including successful and failed requests. . Type: counter . Label count: 15 labels

Labels

endpoint, inbound_protocol, provider, model, upstream_model, provider_key_id, provider_key_name, api_key_id, team_id, user_id, user_name, stream, is_fallback, status, outcome

Values for `inbound_protocol`

openai, anthropic, realtime

Values for `stream`

false, true

Values for `is_fallback`

false, true

Values for `outcome`

success, client_error, rate_limited, upstream_error

success covers HTTP 200–399, client_error covers HTTP 400–499 except 429, rate_limited is HTTP 429, and all other statuses map to upstream_error.

Behavior

Covers the endpoints that call a model: chat-completions, completions, messages, count-tokens, responses, embeddings, rerank, audio, image generation, videos, and realtime sessions. Requests that reach no model are counted in aisix_proxy_requests_total only — MCP tool calls, A2A agent calls, the provider passthrough, and the file, batch, and fine-tuning management routes.

A request refused before dispatch, such as an oversized body, is counted against the endpoint it targeted, so a success rate over an endpoint includes those failures in its denominator.

aisix_proxy_requests_total . Description: Detailed request outcomes for all proxied traffic, model-inference and otherwise. . Type: counter . Label count: 15 labels

Labels

Values for `inbound_protocol`

openai, anthropic, mcp, a2a, realtime

Values for `stream`

false, true

Values for `is_fallback`

false, true

Values for `outcome`

success, client_error, rate_limited, upstream_error

success covers HTTP 200–399, client_error covers HTTP 400–499 except 429, rate_limited is HTTP 429, and all other statuses map to upstream_error.

Behavior

One sample per client request, carrying the status the caller received. Retries and failovers within a request do not add samples, so this counter answers "how many requests did callers make, and how did they end" — not "how many calls did the gateway place upstream".

aisix_proxy_failed_requests_total . Description: The subset of proxy requests whose outcome is not success. . Type: counter . Label count: 15 labels

Labels

Values for `inbound_protocol`

openai, anthropic, mcp, a2a, realtime

Values for `stream`

false, true

Values for `is_fallback`

false, true

Values for `outcome`

client_error, rate_limited, upstream_error

client_error covers HTTP 400–499 except 429, rate_limited is HTTP 429, and all other statuses map to upstream_error.

aisix_proxy_in_flight_requests . Description: Requests currently being handled by the proxy, grouped by normalized endpoint and inbound protocol. . Type: gauge . Label count: 2 labels

Labels

endpoint, inbound_protocol

Values for `inbound_protocol`

openai, anthropic, mcp, a2a, realtime

Behavior

MCP requests use inbound_protocol="mcp", with endpoint="/mcp" for the aggregated gateway and endpoint="/mcp/{server}" for the per-server endpoint. A2A calls use endpoint="/a2a" and inbound_protocol="a2a".

PromQL example

sum(aisix_proxy_in_flight_requests) by (endpoint, inbound_protocol)

aisix_proxy_client_cancelled_requests_total . Description: Requests whose caller disconnected before the gateway sent response headers. . Type: counter . Label count: 1 label

Labels

endpoint

Behavior

These requests never reach a normal outcome, so they do not appear in the other request counters. The gateway records them here and in the access log with status 499.

A rising rate usually means callers give up while waiting for the first token. Compare it against aisix_llm_time_to_first_token_seconds for the same models.

A caller that disconnects after response headers were sent is not counted here. That request already has a normal outcome and a usage event.

PromQL example

sum(rate(aisix_proxy_client_cancelled_requests_total[5m])) by (endpoint)

aisix_proxy_request_body_limit_rejections_total . Description: Requests refused for exceeding proxy.request_body_limit_bytes, grouped by how the gateway finished reading the refused body. . Type: counter . Label count: 3 labels

Labels

endpoint, inbound_protocol, outcome

Values for `inbound_protocol`

openai, anthropic, mcp, a2a, realtime

Values for `outcome`

completed, cap_reached, timeout, client_read_error

completed means the caller sent the whole body it declared, so it could read the 413. cap_reached and timeout mean the gateway stopped absorbing the body first, and client_read_error means the caller went away mid-body — in all three the caller usually sees a closed connection instead of the response.

Behavior

The gateway reads and discards the body of a refused request so the caller can receive the 413 on the same connection. That read is bounded, and outcome reports how it ended.

A rising share of any outcome other than completed means callers are seeing closed connections instead of 413 responses. The matching aisix::body_limit log entry carries the declared size, the configured limit, and the bytes read for the same request_id.

Only requests that declare a Content-Length over the limit are counted here. A chunked body over the limit is rejected while it is being read, has no comparable outcome, and appears in aisix_requests_total with status 413.

PromQL example

sum(rate(aisix_proxy_request_body_limit_rejections_total[5m])) by (endpoint, inbound_protocol, outcome)

aisix_auth_decisions_total . Description: Caller authentication decisions across API key, JWT, and missing-credential paths. . Type: counter . Label count: 3 labels

Labels

method, result, reason

Values for `method`

api_key, jwt, none

Values for `result`

allowed, denied

Behavior

reason is none for an allowed request. Denied requests use a bounded reason such as missing_credentials, unknown_key, key_expired, jwt_bad_signature, jwt_untrusted_issuer, or jwt_identity_unmapped.

PromQL example

sum(rate(aisix_auth_decisions_total{result="denied"}[5m])) by (method, reason)

Latency Metrics

Inspect latency on one gateway or aggregate histogram buckets across gateway instances.

Metric entries: 6

Latency Aggregation and Labels

Use histograms for service-level dashboards and alerts across gateway instances because their _bucket, _sum, and _count series can be aggregated before using histogram_quantile(). Use summaries to inspect precomputed quantiles from one gateway instance. Summary quantiles cannot be aggregated across instances, so do not average them.

Each histogram has its own bucket boundaries, because the two distributions differ: end-to-end latency starts in the milliseconds, while time to first token cannot be faster than the upstream that produces the token. Both sets are configurable. env_id identifies the environment served by an AISIX gateway connected to AISIX Cloud and is unknown when the AISIX gateway is not connected to AISIX Cloud. status_class is one of 2xx, 3xx, 4xx, 5xx, or other. Per-key and per-user labels are excluded to keep the number of bucket series manageable; use usage analytics for those dimensions.

aisix_request_duration_seconds . Description: Request duration across proxy endpoints in the compatibility series. HTTP streams record time until the response starts; realtime records the full WebSocket session until it closes. . Type: summary . Label count: 3 labels

Labels

provider, model, status

Behavior

Summary quantiles are calculated by each AISIX instance and cannot be aggregated across instances.

aisix_llm_request_duration_seconds . Description: Detailed request duration for model-inference endpoints. HTTP streams record time until the response starts; realtime records the full WebSocket session until it closes. . Type: summary . Label count: 14 labels

Labels

endpoint, inbound_protocol, provider, model, upstream_model, provider_key_id, provider_key_name, api_key_id, team_id, user_id, user_name, stream, status, outcome

Values for `inbound_protocol`

openai, anthropic, realtime

Values for `stream`

false, true

Values for `outcome`

success, client_error, rate_limited, upstream_error

success covers HTTP 200–399, client_error covers HTTP 400–499 except 429, rate_limited is HTTP 429, and all other statuses map to upstream_error.

aisix_proxy_request_duration_seconds . Description: Detailed request duration for all proxied traffic. HTTP streams record time until the response starts; realtime records the full WebSocket session until it closes. . Type: summary . Label count: 14 labels

Labels

endpoint, inbound_protocol, provider, model, upstream_model, provider_key_id, provider_key_name, api_key_id, team_id, user_id, user_name, stream, status, outcome

Values for `inbound_protocol`

openai, anthropic, mcp, a2a, realtime

Values for `stream`

false, true

Values for `outcome`

success, client_error, rate_limited, upstream_error

success covers HTTP 200–399, client_error covers HTTP 400–499 except 429, rate_limited is HTTP 429, and all other statuses map to upstream_error.

aisix_llm_time_to_first_token_seconds . Description: Time from request entry to the first non-empty generated output for streaming chat-completions and messages requests. . Type: summary . Label count: 11 labels

Labels

endpoint, inbound_protocol, provider, model, upstream_model, provider_key_id, provider_key_name, api_key_id, team_id, user_id, user_name

Values for `inbound_protocol`

openai, anthropic

aisix_request_e2e_latency_seconds . Description: Client-perceived latency for chat completions, messages, and responses, including the full duration of streams. . Type: histogram . Label count: 6 labels

Labels

env_id, endpoint, model, provider, status_class, streaming

Values for `status_class`

2xx, 3xx, 4xx, 5xx, other

Values for `streaming`

false, true

Behavior

Default buckets range from 5 milliseconds to 600 seconds, and are configurable with observability.metrics.buckets.request_e2e_latency. The low boundaries record fast responses such as cache hits and requests rejected before dispatch. Aggregate _bucket series before calculating percentiles across gateway instances.

Each request is observed once. Non-streaming requests and failures are recorded when the handler returns. Streaming requests are recorded when the stream finishes, including client cancellation; a canceled stream retains its committed status and records the duration up to cancellation.

PromQL example

histogram_quantile(
  0.90,
  sum by (le) (rate(aisix_request_e2e_latency_seconds_bucket[5m]))
)

aisix_request_ttft_seconds . Description: Time to first non-empty generated output for streaming chat-completions and messages requests. . Type: histogram . Label count: 6 labels

Labels

env_id, endpoint, model, provider, status_class, streaming

Values for `status_class`

2xx, 3xx, 4xx, 5xx, other

Values for `streaming`

true

Behavior

Generated output includes non-empty content, non-empty reasoning content, or a tool-call delta. An empty role-only stream opener does not stop the timer.

Default buckets range from 50 milliseconds to 300 seconds, and are configurable with observability.metrics.buckets.request_ttft. Lower the first boundaries when a nearby model server can produce output in under 50 milliseconds. Deployments that use only hosted providers can raise the floor to remove consistently empty buckets.

PromQL example

histogram_quantile(
  0.90,
  sum by (le) (rate(aisix_request_ttft_seconds_bucket[5m]))
)

Usage and Cost Metrics

Measure token volume, estimated spend, and normalized client usage.

Metric entries: 6

Which Endpoints Report Tokens

Every endpoint that receives token counts from the upstream records them here: chat-completions, completions, messages, responses, embeddings, rerank, the audio transcription routes, image generation, and realtime sessions.

Two model-inference endpoints report no tokens because they are not billed that way — /v1/audio/speech is billed per input character and /v1/videos per video. Both still count as requests, so divide token totals by requests only within one endpoint, never across all of them.

The per-request token and spend series — the three aisix_llm_*_tokens_total counters and aisix_llm_spend_micro_usd_total — carry the same labels as the detailed request counters, so a query can join token volume to request outcomes on endpoint, model, provider, and the caller identity labels. The other two do not: aisix_tokens_consumed_total is labeled by provider and model only, and aisix_llm_tokens_by_client_total by client_type, model, and token_type.

aisix_tokens_consumed_total . Description: The sum of total tokens across every endpoint that reports token usage, in the compatibility series with the broadest coverage. . Type: counter . Label count: 2 labels

Labels

provider, model

aisix_llm_input_tokens_total . Description: Input tokens reported by the upstream, across every endpoint that reports token usage. . Type: counter . Label count: 11 labels

Labels

endpoint, inbound_protocol, provider, model, upstream_model, provider_key_id, provider_key_name, api_key_id, team_id, user_id, user_name

Values for `inbound_protocol`

openai, anthropic, realtime

aisix_llm_output_tokens_total . Description: Output tokens reported by the upstream, across every endpoint that reports token usage. . Type: counter . Label count: 11 labels

Labels

endpoint, inbound_protocol, provider, model, upstream_model, provider_key_id, provider_key_name, api_key_id, team_id, user_id, user_name

Values for `inbound_protocol`

openai, anthropic, realtime

aisix_llm_total_tokens_total . Description: Total tokens reported by the upstream, across every endpoint that reports token usage. . Type: counter . Label count: 11 labels

Labels

endpoint, inbound_protocol, provider, model, upstream_model, provider_key_id, provider_key_name, api_key_id, team_id, user_id, user_name

Values for `inbound_protocol`

openai, anthropic, realtime

aisix_llm_spend_micro_usd_total . Description: Estimated spend in micro-USD, where 1 USD equals 1,000,000 micro-USD. Recorded wherever the gateway resolves a price for the request. . Type: counter . Label count: 11 labels

Labels

endpoint, inbound_protocol, provider, model, upstream_model, provider_key_id, provider_key_name, api_key_id, team_id, user_id, user_name

Values for `inbound_protocol`

openai, anthropic, realtime

aisix_llm_tokens_by_client_total . Description: Token volume across every endpoint that reports token usage, grouped by normalized client, requested model, and token type. . Type: counter . Label count: 3 labels

Labels

client_type, model, token_type

Values for `client_type`

openai-python, openai-node, anthropic-python, anthropic-typescript, claude-code, codex, cline, roo-code, kilocode, zoo-code, github-copilot, cursor, opencode, qwen-code, gemini-cli, crush, zed, aider, vercel-ai-sdk, langchain, llamaindex, litellm, curl, python-requests, httpx, aiohttp, okhttp, go-http-client, node, postman, browser, other, unknown

An unrecognized User-Agent maps to other; a missing User-Agent maps to unknown. Deployments can extend this set with admin-defined mapping rules (observability.metrics.client_type_rules). Full user-agent strings and versions remain in request logs and usage analytics.

Values for `token_type`

input, output, total

total includes input, output, and Anthropic cache-creation and cache-read tokens.

Behavior

model is the caller-requested model name, matching the model label on the aisix_llm_* token series. Routing, semantic, ensemble, and fallback dispatch retain this alias instead of reporting the selected direct model.

The bounded client_type allowlist prevents a client-controlled user-agent from creating unbounded Prometheus cardinality. Concrete model aliases are bounded by the configured model set, but a wildcard alias records each concrete model name requested through that pattern. Restrict wildcard access and monitor label growth when cardinality matters.

Admin-defined mapping rules (observability.metrics.client_type_rules) can classify additional clients. Rules are matched before the built-in allowlist and emit a fixed, validated label value, so the label set stays bounded.

Aggregated across all labels, the cache-inclusive total aligns with aisix_llm_total_tokens_total for the same endpoint population. Individual series do not align because the two metric families use different label sets; the dedicated client-type series avoids multiplying the per-key token series by another label dimension.

Anthropic reports cache tokens separately from input tokens, so total can exceed input plus output.

PromQL example

sum by (client_type, model, token_type) (
  rate(aisix_llm_tokens_by_client_total[5m])
)

Rate Limit and Budget Metrics

Monitor rate-limit rejections and the latest quota or budget state for each label set.

Metric entries: 8

aisix_ratelimit_rejections_total . Description: Chat-completions requests rejected by a rate limit. . Type: counter . Label count: 1 label

Labels

scope

Values for `scope`

requests, tokens

aisix_ratelimit_remaining_requests . Description: Remaining request quota reported while processing a chat-completions request, grouped by API key and model. . Type: gauge . Label count: 2 labels

Labels

api_key_id, model

aisix_ratelimit_remaining_tokens . Description: Remaining token quota reported while processing a chat-completions request, grouped by API key and model. . Type: gauge . Label count: 2 labels

Labels

api_key_id, model

aisix_budget_limit_usd . Description: Budget limit in USD. . Type: gauge . Label count: 3 labels

Labels

api_key_id, team_id, user_id

aisix_budget_spent_usd . Description: Budget spent in USD. . Type: gauge . Label count: 3 labels

Labels

api_key_id, team_id, user_id

aisix_budget_remaining_usd . Description: Budget remaining in USD. . Type: gauge . Label count: 3 labels

Labels

api_key_id, team_id, user_id

aisix_budget_reset_seconds . Description: Seconds until the budget period resets. . Type: gauge . Label count: 3 labels

Labels

api_key_id, team_id, user_id

aisix_budget_details_present . Description: Whether budget details are populated. . Type: gauge . Label count: 3 labels

Labels

api_key_id, team_id, user_id

Values for `metric value`

0, 1

1 means budget details are present; 0 means they have been cleared.

Cache Metrics

Measure response-cache effectiveness per policy and watch the semantic layer’s embedding and store health.

Metric entries: 4

Cache Hit Rate

Every request covered by an enabled cache policy with an available backend records one aisix_cache_requests_total observation. Requests with no matching policy or an unavailable backend are not counted — the gate never opened — so the series measures policy effectiveness, not total traffic.

Hit rate per policy: sum by (policy) (rate(aisix_cache_requests_total{outcome=~"hit_exact|hit_semantic"}[5m])) / sum by (policy) (rate(aisix_cache_requests_total[5m])). Split by outcome to see how much the semantic layer adds over exact matching.

Semantic-layer failures degrade to ordinary misses, which makes a broken embedding model or store indistinguishable from a healthy low hit rate in the outcome counter alone. Alert on aisix_cache_semantic_embedding_failures_total and aisix_cache_semantic_store_failures_total to separate the two.

aisix_cache_requests_total . Description: Cache-eligible requests by policy name and outcome, counted once per request when a matching enabled policy with an available backend opened the cache gate. . Type: counter . Label count: 2 labels

Labels

policy, outcome

Values for `outcome`

hit_exact, hit_semantic, miss, bypass

bypass means the caller sent Cache-Control: no-cache, which skips the read path. no-store requests also count as bypass.

aisix_cache_semantic_embedding_seconds . Description: Latency of the semantic layer’s embedding calls, per policy. Summary series without fixed buckets. . Type: summary . Label count: 1 label

Labels

policy

aisix_cache_semantic_embedding_failures_total . Description: Embedding failures on the semantic layer. Failed requests proceed to the upstream uncached. . Type: counter . Label count: 2 labels

Labels

policy, cause

Values for `cause`

resolve, embed

resolve (embedding model missing or not an embedding model) is counted once per eligible request, including requests that then hit the exact layer; embed (provider call failed or timed out) is counted per embedding call.

aisix_cache_semantic_store_failures_total . Description: Semantic-store operation failures by operation. The in-process store cannot fail; shared (Redis) stores can. Failures degrade to an ordinary miss. . Type: counter . Label count: 2 labels

Labels

policy, op

Values for `op`

lookup, store

Deployment Metrics

Monitor how each target model behaves: upstream attempts and their outcomes, fallbacks between targets, and whether a target remains in rotation.

Metric entries: 7

Attempts, Not Requests

These counters sample once per upstream **attempt**, and each sample is filed under the target that was attempted rather than the Model Group the caller named. One client request that failed over across three targets is three samples here and one sample in the [request metrics](#request-metrics). Use this family to ask which target is failing, and the request family to ask what callers experienced.

Only attempts that reached the upstream are counted. An attempt the gateway refused on its own — the target was over its own rate limit, or its credentials or endpoint failed validation before anything was sent — produced no upstream response, so it is absent here while still appearing in the usage log. That keeps a misconfiguration from reading as an unhealthy target.

Emitted by the endpoints that dispatch through a Model Group: /v1/chat/completions, /v1/messages, and /v1/responses. The other endpoints call one model per request and are fully described by the request metrics.

aisix_deployment_requests_total . Description: Upstream attempts dispatched to a target model, whatever their outcome. . Type: counter . Label count: 4 labels

Labels

provider, model, upstream_model, provider_key_id

PromQL example

sum(rate(aisix_deployment_requests_total[5m])) by (model)

aisix_deployment_success_responses_total . Description: Upstream attempts a target model answered successfully. . Type: counter . Label count: 4 labels

Labels

provider, model, upstream_model, provider_key_id

Behavior

For a streamed response, an attempt counts as successful once the upstream stream is established. A stream that breaks after that point is not reclassified here.

aisix_deployment_failure_responses_total . Description: Upstream attempts that failed at the target model, including the failures a fallback went on to rescue. . Type: counter . Label count: 4 labels

Labels

provider, model, upstream_model, provider_key_id

PromQL example

sum(rate(aisix_deployment_failure_responses_total[5m])) by (model)
  / sum(rate(aisix_deployment_requests_total[5m])) by (model)

aisix_routing_successful_fallbacks_total . Description: Fallback attempts that served the request after an earlier target failed. . Type: counter . Label count: 2 labels

Labels

model, fallback_model

Behavior

model is what the caller asked for — the Model Group name. fallback_model is the target the gateway moved to.

aisix_routing_failed_fallbacks_total . Description: Fallback attempts that failed in turn. . Type: counter . Label count: 2 labels

Labels

model, fallback_model

Behavior

A request rescued by its second fallback contributes one sample here and one to the successful family.

aisix_deployment_state . Description: Whether a target model is in rotation. . Type: gauge . Label count: 4 labels

Labels

provider, model, upstream_model, provider_key_id

Values for `metric value`

0, 2

0 is healthy. 2 is out of rotation because the target is cooling down or failing a background health check.

aisix_deployment_cooled_down_total . Description: Number of times a target model entered cooldown. . Type: counter . Label count: 4 labels

Labels

provider, model, upstream_model, provider_key_id

Guardrail Metrics

Track per-guardrail execution latency and outcomes across all endpoints, plus aggregate blocks and fail-open bypasses for chat-completions requests.

Metric entries: 3

Guardrail Execution Latency

Every timed guardrail execution records an aisix_guardrail_latency_seconds observation. A guardrail normally runs once per applicable phase, while streaming window scans can record several output executions for one response. The histogram uses configurable buckets that default to 1 ms through 30 s, so histogram_quantile yields P50/P95/P99 per guardrail.

The kind label separates local, in-process detection (keyword, pii) from remote moderation services (every other kind), and the result label separates fail-open bypasses (bypassed, with the failure tag in error_type) from policy decisions. A fail-closed remote failure surfaces as blocked; its latency clusters at the configured provider timeout.

The _count series doubles as a per-guardrail execution counter: sum by (guardrail, result) (rate(aisix_guardrail_latency_seconds_count[5m])) gives execution and block rates without a separate counter.

aisix_guardrail_latency_seconds . Description: Wall-clock duration of one timed guardrail execution across all endpoints. guardrail is the configured guardrail name. . Type: histogram . Label count: 6 labels

Labels

env_id, guardrail, kind, phase, result, error_type

Values for `kind`

keyword, pii, aliyun_ai_guardrail, aliyun_text_moderation, azure_content_safety, azure_content_safety_text_moderation, bedrock, lakera, openai_moderation, presidio

keyword and pii run in-process (local detection); every other kind calls a remote moderation service.

Values for `phase`

input, output

Values for `result`

allowed, blocked, masked, bypassed, would_block, would_mask

bypassed is a remote failure that failed open; a fail-closed remote failure records as blocked. would_block / would_mask come from enforcement_mode: monitor guardrails.

Values for `error_type`

none, aliyun_5xx, aliyun_config_error, aliyun_throttled, aliyun_timeout, azure_cs_5xx, azure_cs_config_error, azure_cs_throttled, azure_cs_timeout, bedrock_5xx, bedrock_throttled, bedrock_timeout, lakera_5xx, lakera_config_error, lakera_throttled, lakera_timeout, openai_moderation_5xx, openai_moderation_config_error, openai_moderation_throttled, openai_moderation_timeout, presidio_5xx, presidio_config_error, presidio_throttled, presidio_timeout

Set to the bounded failure tag when result="bypassed" (the same values as aisix_guardrail_bypasses_total{reason}), otherwise none.

Behavior

Default buckets: 0.001, 0.0025, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10, and 30 seconds, configurable with observability.metrics.buckets.guardrail_latency.

Monitor-mode executions record would_block / would_mask while the request proceeds, so a staged policy can be sized before enforcing it.

PII and keyword masking performed by synchronous per-field operations is not timed by this series; its in-process cost is measured in microseconds.

aisix_guardrail_blocks_total . Description: Requests rejected by an input or output guardrail, including policy blocks and fail-closed results. Chat-completions requests only; for per-guardrail block rates across all endpoints, use the _count series of aisix_guardrail_latency_seconds with result="blocked". . Type: counter . Label count: no labels

Labels

None.

aisix_guardrail_bypasses_total . Description: Fail-open events where a remote guardrail was unreachable and fail_open allowed the request to continue. . Type: counter . Label count: 1 label

Labels

reason

Values for `reason`

aliyun_5xx, aliyun_config_error, aliyun_throttled, aliyun_timeout, azure_cs_5xx, azure_cs_config_error, azure_cs_throttled, azure_cs_timeout, bedrock_5xx, bedrock_throttled, bedrock_timeout, lakera_5xx, lakera_config_error, lakera_throttled, lakera_timeout, openai_moderation_5xx, openai_moderation_config_error, openai_moderation_throttled, openai_moderation_timeout, presidio_5xx, presidio_config_error, presidio_throttled, presidio_timeout

A2A Metrics

Measure Agent-to-Agent traffic by the agent that was reached and the operation that was invoked.

Metric entries: 4

Agent and Operation Labels

agent is a registered A2A agent, and operation is the canonical operation a caller invoked. A2A 0.3 and 1.0 spell one operation two ways, so AISIX records the canonical form and a deployment fronting both versions still aggregates as one series. An unrecognized method becomes unknown.

Task ids, context ids, and JSON-RPC request ids are deliberately absent. They are what makes an individual call traceable, which is exactly what makes them unusable as label values; find them in usage events and traces instead.

aisix_a2a_requests_total does not agree with aisix_proxy_requests_total{endpoint="/a2a"}, by design. A call refused before its agent is resolved has no agent to file under and is counted only in the proxy family, and a stream the caller abandons is a 4xx here but a 2xx there, because the response really did begin as a 200. Read this family for agent health and the proxy family for route traffic. Request duration is not repeated here: aisix_proxy_request_duration_seconds already times the route.

aisix_a2a_requests_total . Description: A2A calls by the agent reached, the canonical operation invoked, and the status class. . Type: counter . Label count: 3 labels

Labels

agent, operation, status

Behavior

Counted at the same point the usage event is emitted, so a call that is accounted for is also metered.

PromQL example

sum by (agent, operation) (
  rate(aisix_a2a_requests_total{status!="2xx"}[5m])
)

aisix_a2a_ttfb_seconds . Description: Time to the upstream agent's first streamed event on a streaming A2A call. . Type: histogram . Label count: 2 labels

Labels

agent, operation

Behavior

Named for the event rather than a token because an agent stream carries task updates, not tokens. Default buckets match aisix_request_ttft_seconds, and are configurable with observability.metrics.buckets.a2a_ttfb.

Observed only on streaming operations that produced at least one event.

PromQL example

histogram_quantile(
  0.90,
  sum by (le, agent) (rate(aisix_a2a_ttfb_seconds_bucket[5m]))
)

aisix_a2a_stream_events_total . Description: Events relayed downstream on streaming A2A calls. . Type: counter . Label count: 2 labels

Labels

agent, operation

Behavior

Divided by aisix_a2a_requests_total over the streaming operations, it gives events per call — how chatty an agent is, and whether that changed. Both message/stream and tasks/resubscribe stream, so filter the denominator to both or the ratio is inflated.

PromQL example

sum by (agent) (rate(aisix_a2a_stream_events_total[5m]))
/
sum by (agent) (
  rate(aisix_a2a_requests_total{operation=~"message/stream|tasks/resubscribe"}[5m])
)

aisix_a2a_task_state_total . Description: A2A calls by the task state the agent last reported, normalized to the specification's set plus unknown. . Type: counter . Label count: 2 labels

Labels

agent, state

Behavior

A cumulative counter of the state each call ENDED on, so read it as a rate rather than as a live backlog: a task that later moves on does not decrement an earlier sample, and a client polling one task with tasks/get reports its state on every poll. A call whose upstream never answered reports no state and is not counted here.

PromQL example

sum by (agent, state) (rate(aisix_a2a_task_state_total[5m]))

Usage Event Metrics

Distinguish usage-event emission attempts from events the delivery queue did not accept.

Metric entries: 2

Usage Event Delivery

For each emission attempt, the event is either accepted by the queue or counted as a drop. After summing across labels, subtract the drop rate from the emission-attempt rate to calculate the accepted rate.

The status_code label is one of 2xx, 3xx, 4xx, 5xx, or other. Example handler values include chat, embeddings, messages, and mcp.

MCP tool calls use handler="mcp" and inbound_protocol="mcp". Their usage-event payloads identify the MCP server and tool; token and cost fields are zero.

A2A agent calls use handler="a2a". The metric maps their bounded inbound_protocol label to other, while the delivered event uses inbound_protocol="a2a" and identifies the agent name and JSON-RPC method. Token and cost fields are zero.

aisix_usage_events_emitted_total . Description: Usage-event emission attempts, counted before AISIX tries to enqueue the event, including both delivered and dropped events. . Type: counter . Label count: 3 labels

Labels

handler, status_code, inbound_protocol

Values for `handler`

a2a, audio, batch, batches, chat, completions, embeddings, files, fine_tuning, images, mcp, messages, passthrough, realtime, rerank, responses, videos

Values for `status_code`

2xx, 3xx, 4xx, 5xx, other

Values for `inbound_protocol`

openai, anthropic, mcp, other

other includes A2A, realtime, passthrough, and every other value outside the three named protocol buckets.

aisix_usage_event_drops_total . Description: Usage events that were not accepted by the queue. . Type: counter . Label count: 1 label

Labels

reason

Values for `reason`

sink_disabled, sink_full, sink_closed

Configuration Metrics

Monitor whether AISIX can load and apply changes from its configuration source.

Metric entries: 11

Configuration State

AISIX reflects the same live configuration state in these metrics and at GET /status/config on the metrics and status listener.

Reload metrics apply to both file and etcd configuration sources. Revision and source-connection metrics are emitted only when AISIX loads configuration from etcd.

Compare the observed and applied revisions to determine whether the gateway is serving the latest etcd configuration.

aisix_config_last_reload_successful . Description: Whether the latest configuration load succeeded. . Type: gauge . Label count: no labels

Labels

None.

Values for `metric value`

0, 1

1 indicates success and 0 indicates failure.

aisix_config_last_reload_success_timestamp_seconds . Description: Unix timestamp in seconds of the last successful configuration load. . Type: gauge . Label count: no labels

Labels

None.

Behavior

The series is not emitted until a configuration load succeeds.

aisix_config_reloads_total . Description: Full configuration reload attempts, including source fetch failures. . Type: counter . Label count: no labels

Labels

None.

Behavior

Incremental etcd watch events do not increment this counter.

aisix_config_reload_failures_total . Description: Configuration reload failures grouped by reason. . Type: counter . Label count: 1 label

Labels

reason

Values for `reason`

fetch, parse, validate

fetch indicates that the source could not be read, parse indicates invalid source data, and validate indicates an invalid resource.

aisix_config_rejected_resources . Description: Current number of rejected resources, grouped by resource kind. . Type: gauge . Label count: 1 label

Labels

kind

Behavior

When all rejections for a resource kind clear, AISIX sets its existing series to 0.

aisix_config_partially_compatible_resources . Description: Served resources carrying at least one field this gateway version does not recognize, grouped by resource kind. . Type: gauge . Label count: 1 label

Labels

kind

Behavior

A resource with several ignored fields counts once. Inspect partially_compatible at GET /status/config for the field paths and per-field counts.

When all partially compatible resources for a kind clear, AISIX sets its existing series to 0.

aisix_config_stale_served_resources . Description: Resources whose latest source value was rejected while their last known good value remains in service, grouped by resource kind. . Type: gauge . Label count: 1 label

Labels

kind

Behavior

Inspect rejected at GET /status/config to identify each resource and when stale serving began.

When no stale value remains in service for a kind, AISIX sets its existing series to 0.

aisix_config_observed_revision . Description: Latest etcd revision observed by the gateway. . Type: gauge . Label count: no labels

Labels

None.

aisix_config_applied_revision . Description: The etcd revision represented by the configuration currently served by the gateway. . Type: gauge . Label count: no labels

Labels

None.

aisix_config_hash_info . Description: Applied configuration hash. . Type: gauge . Label count: 1 label

Labels

hash

Values for `metric value`

0, 1

Filter for value 1 to select the current hash. AISIX retains earlier hash labels with value 0 after the applied configuration changes.

aisix_config_source_connected . Description: Whether the gateway is connected to its etcd configuration source. . Type: gauge . Label count: no labels

Labels

None.

Values for `metric value`

0, 1

1 indicates connected and 0 indicates disconnected.

Analyze Metrics with PromQL

Use these PromQL examples in the Prometheus expression browser or another compatible monitoring interface after configuring it to scrape the AISIX metrics endpoint. Adjust the time window, label filters, and grouping dimensions to match the traffic and gateway instances you want to inspect.

Calculate Success Rate

Calculate the success rate by dividing the successful request rate by the total request rate. The following query combines all model-inference traffic over a five-minute window:

sum(rate(aisix_llm_requests_total{outcome="success"}[5m]))
  /
sum(rate(aisix_llm_requests_total[5m]))

Add an endpoint filter or group by endpoint when you want to analyze one API separately.

To include traffic that is not counted as model inference — MCP tool calls, A2A agent calls, the provider passthrough, and the file, batch, and fine-tuning routes — run the same query against aisix_proxy_requests_total, which carries every proxied request.

To measure only the primary routing path, restrict the numerator and denominator to requests that were not served by a fallback target:

sum(rate(aisix_llm_requests_total{outcome="success", is_fallback="false"}[5m]))
  /
sum(rate(aisix_llm_requests_total{is_fallback="false"}[5m]))

Whether rate-limited requests belong in the population is an operational policy decision. To exclude clients that reached their quota from the denominator, use:

sum(rate(aisix_llm_requests_total{outcome="success"}[5m]))
  /
sum(rate(aisix_llm_requests_total{outcome!="rate_limited"}[5m]))

Calculate Aggregate Latency Percentiles

Combine histogram buckets across gateway instances before calculating a percentile. P90 is the value at or below which 90% of observations fall. Keep le in the sum by grouping, and add labels such as model or provider when you need a breakdown:

# P90 end-to-end latency across all matched gateway instances
histogram_quantile(
  0.90,
  sum by (le) (rate(aisix_request_e2e_latency_seconds_bucket{status_class="2xx"}[5m]))
)

# P90 end-to-end latency per model
histogram_quantile(
  0.90,
  sum by (le, model) (rate(aisix_request_e2e_latency_seconds_bucket{status_class="2xx"}[5m]))
)

# P90 time to first token per provider
histogram_quantile(
  0.90,
  sum by (le, provider) (rate(aisix_request_ttft_seconds_bucket[5m]))
)

Streaming end-to-end time covers the full generation, so streaming and non-streaming requests have different latency distributions. Use the streaming label to analyze them separately:

# P90 end-to-end latency for successful streaming requests
histogram_quantile(
  0.90,
  sum by (le) (rate(aisix_request_e2e_latency_seconds_bucket{streaming="true", status_class="2xx"}[5m]))
)

Compare Single-Instance Streaming Latency

Summary series expose precomputed quantile labels for each gateway instance. Select one scrape target and the model or provider you want to inspect:

# P90 time to first token for streaming chat completions
aisix_llm_time_to_first_token_seconds{endpoint="/v1/chat/completions", quantile="0.9"}

# P90 time to response start for that same traffic
aisix_llm_request_duration_seconds{endpoint="/v1/chat/completions", stream="true", quantile="0.9"}

Both queries pin endpoint because the two series do not cover the same traffic by default. Time to first token is recorded only for streaming /v1/chat/completions and /v1/messages, while the duration series covers every model-inference endpoint — so an unpinned comparison lets streamed /v1/responses traffic move one P90 and not the other.

aisix_llm_request_duration_seconds is recorded when the gateway hands the response to the client. On a streamed request that happens before any frame is read, so the value covers the work up to response start and excludes the whole generation. It is not an end-to-end figure for streaming traffic and does not become one by filtering on stream="true". The same applies to aisix_request_duration_seconds and aisix_proxy_request_duration_seconds, which are recorded at the same moment. On non-streamed traffic all three do cover the full request.

For a streamed request's end-to-end latency use aisix_request_e2e_latency_seconds, which is recorded at stream completion. It is a histogram rather than a summary, so read it with the quantile queries above rather than per-instance.

Do not average summary quantiles across instances. Use the histogram queries above to calculate percentiles across gateway instances.

Measure Guardrail Latency and Outcomes

aisix_guardrail_latency_seconds records each timed guardrail execution, labeled with the guardrail name, kind, phase, and result. A guardrail normally runs once per applicable phase. Streaming window scans can run the same output guardrail several times for one response. Synchronous per-field PII and keyword masking operations are not included. Calculate P95 execution latency per guardrail to verify a moderation budget:

histogram_quantile(
  0.95,
  sum by (le, guardrail) (rate(aisix_guardrail_latency_seconds_bucket[5m]))
)

Because guardrails run sequentially, their total contribution to request latency is the sum of their execution times for that request. Calculate the mean execution time for each guardrail and phase, and compare it with your moderation budget:

sum by (guardrail, phase) (rate(aisix_guardrail_latency_seconds_sum[5m]))
/
sum by (guardrail, phase) (rate(aisix_guardrail_latency_seconds_count[5m]))

Compare local detection with remote moderation services using the kind label. keyword and pii run in-process; every other kind invokes a remote service:

histogram_quantile(
  0.95,
  sum by (le, kind) (rate(aisix_guardrail_latency_seconds_bucket[5m]))
)

The _count series doubles as an execution counter. Track block and fail-open bypass rates per guardrail, and alert when a remote guardrail starts failing open:

sum by (guardrail, result) (rate(aisix_guardrail_latency_seconds_count[5m]))

# Fail-open bypasses by failure cause
sum by (guardrail, error_type) (rate(aisix_guardrail_latency_seconds_count{result="bypassed"}[5m]))

Calculate Token Volume by Client

Use token_type="total" to calculate cache-inclusive token volume by normalized client type:

sum by (client_type) (rate(aisix_llm_tokens_by_client_total{token_type="total"}[5m]))

To compare input and output volume, select both token types and include token_type in the grouping:

sum by (client_type, token_type) (rate(aisix_llm_tokens_by_client_total{token_type=~"input|output"}[5m]))

Break Down Token Volume by Client Type and Model

The model label records the model name the client requested, not the direct model AISIX selected. Group by client_type and model to see how each normalized client type distributes token volume across models:

sum by (client_type, model) (rate(aisix_llm_tokens_by_client_total{token_type="total"}[5m]))

To inspect one client type, filter on client_type and group by model only:

sum by (model) (rate(aisix_llm_tokens_by_client_total{client_type="claude-code", token_type="total"}[5m]))

Configure Metrics

Configure custom client classification and histogram bucket boundaries at startup. Changes take effect after a gateway restart and can alter label values or bucket series, so coordinate them with dashboards, alerts, and recording rules.

Map Custom Clients to a Client Type

AISIX recognizes common AI coding clients and SDKs out of the box. To classify in-house tools — or re-bucket a client that the built-in rules place in a generic bucket such as node — define mapping rules in the gateway configuration:

config.yaml
observability:
  metrics:
    client_type_rules:
      - pattern: "^billing-batcher/"
        client: billing-batcher
      - pattern: "internal-eval-harness"
        client: eval-harness

Each rule maps a regular expression to a fixed client value, which AISIX emits as the client_type label. Rules are evaluated in order against the raw User-Agent header, before the built-in rules, and the first match wins. Matching is case-insensitive and unanchored, so anchor the pattern with ^ when you need a prefix match.

The client value — never the request's User-Agent — becomes the label, so the label set stays bounded regardless of what clients send. Configuration limits keep it that way: at most 64 rules, patterns up to 512 bytes, and client values of up to 64 characters matching [a-z0-9][a-z0-9._-]*. AISIX validates the rules at startup and refuses to start on an invalid rule; changes take effect after a restart.

Requests with an empty User-Agent always report unknown, and requests that match no rule fall through to the built-in classification.

Customize Histogram Buckets

Four metrics are true Prometheus histograms with le bucket boundaries, and each one ships its own default boundaries because the distributions differ:

Metric	Default boundaries (seconds)
`aisix_request_e2e_latency_seconds`	0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2, 5, 10, 30, 60, 120, 300, 420, 600
`aisix_request_ttft_seconds`	0.05, 0.1, 0.25, 0.5, 1, 2, 5, 10, 30, 60, 120, 300
`aisix_guardrail_latency_seconds`	0.001, 0.0025, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10, 30
`aisix_a2a_ttfb_seconds`	0.05, 0.1, 0.25, 0.5, 1, 2, 5, 10, 30, 60, 120, 300

End-to-end latency includes cache hits and requests rejected before dispatch, so it needs millisecond boundaries. Time to first token (TTFT) is recorded at the first frame an upstream streams, whatever it carries; against a hosted provider, sub-50 ms boundaries are usually empty. Guardrail metrics need low boundaries for fast in-process checks and high boundaries for remote moderation services. An A2A agent's time to its first streamed event starts from the same defaults as TTFT. It takes its own a2a_ttfb override, because an agent task can think for minutes before it says anything.

Override the boundaries per metric when your traffic has a different distribution. For example, a vLLM or Ollama model server on the same node can produce its first output in milliseconds:

config.yaml
observability:
  metrics:
    buckets:
      request_ttft: [0.005, 0.01, 0.025, 0.05, 0.1, 0.5, 1, 5, 30]

Each field is optional and replaces only the metric it names; omitted metrics keep their defaults. A supplied list must contain 1–64 finite, positive, strictly increasing boundaries. Do not list +Inf — AISIX appends that bucket itself. AISIX validates the configuration at startup and refuses to start on an invalid list; changes take effect after a restart.

Every boundary adds one _bucket time series for each label combination, so a longer list increases the number of series a collector stores.

For a gateway deployed with the AISIX Helm chart, provide the comma-separated list through extraEnvVars:

extraEnvVars:
  - name: AISIX_OBSERVABILITY__METRICS__BUCKETS__REQUEST_TTFT
    value: "0.005,0.01,0.025,0.05,0.1,0.5,1,5,30"

caution

Changing boundaries changes the emitted _bucket series. A dashboard or recording rule that selects a removed le value stops matching. Do not compare bucket-derived quantiles from before and after the change.

See Set Any Other Gateway Configuration for the Helm configuration pattern.

Metrics Catalog

Metric Types

Request Metrics

Labels

Values for outcome

Behavior

PromQL example

Labels

Values for inbound_protocol

Values for stream

Values for is_fallback

Values for outcome

Behavior

Labels

Values for inbound_protocol

Values for stream

Values for is_fallback

Values for outcome

Behavior

Labels

Values for inbound_protocol

Values for stream

Values for is_fallback

Values for outcome

Labels

Values for inbound_protocol

Behavior

PromQL example

Labels

Behavior

PromQL example

Labels

Values for inbound_protocol

Values for outcome

Behavior

PromQL example

Labels

Values for method

Values for result

Behavior

PromQL example

Latency Metrics

Labels

Behavior

Labels

Values for inbound_protocol

Values for stream

Values for outcome

Labels

Values for inbound_protocol

Values for stream

Values for outcome

Labels

Values for inbound_protocol

Labels

Values for status_class

Values for streaming

Behavior

PromQL example

Labels

Values for status_class

Values for streaming

Behavior

PromQL example

Usage and Cost Metrics

Labels

Labels

Values for inbound_protocol

Labels

Values for inbound_protocol

Labels

Values for inbound_protocol

Labels

Values for inbound_protocol

Labels

Values for client_type

Values for token_type

Behavior

PromQL example

Rate Limit and Budget Metrics

Values for `outcome`

Values for `inbound_protocol`

Values for `stream`

Values for `is_fallback`

Values for `outcome`

Values for `inbound_protocol`

Values for `stream`

Values for `is_fallback`

Values for `outcome`

Values for `inbound_protocol`

Values for `stream`

Values for `is_fallback`

Values for `outcome`

Values for `inbound_protocol`

Values for `inbound_protocol`

Values for `outcome`

Values for `method`

Values for `result`

Values for `inbound_protocol`

Values for `stream`

Values for `outcome`

Values for `inbound_protocol`

Values for `stream`

Values for `outcome`

Values for `inbound_protocol`

Values for `status_class`

Values for `streaming`

Values for `status_class`

Values for `streaming`

Values for `inbound_protocol`

Values for `inbound_protocol`

Values for `inbound_protocol`

Values for `inbound_protocol`

Values for `client_type`

Values for `token_type`

Values for `scope`

Values for `metric value`

Values for `outcome`

Values for `cause`

Values for `op`

Values for `metric value`

Values for `kind`

Values for `phase`

Values for `result`

Values for `error_type`

Values for `reason`

Values for `handler`

Values for `status_code`

Values for `inbound_protocol`

Values for `reason`

Values for `metric value`