Skip to main content
API7.ai Logo

Metrics Reference

AISIX exposes operational metrics in Prometheus text format. A Prometheus server or another compatible collector can scrape these metrics for dashboards, alerts, and PromQL queries.

Prometheus metrics are enabled by default on a dedicated listener. The default startup configuration is:

config.yaml
observability:
  metrics:
    prometheus:
      enabled: true
      addr: 0.0.0.0:9090
      path: /metrics

Prometheus scrapes GET /metrics from this listener. The Admin API listener does not serve metrics.

The metrics endpoint is unauthenticated by design. Keep its listener private to your monitoring network.

AISIX publishes configuration status whenever the endpoint is scraped. Other metric families are registered when AISIX first records the corresponding activity, so traffic metrics might not appear immediately after startup. Send a request through the proxy, then scrape again for the corresponding series to appear.

Metrics Catalog

Search metric names, descriptions, labels, and values, or filter the catalog by family and type. Expand an entry to review its details.

Metrics
41
Families
8
Query behavior

Metric Types

counter

A cumulative value that increases until the process restarts, such as a request or token total. Use rate() to calculate how quickly it changes.

gauge

A current value that can increase or decrease, such as active requests or remaining quota.

histogram

Observations counted in configurable buckets. The _bucket, _sum, and _count series can be aggregated before calculating a percentile.

summary

Observations with quantiles calculated by each gateway instance. Summaries also expose _sum and _count, but their quantiles cannot be aggregated across instances.

Family
Type
Showing 41 of 41 entries

Request Metrics

Track request outcomes and the work currently in progress at the proxy.

Metric entries: 5
Detailed Request Labels

For the three detailed request counters, stream records whether the client requested streaming. is_fallback records whether a fallback target served the request and does not appear on latency metrics.

provider_key_name and user_name are human-readable companions to their IDs. Each name has a one-to-one relationship with its ID, so it does not add another series dimension. user_name is unknown until the control plane supplies it.

inbound_protocol is a bounded protocol family. Detailed chat and messages series use openai and anthropic; the in-flight gauge can also use mcp, a2a, and realtime.

aisix_requests_total . Description: Proxy request outcomes in the compatibility series with the broadest endpoint coverage. . Type: counter . Label count: 4 labels
Labels
provider, model, status, outcome
Values for outcome
success, client_error, rate_limited, upstream_error

success covers HTTP 200–399, client_error covers HTTP 400–499 except 429, rate_limited is HTTP 429, and all other statuses map to upstream_error.

Behavior

A2A agent calls use provider="a2a" and model="a2a".

PromQL example
sum(rate(aisix_requests_total[5m])) by (outcome)
aisix_llm_requests_total . Description: Chat-completions and messages request outcomes, including successful and failed requests. . Type: counter . Label count: 15 labels
Labels
endpoint, inbound_protocol, provider, model, upstream_model, provider_key_id, provider_key_name, api_key_id, team_id, user_id, user_name, stream, is_fallback, status, outcome
Values for inbound_protocol
openai, anthropic
Values for stream
false, true
Values for is_fallback
false, true
Values for outcome
success, client_error, rate_limited, upstream_error

success covers HTTP 200–399, client_error covers HTTP 400–499 except 429, rate_limited is HTTP 429, and all other statuses map to upstream_error.

aisix_proxy_requests_total . Description: Detailed request outcomes for chat-completions and messages traffic. . Type: counter . Label count: 15 labels
Labels
endpoint, inbound_protocol, provider, model, upstream_model, provider_key_id, provider_key_name, api_key_id, team_id, user_id, user_name, stream, is_fallback, status, outcome
Values for inbound_protocol
openai, anthropic
Values for stream
false, true
Values for is_fallback
false, true
Values for outcome
success, client_error, rate_limited, upstream_error

success covers HTTP 200–399, client_error covers HTTP 400–499 except 429, rate_limited is HTTP 429, and all other statuses map to upstream_error.

aisix_proxy_failed_requests_total . Description: The subset of proxy requests whose outcome is not success. . Type: counter . Label count: 15 labels
Labels
endpoint, inbound_protocol, provider, model, upstream_model, provider_key_id, provider_key_name, api_key_id, team_id, user_id, user_name, stream, is_fallback, status, outcome
Values for inbound_protocol
openai, anthropic
Values for stream
false, true
Values for is_fallback
false, true
Values for outcome
client_error, rate_limited, upstream_error

client_error covers HTTP 400–499 except 429, rate_limited is HTTP 429, and all other statuses map to upstream_error.

aisix_proxy_in_flight_requests . Description: Requests currently being handled by the proxy, grouped by normalized endpoint and inbound protocol. . Type: gauge . Label count: 2 labels
Labels
endpoint, inbound_protocol
Values for inbound_protocol
openai, anthropic, mcp, a2a, realtime
Behavior

MCP requests use endpoint="/mcp" and inbound_protocol="mcp". A2A calls use endpoint="/a2a" and inbound_protocol="a2a".

PromQL example
sum(aisix_proxy_in_flight_requests) by (endpoint, inbound_protocol)

Latency Metrics

Inspect latency on one gateway or aggregate histogram buckets across gateway instances.

Metric entries: 6
Latency Aggregation and Labels

Use histograms for service-level dashboards and alerts across gateway instances because their _bucket, _sum, and _count series can be aggregated before using histogram_quantile(). Use summaries to inspect precomputed quantiles from one gateway instance. Summary quantiles cannot be aggregated across instances, so do not average them.

Both histograms use bucket boundaries from 10 milliseconds to 300 seconds. env_id identifies the environment served by a managed gateway and is unknown when the gateway is not connected to AISIX Cloud. status_class is one of 2xx, 3xx, 4xx, 5xx, or other. Per-key and per-user labels are excluded to keep the number of bucket series manageable; use usage analytics for those dimensions.

aisix_request_duration_seconds . Description: End-to-end request latency in the compatibility series. . Type: summary . Label count: 3 labels
Labels
provider, model, status
Behavior

Summary quantiles are calculated by each AISIX instance and cannot be aggregated across instances.

aisix_llm_request_duration_seconds . Description: Chat-completions and messages latency. For streaming requests, this measures time until the response starts. . Type: summary . Label count: 14 labels
Labels
endpoint, inbound_protocol, provider, model, upstream_model, provider_key_id, provider_key_name, api_key_id, team_id, user_id, user_name, stream, status, outcome
Values for inbound_protocol
openai, anthropic
Values for stream
false, true
Values for outcome
success, client_error, rate_limited, upstream_error

success covers HTTP 200–399, client_error covers HTTP 400–499 except 429, rate_limited is HTTP 429, and all other statuses map to upstream_error.

aisix_proxy_request_duration_seconds . Description: Detailed chat-completions and messages latency with the same streaming semantics as the LLM duration series. . Type: summary . Label count: 14 labels
Labels
endpoint, inbound_protocol, provider, model, upstream_model, provider_key_id, provider_key_name, api_key_id, team_id, user_id, user_name, stream, status, outcome
Values for inbound_protocol
openai, anthropic
Values for stream
false, true
Values for outcome
success, client_error, rate_limited, upstream_error

success covers HTTP 200–399, client_error covers HTTP 400–499 except 429, rate_limited is HTTP 429, and all other statuses map to upstream_error.

aisix_llm_time_to_first_token_seconds . Description: Time from request entry to the first generated token for streaming chat-completions and messages requests. . Type: summary . Label count: 11 labels
Labels
endpoint, inbound_protocol, provider, model, upstream_model, provider_key_id, provider_key_name, api_key_id, team_id, user_id, user_name
Values for inbound_protocol
openai, anthropic
aisix_request_e2e_latency_seconds . Description: Client-perceived latency for chat completions, messages, and responses, including the full duration of streams. . Type: histogram . Label count: 6 labels
Labels
env_id, endpoint, model, provider, status_class, streaming
Values for status_class
2xx, 3xx, 4xx, 5xx, other
Values for streaming
false, true
Behavior

Buckets range from 10 milliseconds to 300 seconds. Aggregate _bucket series before calculating percentiles across gateway instances.

Each request is observed once. Non-streaming requests and failures are recorded when the handler returns. Streaming requests are recorded when the stream finishes, including client cancellation; a canceled stream retains its committed status and records the duration up to cancellation.

PromQL example
histogram_quantile(
  0.90,
  sum by (le) (rate(aisix_request_e2e_latency_seconds_bucket[5m]))
)
aisix_request_ttft_seconds . Description: Time to first token for streaming chat-completions and messages requests. . Type: histogram . Label count: 6 labels
Labels
env_id, endpoint, model, provider, status_class, streaming
Values for status_class
2xx, 3xx, 4xx, 5xx, other
Values for streaming
true
PromQL example
histogram_quantile(
  0.90,
  sum by (le) (rate(aisix_request_ttft_seconds_bucket[5m]))
)

Usage and Cost Metrics

Measure token volume, estimated spend, and normalized client usage.

Metric entries: 6
aisix_tokens_consumed_total . Description: The sum of usage.total_tokens across completed, non-streaming chat-completions requests. . Type: counter . Label count: 2 labels
Labels
provider, model
aisix_llm_input_tokens_total . Description: Input tokens reported by the upstream for chat-completions and messages requests. . Type: counter . Label count: 11 labels
Labels
endpoint, inbound_protocol, provider, model, upstream_model, provider_key_id, provider_key_name, api_key_id, team_id, user_id, user_name
Values for inbound_protocol
openai, anthropic
aisix_llm_output_tokens_total . Description: Output tokens reported by the upstream for chat-completions and messages requests. . Type: counter . Label count: 11 labels
Labels
endpoint, inbound_protocol, provider, model, upstream_model, provider_key_id, provider_key_name, api_key_id, team_id, user_id, user_name
Values for inbound_protocol
openai, anthropic
aisix_llm_total_tokens_total . Description: Total tokens reported by the upstream for chat-completions and messages requests. . Type: counter . Label count: 11 labels
Labels
endpoint, inbound_protocol, provider, model, upstream_model, provider_key_id, provider_key_name, api_key_id, team_id, user_id, user_name
Values for inbound_protocol
openai, anthropic
aisix_llm_spend_micro_usd_total . Description: Estimated non-streaming chat-completions spend in micro-USD, where 1 USD equals 1,000,000 micro-USD. . Type: counter . Label count: 11 labels
Labels
endpoint, inbound_protocol, provider, model, upstream_model, provider_key_id, provider_key_name, api_key_id, team_id, user_id, user_name
Values for inbound_protocol
openai, anthropic
aisix_llm_tokens_by_client_total . Description: Token volume for chat-completions, messages, and responses requests, grouped by normalized client, requested model, and token type. . Type: counter . Label count: 3 labels
Labels
client_type, model, token_type
Values for client_type
openai-python, openai-node, anthropic-python, anthropic-typescript, claude-code, codex, cline, roo-code, kilocode, zoo-code, github-copilot, cursor, opencode, qwen-code, gemini-cli, crush, zed, aider, vercel-ai-sdk, langchain, llamaindex, litellm, curl, python-requests, httpx, aiohttp, okhttp, go-http-client, node, postman, browser, other, unknown

An unrecognized User-Agent maps to other; a missing User-Agent maps to unknown. Deployments can extend this set with admin-defined mapping rules (observability.metrics.client_type_rules). Full user-agent strings and versions remain in request logs and usage analytics.

Values for token_type
input, output, total

total includes input, output, and Anthropic cache-creation and cache-read tokens.

Behavior

model is the caller-requested model name, matching the model label on the aisix_llm_* token series. Routing, semantic, ensemble, and fallback dispatch retain this alias instead of reporting the selected direct model.

The bounded client_type allowlist prevents a client-controlled user-agent from creating unbounded Prometheus cardinality. Concrete model aliases are bounded by the configured model set, but a wildcard alias records each concrete model name requested through that pattern. Restrict wildcard access and monitor label growth when cardinality matters.

Admin-defined mapping rules (observability.metrics.client_type_rules) can classify additional clients. Rules are matched before the built-in allowlist and emit a fixed, validated label value, so the label set stays bounded.

For chat-completions and messages traffic, the cache-inclusive total aggregated across labels aligns with aisix_llm_total_tokens_total. /v1/responses token usage appears in this client-type series but not in that per-key family, so this total can be larger.

Individual series do not align because the two metric families use different label sets. The dedicated client-type series avoids multiplying the per-key token series by another label dimension.

Anthropic reports cache tokens separately from input tokens, so total can exceed input plus output.

PromQL example
sum by (client_type, model, token_type) (
  rate(aisix_llm_tokens_by_client_total[5m])
)

Rate Limit and Budget Metrics

Monitor rate-limit rejections and the latest quota or budget state for each label set.

Metric entries: 8
aisix_ratelimit_rejections_total . Description: Chat-completions requests rejected by a rate limit. . Type: counter . Label count: 1 label
Labels
scope
Values for scope
requests, tokens
aisix_ratelimit_remaining_requests . Description: Remaining request quota reported while processing a chat-completions request, grouped by API key and model. . Type: gauge . Label count: 2 labels
Labels
api_key_id, model
aisix_ratelimit_remaining_tokens . Description: Remaining token quota reported while processing a chat-completions request, grouped by API key and model. . Type: gauge . Label count: 2 labels
Labels
api_key_id, model
aisix_budget_limit_usd . Description: Budget limit in USD. . Type: gauge . Label count: 3 labels
Labels
api_key_id, team_id, user_id
aisix_budget_spent_usd . Description: Budget spent in USD. . Type: gauge . Label count: 3 labels
Labels
api_key_id, team_id, user_id
aisix_budget_remaining_usd . Description: Budget remaining in USD. . Type: gauge . Label count: 3 labels
Labels
api_key_id, team_id, user_id
aisix_budget_reset_seconds . Description: Seconds until the budget period resets. . Type: gauge . Label count: 3 labels
Labels
api_key_id, team_id, user_id
aisix_budget_details_present . Description: Whether budget details are populated. . Type: gauge . Label count: 3 labels
Labels
api_key_id, team_id, user_id
Values for metric value
0, 1

1 means budget details are present; 0 means they have been cleared.

Deployment Metrics

Monitor whether target models remain in rotation and how often they enter cooldown.

Metric entries: 2
aisix_deployment_state . Description: Whether a target model is in rotation. . Type: gauge . Label count: 4 labels
Labels
provider, model, upstream_model, provider_key_id
Values for metric value
0, 2

0 is healthy. 2 is out of rotation because the target is cooling down or failing a background health check.

aisix_deployment_cooled_down_total . Description: Number of times a target model entered cooldown. . Type: counter . Label count: 4 labels
Labels
provider, model, upstream_model, provider_key_id

Guardrail Metrics

Track per-guardrail execution latency and outcomes across all endpoints, plus aggregate blocks and fail-open bypasses for chat-completions requests.

Metric entries: 3
Guardrail Execution Latency

Every guardrail consulted on a request records one aisix_guardrail_latency_seconds observation per phase, on every endpoint — including streaming end-of-stream and window scans, cache-hit output checks, and the segment moderation pass. The histogram uses explicit buckets from 1 ms to 30 s, so histogram_quantile yields P50/P95/P99 per guardrail.

The kind label separates local, in-process detection (keyword, pii) from remote moderation services (every other kind), and the result label separates fail-open bypasses (bypassed, with the failure tag in error_type) from policy decisions. A fail-closed remote failure surfaces as blocked; its latency clusters at the configured provider timeout.

The _count series doubles as a per-guardrail execution counter: sum by (guardrail, result) (rate(aisix_guardrail_latency_seconds_count[5m])) gives execution and block rates without a separate counter.

aisix_guardrail_latency_seconds . Description: Wall-clock duration of one guardrail execution, recorded per consulted guardrail per phase on every endpoint. guardrail is the configured guardrail name. . Type: histogram . Label count: 6 labels
Labels
env_id, guardrail, kind, phase, result, error_type
Values for kind
keyword, pii, aliyun_text_moderation, azure_content_safety, azure_content_safety_text_moderation, bedrock, lakera, openai_moderation, presidio

keyword and pii run in-process (local detection); every other kind calls a remote moderation service.

Values for phase
input, output
Values for result
allowed, blocked, masked, bypassed, would_block, would_mask

bypassed is a remote failure that failed open; a fail-closed remote failure records as blocked. would_block / would_mask come from enforcement_mode: monitor guardrails.

Values for error_type
none, aliyun_5xx, aliyun_config_error, aliyun_throttled, aliyun_timeout, azure_cs_5xx, azure_cs_config_error, azure_cs_throttled, azure_cs_timeout, bedrock_5xx, bedrock_throttled, bedrock_timeout, lakera_5xx, lakera_config_error, lakera_throttled, lakera_timeout, openai_moderation_5xx, openai_moderation_config_error, openai_moderation_throttled, openai_moderation_timeout, presidio_5xx, presidio_config_error, presidio_throttled, presidio_timeout

Set to the bounded failure tag when result="bypassed" (the same values as aisix_guardrail_bypasses_total{reason}), otherwise none.

Behavior

Explicit buckets: 0.001, 0.0025, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10, and 30 seconds.

Monitor-mode executions record would_block / would_mask while the request proceeds, so a staged policy can be sized before enforcing it.

PII and keyword mask rewrites applied through the synchronous per-field redactors are not timed by this series; their in-process cost is microseconds.

aisix_guardrail_blocks_total . Description: Requests rejected by an input or output guardrail, including policy blocks and fail-closed results. Chat-completions requests only; for per-guardrail block rates across all endpoints, use the _count series of aisix_guardrail_latency_seconds with result="blocked". . Type: counter . Label count: no labels
Labels

None.

aisix_guardrail_bypasses_total . Description: Fail-open events where a remote guardrail was unreachable and fail_open allowed the request to continue. . Type: counter . Label count: 1 label
Labels
reason
Values for reason
aliyun_5xx, aliyun_config_error, aliyun_throttled, aliyun_timeout, azure_cs_5xx, azure_cs_config_error, azure_cs_throttled, azure_cs_timeout, bedrock_5xx, bedrock_throttled, bedrock_timeout, lakera_5xx, lakera_config_error, lakera_throttled, lakera_timeout, openai_moderation_5xx, openai_moderation_config_error, openai_moderation_throttled, openai_moderation_timeout, presidio_5xx, presidio_config_error, presidio_throttled, presidio_timeout

Usage Event Metrics

Distinguish usage-event emission attempts from events the delivery queue did not accept.

Metric entries: 2
Usage Event Delivery

For each emission attempt, the event is either accepted by the queue or counted as a drop. After summing across labels, subtract the drop rate from the emission-attempt rate to calculate the accepted rate.

The status_code label is one of 2xx, 3xx, 4xx, 5xx, or other. Example handler values include chat, embeddings, messages, and mcp.

MCP tool calls use handler="mcp" and inbound_protocol="mcp". Their usage-event payloads identify the MCP server and tool; token and cost fields are zero.

A2A agent calls use handler="a2a". The metric maps their bounded inbound_protocol label to other, while the delivered event uses inbound_protocol="a2a" and identifies the agent name and JSON-RPC method. Token and cost fields are zero.

aisix_usage_events_emitted_total . Description: Usage-event emission attempts, counted before AISIX tries to enqueue the event, including both delivered and dropped events. . Type: counter . Label count: 3 labels
Labels
handler, status_code, inbound_protocol
Values for handler
a2a, audio, batch, batches, chat, completions, embeddings, files, fine_tuning, images, mcp, messages, passthrough, realtime, rerank, responses
Values for status_code
2xx, 3xx, 4xx, 5xx, other
Values for inbound_protocol
openai, anthropic, mcp, other
aisix_usage_event_drops_total . Description: Usage events that were not accepted by the queue. . Type: counter . Label count: 1 label
Labels
reason
Values for reason
sink_disabled, sink_full, sink_closed

Configuration Metrics

Monitor whether AISIX can load and apply changes from its configuration source.

Metric entries: 9
Configuration State

AISIX reflects the same live configuration state in these metrics and at GET /status/config on the metrics and status listener.

Reload metrics apply to both file and etcd configuration sources. Revision and source-connection metrics are emitted only when AISIX loads configuration from etcd.

Compare the observed and applied revisions to determine whether the gateway is serving the latest etcd configuration.

aisix_config_last_reload_successful . Description: Whether the latest configuration load succeeded. . Type: gauge . Label count: no labels
Labels

None.

Values for metric value
0, 1

1 indicates success and 0 indicates failure.

aisix_config_last_reload_success_timestamp_seconds . Description: Unix timestamp in seconds of the last successful configuration load. . Type: gauge . Label count: no labels
Labels

None.

Behavior

The series is not emitted until a configuration load succeeds.

aisix_config_reloads_total . Description: Full configuration reload attempts, including source fetch failures. . Type: counter . Label count: no labels
Labels

None.

Behavior

Incremental etcd watch events do not increment this counter.

aisix_config_reload_failures_total . Description: Configuration reload failures grouped by reason. . Type: counter . Label count: 1 label
Labels
reason
Values for reason
fetch, parse, validate

fetch indicates that the source could not be read, parse indicates invalid source data, and validate indicates an invalid resource.

aisix_config_rejected_resources . Description: Current number of rejected resources, grouped by resource kind. . Type: gauge . Label count: 1 label
Labels
kind
Behavior

When all rejections for a resource kind clear, AISIX sets its existing series to 0.

aisix_config_observed_revision . Description: Latest etcd revision observed by the gateway. . Type: gauge . Label count: no labels
Labels

None.

aisix_config_applied_revision . Description: The etcd revision represented by the configuration currently served by the gateway. . Type: gauge . Label count: no labels
Labels

None.

aisix_config_hash_info . Description: Applied configuration hash. . Type: gauge . Label count: 1 label
Labels
hash
Values for metric value
0, 1

Filter for value 1 to select the current hash. AISIX retains earlier hash labels with value 0 after the applied configuration changes.

aisix_config_source_connected . Description: Whether the gateway is connected to its etcd configuration source. . Type: gauge . Label count: no labels
Labels

None.

Values for metric value
0, 1

1 indicates connected and 0 indicates disconnected.

Analyze Metrics with PromQL​

Use these PromQL examples in the Prometheus expression browser or another compatible monitoring interface after configuring it to scrape the AISIX metrics endpoint. Adjust the time window, label filters, and grouping dimensions to match the traffic and gateway instances you want to inspect.

Calculate Success Rate​

Calculate the success rate by dividing the successful request rate by the total request rate. The following query combines chat-completions and messages traffic over a five-minute window:

sum(rate(aisix_llm_requests_total{outcome="success"}[5m]))
  /
sum(rate(aisix_llm_requests_total[5m]))

Add an endpoint filter or group by endpoint when you want to analyze one API separately.

To measure only the primary routing path, restrict the numerator and denominator to requests that were not served by a fallback target:

sum(rate(aisix_llm_requests_total{outcome="success", is_fallback="false"}[5m]))
  /
sum(rate(aisix_llm_requests_total{is_fallback="false"}[5m]))

Whether rate-limited requests belong in the population is an operational policy decision. To exclude clients that reached their quota from the denominator, use:

sum(rate(aisix_llm_requests_total{outcome="success"}[5m]))
  /
sum(rate(aisix_llm_requests_total{outcome!="rate_limited"}[5m]))

Calculate Aggregate Latency Percentiles​

Combine histogram buckets across gateway instances before calculating a percentile. P90 is the value at or below which 90% of observations fall. Keep le in the sum by grouping, and add labels such as model or provider when you need a breakdown:

# P90 end-to-end latency across all matched gateway instances
histogram_quantile(
  0.90,
  sum by (le) (rate(aisix_request_e2e_latency_seconds_bucket{status_class="2xx"}[5m]))
)

# P90 end-to-end latency per model
histogram_quantile(
  0.90,
  sum by (le, model) (rate(aisix_request_e2e_latency_seconds_bucket{status_class="2xx"}[5m]))
)

# P90 time to first token per provider
histogram_quantile(
  0.90,
  sum by (le, provider) (rate(aisix_request_ttft_seconds_bucket[5m]))
)

Streaming end-to-end time covers the full generation, so streaming and non-streaming requests have different latency distributions. Use the streaming label to analyze them separately:

# P90 end-to-end latency for successful streaming requests
histogram_quantile(
  0.90,
  sum by (le) (rate(aisix_request_e2e_latency_seconds_bucket{streaming="true", status_class="2xx"}[5m]))
)

Compare Single-Instance Streaming Latency​

Summary series expose precomputed quantile labels for each gateway instance. Select one scrape target and the model or provider you want to inspect before comparing time to first token with end-to-end latency:

# P90 time to first token for streaming requests
aisix_llm_time_to_first_token_seconds{quantile="0.9"}

# P90 end-to-end latency for the same streaming population
aisix_llm_request_duration_seconds{stream="true", quantile="0.9"}

Do not average summary quantiles across instances. Use the histogram queries above to calculate percentiles across gateway instances.

Measure Guardrail Latency and Outcomes​

aisix_guardrail_latency_seconds records one observation per consulted guardrail per phase, labeled with the guardrail name, kind, phase, and result. Calculate P95 execution latency per guardrail to verify a moderation budget:

histogram_quantile(
  0.95,
  sum by (le, guardrail) (rate(aisix_guardrail_latency_seconds_bucket[5m]))
)

Guardrails run sequentially within a request, so the latency a request pays is the sum of its guardrail execution times — one execution per attached guardrail per phase. Calculate the mean added latency per execution for each guardrail and phase, and compare it against your moderation budget:

sum by (guardrail, phase) (rate(aisix_guardrail_latency_seconds_sum[5m]))
/
sum by (guardrail, phase) (rate(aisix_guardrail_latency_seconds_count[5m]))

Compare local detection with remote moderation services using the kind label — keyword and pii run in-process, every other kind makes a network call:

histogram_quantile(
  0.95,
  sum by (le, kind) (rate(aisix_guardrail_latency_seconds_bucket[5m]))
)

The _count series doubles as an execution counter. Track block and fail-open bypass rates per guardrail, and alert when a remote guardrail starts failing open:

sum by (guardrail, result) (rate(aisix_guardrail_latency_seconds_count[5m]))

# Fail-open bypasses by failure cause
sum by (guardrail, error_type) (rate(aisix_guardrail_latency_seconds_count{result="bypassed"}[5m]))

Calculate Token Volume by Client​

Use token_type="total" to calculate cache-inclusive token volume by normalized client type:

sum by (client_type) (rate(aisix_llm_tokens_by_client_total{token_type="total"}[5m]))

To compare input and output volume, select both token types and include token_type in the grouping:

sum by (client_type, token_type) (rate(aisix_llm_tokens_by_client_total{token_type=~"input|output"}[5m]))

Break Down Token Volume by Client Type and Model​

The model label records the model name the client requested, not the direct model AISIX selected. Group by client_type and model to see how each normalized client type distributes token volume across models:

sum by (client_type, model) (rate(aisix_llm_tokens_by_client_total{token_type="total"}[5m]))

To inspect one client type, filter on client_type and group by model only:

sum by (model) (rate(aisix_llm_tokens_by_client_total{client_type="claude-code", token_type="total"}[5m]))

Map Custom Clients to a Client Type​

AISIX recognizes common AI coding clients and SDKs out of the box. To classify in-house tools — or re-bucket a client that the built-in rules place in a generic bucket such as node — define mapping rules in the gateway configuration:

config.yaml
observability:
  metrics:
    client_type_rules:
      - pattern: "^billing-batcher/"
        client: billing-batcher
      - pattern: "internal-eval-harness"
        client: eval-harness

Each rule maps a regular expression to a fixed client value, which AISIX emits as the client_type label. Rules are evaluated in order against the raw User-Agent header, before the built-in rules, and the first match wins. Matching is case-insensitive and unanchored, so anchor the pattern with ^ when you need a prefix match.

The client value — never the request's User-Agent — becomes the label, so the label set stays bounded regardless of what clients send. Configuration limits keep it that way: at most 64 rules, patterns up to 512 bytes, and client values of up to 64 characters matching [a-z0-9][a-z0-9._-]*. AISIX validates the rules at startup and refuses to start on an invalid rule; changes take effect after a restart.

Requests with an empty User-Agent always report unknown, and requests that match no rule fall through to the built-in classification.