Metrics and Usage Events
AISIX AI Gateway exposes aggregate metrics and exportable usage events. Together, these signals show service health, traffic trends, and the model, route, and policy outcome behind each request attempt.
Choose a Telemetry Source
Start with the source that matches the operational question, then correlate signals by request, model, or provider when an issue needs deeper investigation.
| Operational Question | Start With | What It Provides |
|---|---|---|
| Is traffic healthy across gateway instances? | Prometheus metrics | Request rates, latency distributions, token and cost counters, policy outcomes, routing health, cache behavior, and exporter delivery health. |
| What happened to one request? | Access logs | Structured request fields, including status, latency, model, provider, request ID, and routing outcome when available. |
| What can the calling application observe? | Response headers | Request correlation, cache outcome, retry timing, and selected-target hints on supported routes. |
| Where can request records be stored, analyzed, or used for accounting? | Usage events | Per-attempt outcome and consumption records delivered through an observability exporter. |
Scrape Prometheus Metrics
AISIX serves Prometheus metrics on the dedicated metrics listener at /metrics by default. Change the path or disable the endpoint through the startup observability settings.
This endpoint is unauthenticated by design. Keep the dedicated metrics listener private.
Configure Prometheus exposure in the startup configuration:
observability:
metrics:
prometheus:
enabled: true
path: "/metrics"
The dedicated listener binds to 0.0.0.0:9090 by default. Set a different listener address when Prometheus should scrape another interface or port.
Scrape the default metrics endpoint:
curl -sS "http://127.0.0.1:9090/metrics"
AISIX publishes configuration status on every scrape. Other metric families are registered on first observation, so traffic metrics might not appear immediately after startup. Send one model request, then check again for series such as aisix_requests_total and aisix_tokens_consumed_total.
AISIX emits native metric names with the aisix_ prefix. Use the histogram series for latency percentiles across gateway instances and the request counters for success-rate and routing analysis. For exact metric names, label scope, and PromQL examples, see Metrics Reference.
Select the labels of individual metric families with observability.metrics.labels. For configuration examples, defaults, and the complete supported-variable list, see Metric Labels and Variables.
Import the Grafana Overview Dashboard
A prebuilt AISIX AI Gateway dashboard provides a starting view of gateway traffic, latency, token throughput, spend, governance (rate limits, guardrails, and inbound authentication), and upstream health. It uses dashboard ID 25746 and only built-in Grafana panel types. The published JSON declares Grafana 11.0.0; validate the dashboard before using it on an older Grafana release.
In Grafana, select Dashboards, then New, then Import. Enter 25746 and select the Prometheus data source that scrapes the gateway. To fetch the dashboard JSON directly instead:
curl -sS "https://grafana.com/api/dashboards/25746/revisions/latest/download" -o aisix-dashboard.json
Account for these coverage limits when interpreting the dashboard:
| Area | Coverage |
|---|---|
| Metric catalog | The dashboard has no dedicated panels for cache, budget, exporter, MCP, A2A, or configuration-status metrics. Use the Metrics Reference for the complete list and task-specific PromQL. |
| Provider and model filters | Filtered panels expect the default provider and model labels. If you remove either label with observability.metrics.labels, update the variables and affected queries. See Metric Labels and Variables. |
| Optional signals | Feature panels remain empty until the feature produces data. Guardrail panels require an attached guardrail, and client-cancellation data appears only for disconnects before response headers. |
| Spend | Gateway spend currently covers Realtime sessions. On an open-source gateway, it requires model cost metadata. AISIX Cloud calculates other request costs, including Chat Completions, Messages, and Responses requests, in the control plane; use Model Pricing and Request Logs for those values. |
| Control-plane metrics | The dashboard reads gateway metrics with the aisix_ prefix, not the Dev-only aisix_cp_ metrics. |
Count Requests and Attempts Separately
A request is one caller interaction. A retry or failover can produce several usage-event attempts for one request. Some attempts stop at a target-level rate limit or during request assembly before calling the provider, so AISIX counts attempts differently depending on the signal:
| Unit | Where it is counted | What one sample or row means |
|---|---|---|
| Request | aisix_proxy_requests_total, aisix_llm_requests_total | One client request, with the status the caller received. |
| Attempt | aisix_deployment_requests_total and the deployment families | One upstream call to one target model. |
| Emission attempt | aisix_usage_events_emitted_total | One attempt to enqueue a usage event, counted before the delivery queue accepts or rejects it. |
| Attempt | Usage-event records, and the usage log built from them | One recorded processing attempt, including one stopped before an upstream call. Sibling attempts share request_id and are ordered by attempt_index. |
If one target returns 502 and a fallback succeeds, the request counter records only the 200 returned to the caller. Deployment counters and usage events retain the failed attempt. A usage log can therefore contain many more 5xx attempts than the request metrics show; the two sources are measuring different units.
Use these queries for the questions they actually answer:
# Requests that ended in a server error — what callers experienced.
sum(increase(aisix_proxy_requests_total{status=~"5.."}[24h]))
# Requests that ended in a server error even after failing over.
sum(increase(aisix_proxy_requests_total{status=~"5..", is_fallback="true"}[24h]))
# Upstream attempts that failed, by target — including the ones a fallback
# rescued. This is the attempt-level view of a 5xx usage-log row count,
# restricted to the endpoints that dispatch through a Model Group.
sum(increase(aisix_deployment_failure_responses_total[24h])) by (model)
# Fallbacks that rescued a request, by group and by the target reached.
sum(increase(aisix_routing_successful_fallbacks_total[24h])) by (model, fallback_model)
# Usage-event emission attempts, and those the handoff queue rejected.
sum(increase(aisix_usage_events_emitted_total{status_code="5xx"}[24h]))
sum(increase(aisix_usage_event_drops_total[24h]))
# One member's rate-limit rejections. `status` carries the raw HTTP code,
# so a single failure mode is addressable without scanning a whole family.
sum(increase(aisix_usage_events_emitted_total{user_id="<member-id>", status="429"}[24h]))
# Whose queue handoffs were rejected. Both counters carry the same model and
# provider-key labels, so the difference holds per model, not only in total.
sum(increase(aisix_usage_event_drops_total[24h])) by (model, provider_key_name)
Compare the Same Population
Before treating two totals as inconsistent, check their coverage:
| Difference | Why it happens | What to check |
|---|---|---|
| Scrape coverage | Request counters have no environment label. A per-environment usage total is comparable only when Prometheus scrapes every gateway serving that environment. | Group the raw counter by job and instance, then compare the contributors with the running gateway instances. |
| Time coverage | increase(...[24h]) uses only samples present in the range. Restarts or shorter Prometheus retention can leave gaps that are absent from a usage-log query. | Graph the raw counter over the same period. |
| Delivery coverage | aisix_usage_event_drops_total counts events that could not enter the delivery queue because the sink was disabled or the queue was full or closed. Failures after queue acceptance do not increment it. | Group queue drops by model. Monitor telemetry batch failed (events dropped) for control-plane delivery and sink delivery dropped after retries for exporter delivery. |
| No upstream call | Target-level rate limits and request-assembly failures still produce usage events but no deployment counter sample. Assembly failures include unusable credentials, a missing model_name, and a missing or malformed api_base. | A misconfigured target produces failed requests with no deployment failures because the provider was never called. |
Deployment metric families cover endpoints dispatched through a Model Group, while usage events also cover direct models. Isolate one difference at a time rather than interpreting the entire gap as gateway-side rejection.
Export Usage Events
Usage events are per-attempt records emitted by supported proxy paths. A request that retries or fails over emits multiple events with the same request_id, ordered by attempt_index. Counting these records therefore counts attempts, not requests — see Count Requests and Attempts Separately before comparing a record count against a request metric.
Usage events are not read from a local endpoint. Configure an exporter in Observability Exporters to deliver them to OTLP/HTTP, object storage, Alibaba Cloud SLS, or Datadog.
Each event includes its outcome, consumption details, the model alias the caller requested, and the resolved model that served the attempt when the gateway can observe those values. A response served from the response cache records the matching layer in cache_hit_layer (exact or semantic) and, for semantic hits, the matched similarity in cache_similarity.
Latency fields distinguish provider time from caller-visible time:
| Field | Scope |
|---|---|
upstream_latency_ms | Time spent on one upstream attempt. It excludes request parsing, guardrails, routing, retry delays, and earlier attempts. |
upstream_ttft_ms | Time from the start of an upstream attempt until its first streamed frame of any type — metadata openers such as response.created or a role-only chat delta included, matching what caller-side proxies measure. It is omitted or zero for non-streaming requests, errors, and cache hits. |
downstream_latency_ms | Time from request receipt to downstream delivery: the complete response for non-streaming, the first token or relayed frame for streaming, and the entire stream for A2A. It includes gateway processing, retries and retry delays, and output holdback, and appears only on the terminal attempt. |
A caller that disconnects part-way through a streamed response still produces a usage event, because the upstream did the work and may have charged for it. That event carries status 499 rather than 200, and its token counts cover only what arrived before the disconnect. Count streamed successes with status_code = 200 to keep abandoned responses out of the total.
Tell Request Kinds Apart
Every event carries operation, which identifies the kind of work from the matched endpoint. Neither inbound_protocol nor the model name reliably distinguishes a conversation, image, video, or other operation, so use this field when grouping traffic by request type.
The values form a fixed set suitable for indexing, grouping, and charting:
| Value | Endpoint |
|---|---|
chat | /v1/chat/completions |
messages | /v1/messages |
count_tokens | /v1/messages/count_tokens |
responses | /v1/responses |
completions | /v1/completions |
embeddings | /v1/embeddings |
rerank | /v1/rerank |
image_generation | /v1/images/generations |
image_edit | /v1/images/edits |
transcription | /v1/audio/transcriptions |
translation | /v1/audio/translations |
speech | /v1/audio/speech |
video_generation | POST /v1/videos |
realtime | /v1/realtime |
files, batches, fine_tuning | The file, batch, and fine-tuning management endpoints |
batch_completion | The gateway's own accounting of a finished batch job, recorded when the job completes rather than when it was submitted |
mcp, a2a | The MCP and A2A gateways |
passthrough | A passthrough route |
Keep these behaviors in mind when querying the field:
- It describes the request, not the outcome. A request that failed, or that a guardrail refused, carries the same value a successful one would. It is the only field on such a record that names the endpoint at all.
- It is request-scoped. A request that retries or fails over emits one event per attempt, and every attempt carries the same value, so counting events per operation counts attempts. See Count Requests and Attempts Separately.
- Polling a video job is not video generation. Only the submission (
POST /v1/videos) produces a usage event; retrieving the job's status or downloading its result does not. A count ofvideo_generationis therefore a count of videos asked for, not of requests made about them.
Because operation is metadata, exporters retain it in metadata_only mode even though they receive no prompt content. In Alibaba Cloud SLS, it arrives as a separate column. Enable analytics for the fields used by a query, then group traffic by operation:
* | SELECT operation, COUNT(*) AS calls, SUM(prompt_tokens + completion_tokens) AS tokens GROUP BY operation ORDER BY calls DESC
To isolate one kind of traffic, filter on the operation directly, for example operation: video_generation. Do not infer it from the model: one model can serve several endpoints, and requested_model identifies the configured alias or group rather than the request type.
Attribute Events to a Member
Each event uses user_id to identify the organization member who owned the caller API key when the request ran:
| Situation | Recorded behavior |
|---|---|
| The caller API key has no member | user_id is absent. Ownership is assigned explicitly, not inferred from whoever created the key. |
| One member uses several credentials | Events can have different api_key_id values and the same user_id. This includes API key and OIDC calls that resolve to different keys owned by one member. |
| A key is reassigned or deleted | Existing events keep the original user_id. Reassignment attributes later requests to the new owner; deletion does not erase earlier attribution. |
| The event predates gateway support for this field | No member is recorded, so member filters cover traffic only from the upgrade forward. |
Filter by user_id to include all credentials owned by a member, or by api_key_id to isolate one credential.
In the dashboard, Logs offers this as the Member filter, alongside a Status filter that accepts a family (4xx), an exact code (429), or a range (500-599). Combining the two answers questions like "which of this member's requests were rate-limited in the last 24 hours" in one query. The CSV export carries user_id and the member's name.
OpenAI Cache-Write Tokens
AISIX recognizes cache_write_tokens inside Chat Completions usage.prompt_tokens_details or Responses usage.input_tokens_details. It preserves that raw value as the optional cache_write_tokens field of the usage event, including requests bridged through /v1/messages or /v1/responses and supported protocol-aware passthrough routes. This requires a gateway newer than 1.1.0.
{
"prompt_tokens": 101,
"completion_tokens": 11,
"cached_prompt_tokens": 19,
"cache_write_tokens": 37
}
The field is omitted when the upstream did not report it; an explicit 0 remains zero. The AISIX Cloud Logs detail, usage-events API, and JSON/CSV export preserve that distinction. CSV uses an empty cell for an absent value. Logs exported through a backend with prefixed field names use its usual prefix, such as aisix.cache_write_tokens in Datadog.
This is separate from Anthropic's additive cache_creation_tokens. It does not increase input/output totals or change the existing cost calculation. In the example above, input plus output is still 112. No existing cache counter is renamed or merged.
Next Steps
Configure Observability Exporters to send usage events to an external collector, log destination, object store, or warehouse workflow. Use Access Logs and Request Correlation to investigate individual requests, and use the Metrics Reference when building Prometheus dashboards or alerts.