# Prometheus

The `prometheus` plugin provides the capability to integrate APISIX with [Prometheus](https://prometheus.io).

After enabling the plugin, APISIX will start collecting relevant metrics, such as API requests and latencies, and exporting them in a [text-based exposition format](https://prometheus.io/docs/instrumenting/exposition_formats/#exposition-formats) to Prometheus. You can then create event monitoring and alerting in Prometheus to monitor the health of your API gateway and APIs.

## Metrics[​](#metrics "Direct link to Metrics")

There are different types of metrics in Prometheus. To understand their differences, see [metrics types](https://prometheus.io/docs/concepts/metric_types/).

The following metrics are exported by the `prometheus` plugin by default. See [get APISIX metrics](#get-apisix-metrics) for an example. Note that some metrics, such as `apisix_batch_process_entries`, are not readily visible if there are no data.

| Name                                      | Type      | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| ----------------------------------------- | --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| apisix\_bandwidth                         | counter   | Total amount of traffic flowing through APISIX in bytes.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| apisix\_etcd\_modify\_indexes             | gauge     | Number of changes to etcd by APISIX keys.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| apisix\_batch\_process\_entries           | gauge     | Number of remaining entries in a batch when sending data in batches, such as with `http logger`, and other logging plugins.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| apisix\_etcd\_reachable                   | gauge     | Whether APISIX can reach etcd. A value of `1` represents reachable and `0` represents unreachable.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| apisix\_http\_status                      | counter   | HTTP status codes returned to clients. This is the status the client receives after plugins and proxying, which can differ from the upstream status.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| apisix\_http\_requests\_total             | gauge     | Number of HTTP requests from clients.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| apisix\_nginx\_http\_current\_connections | gauge     | Number of current connections with clients.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| apisix\_nginx\_metric\_errors\_total      | counter   | Total number of `nginx-lua-prometheus` errors.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| apisix\_http\_latency                     | histogram | HTTP request latency in milliseconds.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| apisix\_node\_info                        | gauge     | Information about the APISIX node, such as the host name and APISIX version.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| apisix\_shared\_dict\_capacity\_bytes     | gauge     | The total capacity of an [NGINX shared dictionary](https://github.com/openresty/lua-nginx-module#ngxshareddict).                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
| apisix\_shared\_dict\_free\_space\_bytes  | gauge     | The remaining space in an [NGINX shared dictionary](https://github.com/openresty/lua-nginx-module#ngxshareddict).                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| apisix\_nginx\_timers                     | gauge     | Number of running and pending Lua timers in each NGINX process and subsystem. Introduced in APISIX 3.20.0.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| apisix\_nginx\_timers\_max                | gauge     | Configured maximum number of running and pending Lua timers. Introduced in APISIX 3.20.0.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| apisix\_nginx\_process\_connections       | gauge     | Number of connection slots in use by each NGINX process. Introduced in APISIX 3.20.0.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| apisix\_nginx\_process\_connections\_max  | gauge     | Number of connection slots available to each NGINX process. Introduced in APISIX 3.20.0.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| apisix\_nginx\_lua\_memory\_bytes         | gauge     | Memory allocated by the Lua VM in each NGINX process and subsystem. Introduced in APISIX 3.20.0.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
| apisix\_upstream\_status                  | gauge     | Health check status of upstream nodes, available if health checks are configured on the upstream. A value of `1` represents healthy and `0` represents unhealthy.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| apisix\_stream\_connection\_total         | counter   | Total number of connections handled per stream route.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| apisix\_stream\_active\_connections       | gauge     | Number of active TCP connections and UDP sessions for each stream listening address. Requires APISIX-Runtime. Introduced in API7 Enterprise 3.9.19 on the 3.9 line and 3.10.6 on the 3.10 line, and in APISIX 3.18.0.                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| apisix\_stream\_status                    | counter   | Number of completed stream sessions by termination status, listening address, and upstream node. Introduced in API7 Enterprise 3.9.19 on the 3.9 line and 3.10.6 on the 3.10 line, and in APISIX 3.18.0.                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| apisix\_stream\_bandwidth                 | counter   | Number of bytes proxied by the stream subsystem by listening address, direction, and side. Requires APISIX-Runtime. Introduced in API7 Enterprise 3.9.19 on the 3.9 line and 3.10.6 on the 3.10 line, and in APISIX 3.18.0.                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| apisix\_llm\_prompt\_tokens               | counter   | Number of prompt tokens. Only exported for AI request types. Introduced in API7 Enterprise 3.9.7 and APISIX 3.17.0.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| apisix\_llm\_completion\_tokens           | counter   | Number of completion tokens. Only exported for AI request types. Introduced in API7 Enterprise 3.9.7 and APISIX 3.17.0.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| apisix\_llm\_latency                      | histogram | LLM request latency in milliseconds. Only exported for AI request types. Introduced in API7 Enterprise 3.9.7 and APISIX 3.17.0. The `type` label distinguishes total response latency (`total`) from time to first token (`ttft`). Introduced in API7 Enterprise 3.9.14 and 3.10.1, and APISIX 3.18.0. Queries that omit `type` match both observations; select `type="total"` to keep the previous total-latency meaning. A successful streaming response with a recorded, nonzero time to first token records one `total` sample and one `ttft` sample. Error responses and streams that end before recording the first token do not record either sample. |
| apisix\_llm\_active\_connections          | gauge     | Number of in-flight LLM upstream requests. Only exported for AI request types. Introduced in API7 Enterprise 3.9.7 and APISIX 3.17.0.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| apisix\_llm\_prompt\_tokens\_dist         | histogram | Distribution of prompt tokens per request. Only exported for AI request types. Introduced in API7 Enterprise 3.9.14 and 3.10.1, and APISIX 3.18.0.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| apisix\_llm\_completion\_tokens\_dist     | histogram | Distribution of completion tokens per request. Only exported for AI request types. Introduced in API7 Enterprise 3.9.14 and 3.10.1, and APISIX 3.18.0.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| apisix\_ai\_cache\_hits\_total            | counter   | Number of requests served from AI Cache, separated by exact or semantic cache layer. Introduced in API7 Enterprise 3.9.16 and 3.10.3, and APISIX 3.18.0.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| apisix\_ai\_cache\_misses\_total          | counter   | Number of AI Cache lookups that did not return a cached response. Introduced in API7 Enterprise 3.9.16 and 3.10.3, and APISIX 3.18.0.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| apisix\_ai\_cache\_bypasses\_total        | counter   | Number of requests that bypassed AI Cache lookup. Introduced in API7 Enterprise 3.9.16 and 3.10.3, and APISIX 3.18.0.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| apisix\_ai\_cache\_embedding\_latency     | histogram | Latency in milliseconds of embedding-provider calls made by the semantic cache layer. Introduced in API7 Enterprise 3.9.16 and 3.10.3, and APISIX 3.18.0.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |

note

LLM metrics (`apisix_llm_prompt_tokens`, `apisix_llm_completion_tokens`, `apisix_llm_latency`, `apisix_llm_prompt_tokens_dist`, and `apisix_llm_completion_tokens_dist`) are only exported when the request is processed by an AI plugin (such as [AI Proxy](https://docs.api7.ai/hub/ai-proxy.md)). Routes without AI plugins do not produce these metrics. `apisix_llm_active_connections` is managed directly by AI plugins and is also only present for AI-enabled routes.

The gauge increases when an LLM upstream attempt starts and decreases once in the request log phase. For a single attempt, it tracks active LLM upstream requests. With `ai-proxy-multi` fallback retries, each retry increments a new per-instance series, and the request is decremented only once using the final instance labels. A failed instance series can therefore stay above the true active count until the metric expires or the storage is reset.

To reduce high-cardinality labels on LLM metrics, use `disabled_labels` in [plugin metadata](https://docs.api7.ai/hub/prometheus/configuration.md#plugin-metadata) to selectively disable labels such as `consumer` or `node`.

The prompt- and completion-token histograms use configurable buckets. See the [static plugin attributes](https://docs.api7.ai/hub/prometheus/configuration.md#static-configurations) for their defaults and configuration.

## LLM Token Accounting[​](#llm-token-accounting "Direct link to LLM Token Accounting")

The prompt-token metrics include provider cache usage in the following releases:

| Provider           | Prompt total                                                           | Introduced in                                       |
| ------------------ | ---------------------------------------------------------------------- | --------------------------------------------------- |
| Anthropic Messages | `input_tokens + cache_read_input_tokens + cache_creation_input_tokens` | APISIX 3.20.0 and API7 Enterprise 3.9.22 and 3.10.9 |
| Bedrock Converse   | `inputTokens + cacheReadInputTokens + cacheWriteInputTokens`           | APISIX 3.20.0 and API7 Enterprise 3.10.9            |

API7 Enterprise 3.9.23 uses only `inputTokens` for the Bedrock prompt total.

## Labels[​](#labels "Direct link to Labels")

[Labels](https://prometheus.io/docs/practices/naming/#labels) are attributes of metrics that are used to differentiate metrics.

For example, the `apisix_http_status` metric can be labeled with `route` information to identify which route the HTTP status originates from.

The following are labels for a non-exhaustive list of APISIX metrics and their descriptions.

### Labels for `apisix_http_status`[​](#labels-for-apisix_http_status "Direct link to labels-for-apisix_http_status")

The following labels are used to differentiate `apisix_http_status` metrics.

| Name                | Description                                                                                                                                                                                                                                                                                              |
| ------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| code                | HTTP response code returned to the client after plugins and proxying.                                                                                                                                                                                                                                    |
| route               | ID of the route that the HTTP status originates from when `prefer_name` is `false` (default), and name of the route when `prefer_name` to `true`. Default to an empty string if a request does not match any route.                                                                                      |
| route\_id           | Available only in API7 Enterprise. ID of the route that the HTTP status originates from regardless of the `prefer_name` setting.                                                                                                                                                                         |
| matched\_uri        | URI of the route that matches the request. Default to an empty string if a request does not match any route.                                                                                                                                                                                             |
| matched\_host       | Host of the route that matches the request. Default to an empty string if a request does not match any route, or host is not configured on the route.                                                                                                                                                    |
| service             | ID of the service that the HTTP status originates from when `prefer_name` is `false` (default), and name of the service when `prefer_name` is `true`. Empty when the matched route does not belong to a service.                                                                                         |
| service\_id         | Available only in API7 Enterprise. ID of the service that the HTTP status originates from regardless of the `prefer_name` setting.                                                                                                                                                                       |
| consumer            | Name of the consumer associated with a request. Default to an empty string if no consumer is associated with the request.                                                                                                                                                                                |
| node                | IP address of the upstream node. For AI routes, this is the LLM instance name from `ai-proxy` or `ai-proxy-multi`.                                                                                                                                                                                       |
| gateway\_group\_id  | ID of the gateway group that the HTTP status originates from. Available only in API7 Enterprise.                                                                                                                                                                                                         |
| instance\_id        | ID of the gateway instance that the HTTP status originates from. Available only in API7 Enterprise.                                                                                                                                                                                                      |
| portal\_id          | Developer portal ID. Available only in API7 Enterprise.                                                                                                                                                                                                                                                  |
| api\_product\_id    | Product ID that the HTTP status originates from. Available only in API7 Enterprise.                                                                                                                                                                                                                      |
| request\_type       | Request type associated with the HTTP status: `traditional_http`, `websocket`, `ai_chat`, or `ai_stream`. A request answered with `101 Switching Protocols` is `websocket`, introduced in API7 Enterprise 3.9.21 and 3.10.7, and in APISIX 3.19.0.                                                       |
| request\_llm\_model | Model name sent in the client request. Empty for ordinary HTTP traffic.                                                                                                                                                                                                                                  |
| llm\_model          | Model the gateway actually targets. If the AI instance configures a model, that value is used; otherwise the client's requested model is used. Empty for ordinary HTTP traffic.                                                                                                                          |
| response\_source    | Source of the HTTP response: `apisix` (generated by APISIX, such as plugin rejections or route-not-found), `nginx` (NGINX proxy errors such as connection refused or upstream timeout), or `upstream` (real response from the upstream service). Introduced in API7 Enterprise 3.9.10 and APISIX 3.17.0. |
| mcp\_request\_type  | MCP request type, such as `tools/list` or `tools/call`. Empty for non-MCP requests. Introduced in API7 Enterprise 3.9.14.                                                                                                                                                                                |
| mcp\_tool\_name     | MCP tool name for `tools/call` requests. Empty for other requests. Introduced in API7 Enterprise 3.9.14.                                                                                                                                                                                                 |

The `response_source` label is mandatory on `apisix_http_status` and can add up to three series per existing status-label combination. Update PromQL joins, recording rules, alerts, and dashboards that match the complete label set, and review the resulting cardinality before rollout.

### Labels for `apisix_bandwidth`[​](#labels-for-apisix_bandwidth "Direct link to labels-for-apisix_bandwidth")

The following labels are used to differentiate `apisix_bandwidth` metrics.

| Name                | Description                                                                                                                                                                                                                                      |
| ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| type                | Type of traffic, `egress` or `ingress`.                                                                                                                                                                                                          |
| route               | ID of the route that bandwidth corresponds to when `prefer_name` is `false` (default), and name of the route when `prefer_name` to `true`. Default to an empty string if a request does not match any route.                                     |
| route\_id           | Available only in API7 Enterprise. ID of the route that bandwidth corresponds to regardless of the `prefer_name` setting.                                                                                                                        |
| service             | ID of the service that bandwidth corresponds to when `prefer_name` is `false` (default), and name of the service when `prefer_name` is `true`. Empty when the matched route does not belong to a service.                                        |
| service\_id         | Available only in API7 Enterprise. ID of the service that bandwidth corresponds to regardless of the `prefer_name` setting.                                                                                                                      |
| consumer            | Name of the consumer associated with a request. Default to an empty string if no consumer is associated with the request.                                                                                                                        |
| node                | IP address of the upstream node. For AI routes, this is the LLM instance name from `ai-proxy` or `ai-proxy-multi`.                                                                                                                               |
| gateway\_group\_id  | Available only in API7 Enterprise. ID of the gateway group that bandwidth corresponds to.                                                                                                                                                        |
| instance\_id        | Available only in API7 Enterprise. ID of the gateway instance that bandwidth corresponds to.                                                                                                                                                     |
| portal\_id          | Available only in API7 Enterprise. Developer portal ID.                                                                                                                                                                                          |
| api\_product\_id    | Available only in API7 Enterprise. Product ID that bandwidth corresponds to.                                                                                                                                                                     |
| request\_type       | Request type that bandwidth corresponds to: `traditional_http`, `websocket`, `ai_chat`, or `ai_stream`. A request answered with `101 Switching Protocols` is `websocket`, introduced in API7 Enterprise 3.9.21 and 3.10.7, and in APISIX 3.19.0. |
| request\_llm\_model | Model name sent in the client request. Empty for ordinary HTTP traffic.                                                                                                                                                                          |
| llm\_model          | Model the gateway actually targets. If the AI instance configures a model, that value is used; otherwise the client's requested model is used. Empty for ordinary HTTP traffic.                                                                  |
| mcp\_request\_type  | MCP request type, such as `tools/list` or `tools/call`. Empty for non-MCP requests. Introduced in API7 Enterprise 3.9.14 and 3.10.1.                                                                                                             |
| mcp\_tool\_name     | MCP tool name for `tools/call` requests. Empty for other requests. Introduced in API7 Enterprise 3.9.14 and 3.10.1.                                                                                                                              |

### Labels for `apisix_http_latency`[​](#labels-for-apisix_http_latency "Direct link to labels-for-apisix_http_latency")

The following labels are used to differentiate `apisix_http_latency` metrics.

| Name                | Description                                                                                                                                                                                                                                     |
| ------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| type                | Type of latencies. See [latency types](#latency-types) for details.                                                                                                                                                                             |
| route               | ID of the route that latencies correspond to when `prefer_name` is `false` (default), and name of the route when `prefer_name` to `true`. Default to an empty string if a request does not match any route.                                     |
| route\_id           | Available only in API7 Enterprise. ID of the route that latencies correspond to regardless of the `prefer_name` setting.                                                                                                                        |
| service             | ID of the service that latencies correspond to when `prefer_name` is `false` (default), and name of the service when `prefer_name` is `true`. Empty when the matched route does not belong to a service.                                        |
| service\_id         | Available only in API7 Enterprise. ID of the service that latencies correspond to regardless of the `prefer_name` setting.                                                                                                                      |
| consumer            | Name of the consumer associated with latencies. Default to an empty string if no consumer is associated with the request.                                                                                                                       |
| node                | IP address of the upstream node associated with latencies. For AI routes, this is the LLM instance name from `ai-proxy` or `ai-proxy-multi`.                                                                                                    |
| gateway\_group\_id  | Available only in API7 Enterprise. ID of the gateway group that latencies correspond to.                                                                                                                                                        |
| instance\_id        | Available only in API7 Enterprise. ID of the gateway instance that latencies correspond to.                                                                                                                                                     |
| portal\_id          | Available only in API7 Enterprise. Developer portal ID.                                                                                                                                                                                         |
| api\_product\_id    | Available only in API7 Enterprise. Product ID that latencies correspond to.                                                                                                                                                                     |
| request\_type       | Request type that latencies correspond to: `traditional_http`, `websocket`, `ai_chat`, or `ai_stream`. A request answered with `101 Switching Protocols` is `websocket`, introduced in API7 Enterprise 3.9.21 and 3.10.7, and in APISIX 3.19.0. |
| request\_llm\_model | Model name sent in the client request. Empty for ordinary HTTP traffic.                                                                                                                                                                         |
| llm\_model          | Model the gateway actually targets. If the AI instance configures a model, that value is used; otherwise the client's requested model is used. Empty for ordinary HTTP traffic.                                                                 |
| mcp\_request\_type  | MCP request type, such as `tools/list` or `tools/call`. Empty for non-MCP requests. Introduced in API7 Enterprise 3.9.14 and 3.10.1.                                                                                                            |
| mcp\_tool\_name     | MCP tool name for `tools/call` requests. Empty for other requests. Introduced in API7 Enterprise 3.9.14 and 3.10.1.                                                                                                                             |

#### Latency Types[​](#latency-types "Direct link to Latency Types")

`apisix_http_latency` can be labeled with one of the three types:

* `request` represents the time elapsed between the first byte was read from the client and the log write after the last byte was sent to the client.

* `upstream` represents the time elapsed waiting on responses from the upstream service.

* `apisix` represents the difference between the `request` latency and `upstream` latency.

For a WebSocket session, `request` latency measures how long the upgraded connection remains open, not only the initial handshake. Routes configured with `enable_websocket` also record the session duration as `upstream` latency. Routes that use the `ws` or `wss` upstream scheme do not emit a separate `upstream` latency sample, so their `apisix` latency covers the full session.

The `request_type="websocket"` label was introduced in API7 Enterprise 3.9.21 and 3.10.7, and in APISIX 3.19.0. You can leave these sessions out of latency queries with `request_type!="websocket"`.

In other words, the APISIX latency is not only attributed to the Lua processing. It should be understood as follows:

```
APISIX latency

  = downstream request time - upstream response time

  = downstream traffic latency + NGINX latency
```

### Labels for `apisix_upstream_status`[​](#labels-for-apisix_upstream_status "Direct link to labels-for-apisix_upstream_status")

The following labels are used to differentiate `apisix_upstream_status` metrics.

| Name | Description                                                                                                                    |
| ---- | ------------------------------------------------------------------------------------------------------------------------------ |
| name | Resource ID corresponding to the upstream configured with health checks, such as `/apisix/routes/1` and `/apisix/upstreams/1`. |
| ip   | IP address of the upstream node.                                                                                               |
| port | Port number of the node.                                                                                                       |

### Labels for Stream Metrics[​](#labels-for-stream-metrics "Direct link to Labels for Stream Metrics")

All four stream metrics (`apisix_stream_connection_total`, `apisix_stream_active_connections`, `apisix_stream_status`, and `apisix_stream_bandwidth`) also carry the following labels. They were introduced in APISIX 3.20.0 and API7 Enterprise 3.9.22 and 3.10.9:

| Name         | Description                                                                                                                                                                       |
| ------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `service`    | Service of the matched stream route. Holds the service ID, or the service name when `prefer_name` is `true`. Empty when the session does not reach a stream route with a service. |
| `service_id` | ID of the service of the matched stream route. Empty when the session does not reach a stream route with a service.                                                               |

Active connections and bandwidth are split per service while a session is still open. Each service seen on a listening address takes a slot in the stream metrics zone. Once the zone is full, the active connections and bandwidth of a new service are counted only in the total of its listening address, with empty `service` and `service_id` labels.

APISIX counts a service from its first stream session, including a session that ends before the next `refresh_interval`.

The additional labels change the identity of existing stream series. To preserve an aggregate that ignores the new service dimensions, use a query such as `sum without (service, service_id) (apisix_stream_connection_total)` and apply the same aggregation to the other stream metrics as needed.

The `apisix_stream_active_connections` metric uses the stream listening address as a label:

| Name          | Description                                                                                 |
| ------------- | ------------------------------------------------------------------------------------------- |
| `listen_addr` | Address on which APISIX accepted the TCP connection or UDP session, such as `0.0.0.0:9100`. |

The `apisix_stream_status` counter records each stream session when it ends:

| Name          | Description                                                                                                                                                                                                                                                                                                                          |
| ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `code`        | How the session ended. `200` is a normal close. `400` is a client-side problem, such as a reset or invalid data from the client. `403` is an access-rule rejection. `500` is an internal error. `502` is an upstream or transport problem, such as a connect failure, reset, or idle timeout. `503` is a connection-limit rejection. |
| `listen_addr` | Address on which APISIX accepted the session.                                                                                                                                                                                                                                                                                        |
| `node`        | Selected upstream address in `IP:port` form. Empty when the session ends before APISIX selects an upstream.                                                                                                                                                                                                                          |

NGINX reports Stream `$status` as 200 for some failures that happen after the upstream connection is established. This metric uses the session termination reason to distinguish a clean close from a later timeout or reset. `200` is also used for worker shutdown and when no recognized reason is recorded.

The `apisix_stream_bandwidth` counter updates while a session remains open:

| Name          | Description                                   |
| ------------- | --------------------------------------------- |
| `listen_addr` | Address on which APISIX accepted the session. |
| `type`        | Traffic direction: `ingress` or `egress`.     |
| `side`        | Connection side: `downstream` or `upstream`.  |

### Labels for NGINX Process Metrics[​](#labels-for-nginx-process-metrics "Direct link to Labels for NGINX Process Metrics")

Five metrics help diagnose resource pressure in individual NGINX processes: `apisix_nginx_timers`, `apisix_nginx_timers_max`, `apisix_nginx_process_connections`, `apisix_nginx_process_connections_max`, and `apisix_nginx_lua_memory_bytes`. Every process publishes its own values at the configured Prometheus `refresh_interval`. The process that serves the metrics exposition refreshes its values before returning the response.

| Label       | Description                                                                                                                                                           |
| ----------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `process`   | Process that owns the value: `worker-<id>` for a worker or `privileged-agent` for the privileged agent. Used by all process metrics except `apisix_nginx_timers_max`. |
| `subsystem` | Lua VM that owns the timers or memory: `http` or `stream`. Used by `apisix_nginx_timers` and `apisix_nginx_lua_memory_bytes`.                                         |
| `state`     | Timer state: `running` or `pending`. Used by `apisix_nginx_timers` and `apisix_nginx_timers_max`.                                                                     |

The timer maximums come from `nginx_config.max_running_timers` and `nginx_config.max_pending_timers`, including NGINX defaults when those fields are omitted. Connection usage includes listening sockets, client connections, and outgoing connections opened from Lua. The maximum is `worker_connections` for a worker and the privileged agent's connection-slot limit for that process.

### Labels for `apisix_llm_latency`[​](#labels-for-apisix_llm_latency "Direct link to labels-for-apisix_llm_latency")

The label set differs between APISIX and API7 Enterprise. API7 Enterprise adds route and service display labels, as well as dimensions for its gateway and API product resources.

| Name                | Product                    | Description                                                                                                                                                                                                                                                                                                                   |
| ------------------- | -------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `type`              | APISIX and API7 Enterprise | LLM latency type: `total` for full response latency, or `ttft` for time to first token on streaming requests. Introduced in API7 Enterprise 3.9.14 and 3.10.1, and APISIX 3.18.0. Dashboards, alerts, and recording rules that previously treated every `apisix_llm_latency` sample as total latency must add `type="total"`. |
| `route`             | API7 Enterprise            | Route ID when `prefer_name` is `false` (default), or route name when `prefer_name` is `true`. Falls back to the route ID when the route has no name, and is empty when the request does not match a route.                                                                                                                    |
| `route_id`          | APISIX and API7 Enterprise | In APISIX, the route ID by default. When `prefer_name` is `true`, APISIX uses the route name when available and otherwise keeps the ID. In API7 Enterprise, this label always contains the route ID. Empty when the request does not match a route.                                                                           |
| `service`           | API7 Enterprise            | Service ID when `prefer_name` is `false` (default), or service name when `prefer_name` is `true`. Falls back to the service ID when the service has no name, and is empty when the route does not belong to a service.                                                                                                        |
| `service_id`        | APISIX and API7 Enterprise | In APISIX, the service ID by default. When `prefer_name` is `true`, APISIX uses the service name when available and otherwise keeps the ID. In API7 Enterprise, this label always contains the service ID. Empty when the route does not belong to a service.                                                                 |
| `consumer`          | APISIX and API7 Enterprise | Consumer name. Empty when no consumer is associated with the request.                                                                                                                                                                                                                                                         |
| `node`              | APISIX and API7 Enterprise | Name of the selected LLM instance from `ai-proxy` or `ai-proxy-multi`, not the upstream IP.                                                                                                                                                                                                                                   |
| `gateway_group_id`  | API7 Enterprise            | Gateway group ID.                                                                                                                                                                                                                                                                                                             |
| `instance_id`       | API7 Enterprise            | Gateway instance ID.                                                                                                                                                                                                                                                                                                          |
| `portal_id`         | API7 Enterprise            | Developer portal ID.                                                                                                                                                                                                                                                                                                          |
| `api_product_id`    | API7 Enterprise            | API product ID.                                                                                                                                                                                                                                                                                                               |
| `request_type`      | APISIX and API7 Enterprise | AI request type, such as `ai_chat` or `ai_stream`.                                                                                                                                                                                                                                                                            |
| `request_llm_model` | APISIX and API7 Enterprise | Model name sent in the client request.                                                                                                                                                                                                                                                                                        |
| `llm_model`         | APISIX and API7 Enterprise | Model the gateway actually targets. If the AI instance configures a model, that value is used; otherwise the client's requested model is used.                                                                                                                                                                                |

### Labels for LLM Token Metrics[​](#labels-for-llm-token-metrics "Direct link to Labels for LLM Token Metrics")

The prompt- and completion-token counters and distribution histograms use different label sets by product.

| Name                | Product                    | Description                                                                                                                                                                                                                                                   |
| ------------------- | -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `route`             | API7 Enterprise            | Route ID when `prefer_name` is `false` (default), or route name when `prefer_name` is `true`. Falls back to the route ID when the route has no name, and is empty when the request does not match a route.                                                    |
| `route_id`          | APISIX and API7 Enterprise | In APISIX, the route ID by default. When `prefer_name` is `true`, APISIX uses the route name when available and otherwise keeps the ID. In API7 Enterprise, this label always contains the route ID. Empty when the request does not match a route.           |
| `matched_uri`       | API7 Enterprise            | URI of the matched route, or an empty string when the request does not match a route.                                                                                                                                                                         |
| `matched_host`      | API7 Enterprise            | Host of the matched route, or an empty string when the request does not match a route or the route has no host.                                                                                                                                               |
| `service`           | API7 Enterprise            | Service ID when `prefer_name` is `false` (default), or service name when `prefer_name` is `true`. Falls back to the service ID when the service has no name, and is empty when the route does not belong to a service.                                        |
| `service_id`        | APISIX and API7 Enterprise | In APISIX, the service ID by default. When `prefer_name` is `true`, APISIX uses the service name when available and otherwise keeps the ID. In API7 Enterprise, this label always contains the service ID. Empty when the route does not belong to a service. |
| `consumer`          | APISIX and API7 Enterprise | Consumer name, or an empty string when no consumer is associated with the request.                                                                                                                                                                            |
| `node`              | APISIX and API7 Enterprise | Name of the selected LLM instance from `ai-proxy` or `ai-proxy-multi`, not the upstream IP.                                                                                                                                                                   |
| `gateway_group_id`  | API7 Enterprise            | Gateway group ID.                                                                                                                                                                                                                                             |
| `instance_id`       | API7 Enterprise            | Gateway instance ID.                                                                                                                                                                                                                                          |
| `portal_id`         | API7 Enterprise            | Developer portal ID.                                                                                                                                                                                                                                          |
| `api_product_id`    | API7 Enterprise            | API product ID.                                                                                                                                                                                                                                               |
| `request_type`      | APISIX and API7 Enterprise | AI request type, such as `ai_chat` or `ai_stream`.                                                                                                                                                                                                            |
| `request_llm_model` | APISIX and API7 Enterprise | Model name sent in the client request. Introduced in API7 Enterprise 3.9.7 and APISIX 3.17.0.                                                                                                                                                                 |
| `llm_model`         | APISIX and API7 Enterprise | Model the gateway actually targets. If the AI instance configures a model, that value is used; otherwise the client's requested model is used.                                                                                                                |

### Labels for `apisix_llm_active_connections`[​](#labels-for-apisix_llm_active_connections "Direct link to labels-for-apisix_llm_active_connections")

The active-connection gauge records route and service names independently of `prefer_name`.

| Name                | Product                    | Description                                                                                                     |
| ------------------- | -------------------------- | --------------------------------------------------------------------------------------------------------------- |
| `route`             | APISIX and API7 Enterprise | Route name, or an empty string when the request does not match a named route.                                   |
| `route_id`          | APISIX and API7 Enterprise | Route ID, or an empty string when the request does not match a route.                                           |
| `matched_uri`       | APISIX and API7 Enterprise | URI of the matched route, or an empty string when the request does not match a route.                           |
| `matched_host`      | APISIX and API7 Enterprise | Host of the matched route, or an empty string when the request does not match a route or the route has no host. |
| `service`           | APISIX and API7 Enterprise | Service name, or an empty string when the route does not belong to a named service.                             |
| `service_id`        | APISIX and API7 Enterprise | Service ID, or an empty string when the route does not belong to a service.                                     |
| `consumer`          | APISIX and API7 Enterprise | Consumer name, or an empty string when no consumer is associated with the request.                              |
| `node`              | APISIX and API7 Enterprise | Name of the selected LLM instance from `ai-proxy` or `ai-proxy-multi`, not the upstream IP.                     |
| `gateway_group_id`  | API7 Enterprise            | Gateway group ID.                                                                                               |
| `instance_id`       | API7 Enterprise            | Gateway instance ID.                                                                                            |
| `portal_id`         | API7 Enterprise            | Developer portal ID.                                                                                            |
| `api_product_id`    | API7 Enterprise            | API product ID.                                                                                                 |
| `request_type`      | APISIX and API7 Enterprise | AI request type, such as `ai_chat` or `ai_stream`.                                                              |
| `request_llm_model` | APISIX and API7 Enterprise | Model name sent in the client request.                                                                          |
| `llm_model`         | APISIX and API7 Enterprise | Model the gateway actually targets.                                                                             |

The `request_llm_model` and `llm_model` values originate from client and provider data. Their values are limited to 128 bytes. Introduced in API7 Enterprise 3.9.14 and 3.10.1, and APISIX 3.18.0. Disable the labels when per-model series are not required.

### Labels for AI Cache Metrics[​](#labels-for-ai-cache-metrics "Direct link to Labels for AI Cache Metrics")

The four AI Cache metrics share the following labels. Only `apisix_ai_cache_hits_total` includes `layer`.

| Name                | Description                                                                                                                                                                                                    |
| ------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `layer`             | Cache layer that served a hit: `exact` or `semantic`. This structural label cannot be disabled.                                                                                                                |
| `route`             | Route name, or an empty string if the route has no name.                                                                                                                                                       |
| `route_id`          | Route ID.                                                                                                                                                                                                      |
| `service`           | Service name, or an empty string if the route does not reference a service.                                                                                                                                    |
| `service_id`        | Service ID, or an empty string if the route does not reference a service.                                                                                                                                      |
| `consumer`          | Consumer name, or an empty string when the request has no consumer.                                                                                                                                            |
| `node`              | Name of the selected LLM instance from `ai-proxy` or `ai-proxy-multi`, not the upstream IP.                                                                                                                    |
| `request_type`      | Request type, such as `ai_chat` or `ai_stream`.                                                                                                                                                                |
| `request_llm_model` | Model name sent in the client request.                                                                                                                                                                         |
| `llm_model`         | Model the gateway actually targets. If the AI instance configures a model, that value is used; otherwise the client's requested model is used. Empty on cache hits, which are served without reaching the LLM. |

API7 Enterprise also exports `matched_uri`, `matched_host`, `gateway_group_id`, `instance_id`, `portal_id`, and `api_product_id` for these metrics.

## Examples[​](#examples "Direct link to Examples")

The examples below demonstrate how you can work with the `prometheus` plugin for different scenarios.

### Get APISIX Metrics[​](#get-apisix-metrics "Direct link to Get APISIX Metrics")

The following example demonstrates how you can get metrics from APISIX.

The default Prometheus metrics endpoint and other Prometheus related configurations can be found in the [static configuration](https://docs.api7.ai/hub/prometheus/configuration.md#static-configurations). If you would like to customize these configurations, see [configuration files](https://docs.api7.ai/apisix/reference/configuration-files.md#configyaml-and-configyamlexample).

If you deploy the gateway in a containerized environment and would like to access the Prometheus metrics endpoint externally, update the Prometheus export address in the gateway static configuration:

* Host or Docker
* Kubernetes (Helm)

Add or update this section in the gateway configuration file:

config.yaml

```
plugin_attr:

  prometheus:

    export_addr:

      ip: 0.0.0.0
```

Reload the gateway for changes to take effect.

For the APISIX Helm chart, enable Prometheus in the chart values. This renders `plugin_attr.prometheus.export_addr.ip` as `0.0.0.0`:

values.yaml

```
apisix:

  prometheus:

    enabled: true
```

For the API7 Gateway Helm chart, update the Prometheus plugin attributes:

values.yaml

```
pluginAttrs:

  prometheus:

    export_addr:

      ip: 0.0.0.0

      port: 9091
```

Then apply the values file with the chart used for this gateway release:

```
helm upgrade <release-name> <chart-name> -n <namespace> -f values.yaml
```

Send a request to the APISIX Prometheus metrics endpoint:

```
curl "http://127.0.0.1:9091/apisix/prometheus/metrics"
```

You should see an output similar to the following:

```
# HELP apisix_bandwidth Total bandwidth in bytes consumed per service in Apisix

# TYPE apisix_bandwidth counter

apisix_bandwidth{type="egress",route="",service="",consumer="",node=""} 8417

apisix_bandwidth{type="egress",route="1",service="",consumer="",node="127.0.0.1"} 1420

apisix_bandwidth{type="egress",route="2",service="",consumer="",node="127.0.0.1"} 1420

apisix_bandwidth{type="ingress",route="",service="",consumer="",node=""} 189

apisix_bandwidth{type="ingress",route="1",service="",consumer="",node="127.0.0.1"} 332

apisix_bandwidth{type="ingress",route="2",service="",consumer="",node="127.0.0.1"} 332

# HELP apisix_etcd_modify_indexes Etcd modify index for APISIX keys

# TYPE apisix_etcd_modify_indexes gauge

apisix_etcd_modify_indexes{key="consumers"} 0

apisix_etcd_modify_indexes{key="global_rules"} 0

...
```

### Reduce Metric Cardinality by Disabling Labels[​](#reduce-metric-cardinality-by-disabling-labels "Direct link to Reduce Metric Cardinality by Disabling Labels")

Plugin metadata can collapse selected label values to an empty string, reducing the number of time series while preserving the metric's label schema. APISIX and API7 Enterprise use different metadata keys for the HTTP status and latency metrics.

The accepted labels also differ where API7 Enterprise exports additional dimensions:

| Metric metadata key                                                                                     | APISIX labels that can be disabled                                                                                                                | API7 Enterprise differences                                                                                                                        |
| ------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- |
| `http_status`                                                                                           | `route`, `matched_uri`, `matched_host`, `service`, `consumer`, `node`, `request_type`, `request_llm_model`, `llm_model`, `response_source`        | Use the key `status`. Adds `route_id`, `service_id`, `mcp_request_type`, and `mcp_tool_name`, but does not allow `response_source` to be disabled. |
| `http_latency`                                                                                          | `route`, `service`, `consumer`, `node`, `request_type`, `request_llm_model`, `llm_model`                                                          | Use the key `latency`. Adds `route_id`, `service_id`, `mcp_request_type`, and `mcp_tool_name`.                                                     |
| `bandwidth`                                                                                             | `route`, `service`, `consumer`, `node`, `request_type`, `request_llm_model`, `llm_model`                                                          | Adds `route_id`, `service_id`, `mcp_request_type`, and `mcp_tool_name`.                                                                            |
| `llm_latency`                                                                                           | `route_id`, `service_id`, `consumer`, `node`, `request_type`, `request_llm_model`, `llm_model`                                                    | Also allows `route` and `service`.                                                                                                                 |
| `llm_prompt_tokens`, `llm_completion_tokens`, `llm_prompt_tokens_dist`, `llm_completion_tokens_dist`    | `route_id`, `service_id`, `consumer`, `node`, `request_type`, `request_llm_model`, `llm_model`                                                    | Also allows `route`, `matched_uri`, `matched_host`, and `service`.                                                                                 |
| `llm_active_connections`                                                                                | `route`, `route_id`, `matched_uri`, `matched_host`, `service`, `service_id`, `consumer`, `node`, `request_type`, `request_llm_model`, `llm_model` | Same labels.                                                                                                                                       |
| `ai_cache_hits_total`, `ai_cache_misses_total`, `ai_cache_bypasses_total`, `ai_cache_embedding_latency` | `route`, `route_id`, `service`, `service_id`, `consumer`, `node`, `request_type`, `request_llm_model`, `llm_model`                                | Also allows `matched_uri` and `matched_host`.                                                                                                      |

Structural labels are excluded from this table because they cannot be disabled.

In API7 Enterprise, the `stream_status` metadata key disables the `node` label on `apisix_stream_status`. Its `code` and `listen_addr` labels are structural. Introduced in API7 Enterprise 3.9.19 on the 3.9 line and 3.10.6 on the 3.10 line.

In APISIX, disable the `node` label on HTTP status and latency metrics:

```
curl "http://127.0.0.1:9180/apisix/admin/plugin_metadata/prometheus" -X PUT \

  -H "X-API-KEY: ${ADMIN_API_KEY}" \

  -d '{

    "disabled_labels": {

      "http_status": ["node"],

      "http_latency": ["node"]

    }

  }'
```

For API7 Enterprise, use `status` and `latency` instead:

```
{

  "disabled_labels": {

    "status": ["node"],

    "latency": ["node"]

  }

}
```

Send a request through a route with the plugin enabled, then fetch the metrics endpoint. The affected series should keep the `node` label with an empty value:

```
apisix_http_status{code="200",route="1",matched_uri="/get",matched_host="",service="",consumer="",node="",request_type="traditional_http",request_llm_model="",llm_model="",response_source="upstream"} 1
```

The schema rejects structural labels that distinguish different measurements, including `code` for HTTP status, `type` for HTTP latency, bandwidth, and LLM latency, and `layer` for AI Cache hits. The exact optional-label set differs between APISIX and API7 Enterprise; use the [plugin metadata reference](https://docs.api7.ai/hub/prometheus/configuration.md#plugin-metadata) for the gateway you are configuring.

### Expose APISIX Metrics on Public API Endpoint[​](#expose-apisix-metrics-on-public-api-endpoint "Direct link to Expose APISIX Metrics on Public API Endpoint")

The following example demonstrates how you can disable the Prometheus export server that, by default, exposes an endpoint on port `9091`, and expose APISIX Prometheus metrics on a new public API endpoint on port `9080`, which APISIX uses to listen to other client requests.

caution

If a large quantity of metrics are being collected, the plugin could take up a significant amount of CPU resources for metric computations and negatively impact the processing of regular requests.

To address this issue, APISIX uses [privileged agent](https://github.com/openresty/lua-resty-core/blob/master/lib/ngx/process.md#enable_privileged_agent) and offloads the metric computations to a separate process. This optimization applies automatically if you use the metric endpoint configured in the configuration files, as demonstrated [above](#get-apisix-metrics). If you expose the metric endpoint with the `public-api` plugin, you will not benefit from this optimization.

To expose metrics through `public-api`, first disable the default Prometheus export server:

* Host or Docker
* Kubernetes (Helm)

Add or update this section in the gateway configuration file:

config.yaml

```
plugin_attr:

  prometheus:

    enable_export_server: false
```

Reload the gateway for changes to take effect.

For Helm deployments, update the chart values that render `plugin_attr.prometheus`. Keep the rest of your values file unchanged.

For the APISIX Helm chart, set the following values:

values.yaml

```
apisix:

  pluginAttrs:

    prometheus:

      enable_export_server: false
```

For the API7 Gateway Helm chart, set the following values:

values.yaml

```
pluginAttrs:

  prometheus:

    enable_export_server: false
```

Then apply the values file with the chart used for this gateway release:

```
helm upgrade <release-name> <chart-name> -n <namespace> -f values.yaml
```

Next, create a route with [`public-api`](https://docs.api7.ai/hub/public-api.md) plugin and expose a public API endpoint for APISIX metrics:

* Admin API
* ADC
* Ingress Controller

```
curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \

  -H "X-API-KEY: ${ADMIN_API_KEY}" \

  -d '{

    "id": "prometheus-metrics",

    "uri": "/prometheus_metrics",

    "plugins": {

      "public-api": {

        "uri": "/apisix/prometheus/metrics"

      }

    }

  }'
```

adc.yaml

```
services:

  - name: prometheus-metrics-service

    routes:

      - uris:

          - /prometheus_metrics

        name: prometheus-metrics

        plugins:

          public-api:

            uri: /apisix/prometheus/metrics

    upstream:

      type: roundrobin

      nodes:

        - host: httpbin.org

          port: 80

          weight: 1
```

Preview the changes and confirm that they only affect resources for this guide:

```
adc diff -f adc.yaml \

  --include-resource-type service \

  --label-selector docs-example=prometheus
```

Synchronize the configuration to the gateway:

```
adc sync -f adc.yaml \

  --include-resource-type service \

  --label-selector docs-example=prometheus
```

* Gateway API
* APISIX CRD

prometheus-ic.yaml

```
apiVersion: v1

kind: Service

metadata:

  name: httpbin-external-domain

  namespace: aic

spec:

  type: ExternalName

  externalName: httpbin.org

  ports:

    - name: http

      port: 80

      protocol: TCP

---

apiVersion: apisix.apache.org/v1alpha1

kind: PluginConfig

metadata:

  namespace: aic

  name: prometheus-public-api-config

spec:

  plugins:

    - name: public-api

      config:

        uri: /apisix/prometheus/metrics

---

apiVersion: gateway.networking.k8s.io/v1

kind: HTTPRoute

metadata:

  namespace: aic

  name: prometheus-metrics-route

spec:

  parentRefs:

    - name: apisix

  rules:

    - matches:

        - path:

            type: Exact

            value: /prometheus_metrics

      filters:

        - type: ExtensionRef

          extensionRef:

            group: apisix.apache.org

            kind: PluginConfig

            name: prometheus-public-api-config

      backendRefs:

        - name: httpbin-external-domain

          port: 80
```

prometheus-ic.yaml

```
apiVersion: apisix.apache.org/v2

kind: ApisixRoute

metadata:

  namespace: aic

  name: prometheus-metrics-route

spec:

  ingressClassName: apisix

  http:

    - name: prometheus-metrics-route

      match:

        paths:

          - /prometheus_metrics

      plugins:

      - name: public-api

        enable: true

        config:

          uri: /apisix/prometheus/metrics
```

Apply the configuration to your cluster:

```
kubectl apply -f prometheus-ic.yaml
```

For this metrics route, `public-api` serves the response before a backend is contacted. The ADC and Gateway API tabs include a conventional backend. Verify that the response contains metrics; a backend response does not prove that the metrics handler ran.

Send a request to the new metrics endpoint to verify:

```
curl "http://127.0.0.1:9080/prometheus_metrics"
```

You should see an output similar to the following:

```
# HELP apisix_http_requests_total The total number of client requests since APISIX started

# TYPE apisix_http_requests_total gauge

apisix_http_requests_total 1

# HELP apisix_nginx_http_current_connections Number of HTTP connections

# TYPE apisix_nginx_http_current_connections gauge

apisix_nginx_http_current_connections{state="accepted"} 1

apisix_nginx_http_current_connections{state="active"} 1

apisix_nginx_http_current_connections{state="handled"} 1

apisix_nginx_http_current_connections{state="reading"} 0

apisix_nginx_http_current_connections{state="waiting"} 0

apisix_nginx_http_current_connections{state="writing"} 1

...
```

### Integrate APISIX with Prometheus and Grafana[​](#integrate-apisix-with-prometheus-and-grafana "Direct link to Integrate APISIX with Prometheus and Grafana")

To learn about how to collect APISIX metrics with Prometheus and visualize them in Grafana, see [how-to guide](https://docs.api7.ai/apisix/how-to-guide/observability/monitor-apisix-with-prometheus.md).

### Monitor Upstream Health Statuses[​](#monitor-upstream-health-statuses "Direct link to Monitor Upstream Health Statuses")

The following example demonstrates how to monitor the health status of upstream nodes.

Create a route with the `prometheus` plugin and configure upstream active health checks:

* Admin API
* ADC
* Ingress Controller

```
curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \

  -H "X-API-KEY: ${ADMIN_API_KEY}" \

  -d '{

    "id": "prometheus-route",

    "uri": "/get",

    "plugins": {

      "prometheus": {}

    },

    "upstream": {

      "type": "roundrobin",

      "nodes": {

        "httpbin.org:80": 1,

        "127.0.0.1:20001": 1

      },

      "checks": {

        "active": {

          "timeout": 5,

          "http_path": "/status/200",

          "healthy": {

            "interval": 2,

            "successes": 1

          },

          "unhealthy": {

            "interval": 1,

            "http_failures": 2

          }

        },

        "passive": {

          "healthy": {

            "http_statuses": [200, 201],

            "successes": 3

          },

          "unhealthy": {

            "http_statuses": [500],

            "http_failures": 3,

            "tcp_failures": 3

          }

        }

      }

    }

  }'
```

adc.yaml

```
services:

  - name: httpbin

    routes:

      - uris:

          - /get

        name: prometheus-route

        plugins:

          prometheus: {}

    upstream:

      type: roundrobin

      nodes:

        - host: httpbin.org

          port: 80

          weight: 1

        - host: 127.0.0.1

          port: 20001

          weight: 1

      checks:

        active:

          timeout: 5

          http_path: /status/200

          healthy:

            interval: 2

            successes: 1

          unhealthy:

            interval: 1

            http_failures: 2

        passive:

          healthy:

            http_statuses:

              - 200

              - 201

            successes: 3

          unhealthy:

            http_statuses:

              - 500

            http_failures: 3

            tcp_failures: 3
```

Preview the changes and confirm that they only affect resources for this guide:

```
adc diff -f adc.yaml \

  --include-resource-type service \

  --label-selector docs-example=prometheus
```

Synchronize the configuration to the gateway:

```
adc sync -f adc.yaml \

  --include-resource-type service \

  --label-selector docs-example=prometheus
```

* Gateway API
* APISIX CRD

prometheus-health-ic.yaml

```
apiVersion: v1

kind: Service

metadata:

  namespace: aic

  name: healthy-httpbin

spec:

  type: ExternalName

  externalName: httpbin.org

---

apiVersion: v1

kind: Service

metadata:

  namespace: aic

  name: unhealthy-httpbin

spec:

  type: ExternalName

  externalName: example.com

---

apiVersion: apisix.apache.org/v1alpha1

kind: BackendTrafficPolicy

metadata:

  namespace: aic

  name: healthy-httpbin-health

spec:

  targetRefs:

    - group: ""

      kind: Service

      name: healthy-httpbin

  healthCheck:

    active:

      type: http

      httpPath: /status/200

      timeout: 5s

      healthy:

        interval: 2s

        successes: 1

      unhealthy:

        interval: 1s

        httpFailures: 2

    passive:

      type: http

      healthy:

        httpCodes:

          - 200

          - 201

        successes: 3

      unhealthy:

        httpCodes:

          - 500

        httpFailures: 3

        tcpFailures: 3

---

apiVersion: apisix.apache.org/v1alpha1

kind: BackendTrafficPolicy

metadata:

  namespace: aic

  name: unhealthy-httpbin-health

spec:

  targetRefs:

    - group: ""

      kind: Service

      name: unhealthy-httpbin

  healthCheck:

    active:

      type: http

      httpPath: /status/200

      timeout: 5s

      healthy:

        interval: 2s

        successes: 1

      unhealthy:

        interval: 1s

        httpFailures: 2

    passive:

      type: http

      healthy:

        httpCodes:

          - 200

          - 201

        successes: 3

      unhealthy:

        httpCodes:

          - 500

        httpFailures: 3

        tcpFailures: 3

---

apiVersion: apisix.apache.org/v1alpha1

kind: PluginConfig

metadata:

  namespace: aic

  name: prometheus-plugin-config

spec:

  plugins:

    - name: prometheus

      config: {}

---

apiVersion: gateway.networking.k8s.io/v1

kind: HTTPRoute

metadata:

  namespace: aic

  name: prometheus-route

spec:

  parentRefs:

    - name: apisix

  rules:

    - matches:

        - path:

            type: Exact

            value: /status/200

      filters:

        - type: ExtensionRef

          extensionRef:

            group: apisix.apache.org

            kind: PluginConfig

            name: prometheus-plugin-config

      backendRefs:

        - name: healthy-httpbin

          port: 80

          weight: 1

        - name: unhealthy-httpbin

          port: 80

          weight: 1
```

prometheus-health-ic.yaml

```
apiVersion: apisix.apache.org/v2

kind: ApisixUpstream

metadata:

  namespace: aic

  name: httpbin-external-domain

spec:

  ingressClassName: apisix

  externalNodes:

  - type: Service

    name: healthy-httpbin

    port: 80

  - type: Service

    name: unhealthy-httpbin

    port: 80

  healthCheck:

    active:

      type: http

      httpPath: /status/200

      timeout: 5

      healthy:

        interval: 2s

        successes: 1

      unhealthy:

        interval: 1s

        httpFailures: 2

    passive:

      type: http

      healthy:

        httpCodes:

          - 200

          - 201

        successes: 3

      unhealthy:

        httpCodes:

          - 500

        httpFailures: 3

        tcpFailures: 3

---

apiVersion: v1

kind: Service

metadata:

  namespace: aic

  name: healthy-httpbin

spec:

  type: ExternalName

  externalName: httpbin.org

---

apiVersion: v1

kind: Service

metadata:

  namespace: aic

  name: unhealthy-httpbin

spec:

  type: ExternalName

  externalName: example.com

---

apiVersion: apisix.apache.org/v2

kind: ApisixRoute

metadata:

  namespace: aic

  name: prometheus-route

spec:

  ingressClassName: apisix

  http:

    - name: prometheus-route

      match:

        paths:

          - /get

      upstreams:

      - name: httpbin-external-domain

      plugins:

      - name: prometheus

        enable: true

        config: {}
```

Apply the configuration to your cluster:

```
kubectl apply -f prometheus-health-ic.yaml
```

Send a request to the APISIX Prometheus metrics endpoint:

```
curl "http://127.0.0.1:9091/apisix/prometheus/metrics"
```

You should see an output similar to the following:

```
# HELP apisix_upstream_status upstream status from health check

# TYPE apisix_upstream_status gauge

apisix_upstream_status{name="/upstreams/<id>",ip="<healthy-node-ip>",port="80"} 1

apisix_upstream_status{name="/upstreams/<id>",ip="<unhealthy-node-ip>",port="80"} 0
```

In that sample output, one upstream node is healthy and another upstream node is unhealthy.

To learn more about how to configure active and passive health checks, see [health checks](https://docs.api7.ai/apisix/how-to-guide/traffic-management/health-check.md).

### Add Extra Labels for Metrics[​](#add-extra-labels-for-metrics "Direct link to Add Extra Labels for Metrics")

The following example demonstrates how to add additional labels to metrics and use [built-in variables](https://docs.api7.ai/apisix/reference/built-in-variables.md) in label values.

Currently, extra labels are supported on:

* `apisix_http_status`
* `apisix_http_latency`
* `apisix_bandwidth`
* all `apisix_llm_*` metrics listed above
* all four `apisix_ai_cache_*` metrics

Both APISIX and API7 Gateway apply extra labels the same way.

Add extra labels to the Prometheus static configuration:

* Host or Docker
* Kubernetes (Helm)

Add or update this section in the gateway configuration file:

config.yaml

```
plugin_attr:

  prometheus:                                # Plugin: prometheus

    metrics:                                 # Create extra labels from built-in variables.

      http_status:

        extra_labels:                        # Set the extra labels for http_status metrics.

          - upstream_addr: $upstream_addr    # Add an extra upstream_addr label with value being the NGINX variable $upstream_addr.

          - route_name: $route_name          # Add an extra route_name label with value being the APISIX variable $route_name.
```

Reload the gateway for changes to take effect.

For Helm deployments, update the chart values that render `plugin_attr.prometheus.metrics`. Keep the rest of your values file unchanged.

For the APISIX Helm chart, set the following values:

values.yaml

```
apisix:

  pluginAttrs:

    prometheus:

      metrics:

        http_status:

          extra_labels:

            - upstream_addr: $upstream_addr

            - route_name: $route_name
```

For the API7 Gateway Helm chart, set the following values:

values.yaml

```
pluginAttrs:

  prometheus:

    metrics:

      http_status:

        extra_labels:

          - upstream_addr: $upstream_addr

          - route_name: $route_name
```

Then apply the values file with the chart used for this gateway release:

```
helm upgrade <release-name> <chart-name> -n <namespace> -f values.yaml
```

Note that if you define a variable in the label value but it does not correspond to any existing [built-in variables](https://docs.api7.ai/apisix/reference/built-in-variables.md), the label value will default to an empty string.

Create a route with the `prometheus` plugin:

* Admin API
* ADC
* Ingress Controller

```
curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \

  -H "X-API-KEY: ${ADMIN_API_KEY}" \

  -d '{

    "id": "prometheus-route",

    "uri": "/get",

    "name": "extra-label",

    "plugins": {

      "prometheus": {}

    },

    "upstream": {

      "nodes": {

        "httpbin.org:80": 1

      }

    }

  }'
```

adc.yaml

```
services:

  - name: httpbin

    routes:

      - uris:

          - /get

        name: extra-label

        plugins:

          prometheus: {}

    upstream:

      type: roundrobin

      nodes:

        - host: httpbin.org

          port: 80

          weight: 1
```

Preview the changes and confirm that they only affect resources for this guide:

```
adc diff -f adc.yaml \

  --include-resource-type service \

  --label-selector docs-example=prometheus
```

Synchronize the configuration to the gateway:

```
adc sync -f adc.yaml \

  --include-resource-type service \

  --label-selector docs-example=prometheus
```

* Gateway API
* APISIX CRD

prometheus-labels-ic.yaml

```
apiVersion: v1

kind: Service

metadata:

  namespace: aic

  name: httpbin-external-domain

spec:

  type: ExternalName

  externalName: httpbin.org

---

apiVersion: apisix.apache.org/v1alpha1

kind: PluginConfig

metadata:

  namespace: aic

  name: prometheus-plugin-config

spec:

  plugins:

    - name: prometheus

      config: {}

---

apiVersion: gateway.networking.k8s.io/v1

kind: HTTPRoute

metadata:

  namespace: aic

  name: prometheus-route

spec:

  parentRefs:

    - name: apisix

  hostnames:

    - "prometheus.example.com"

  rules:

    - matches:

        - path:

            type: Exact

            value: /get

      filters:

        - type: ExtensionRef

          extensionRef:

            group: apisix.apache.org

            kind: PluginConfig

            name: prometheus-plugin-config

      backendRefs:

        - name: httpbin-external-domain

          port: 80
```

prometheus-labels-ic.yaml

```
apiVersion: apisix.apache.org/v2

kind: ApisixUpstream

metadata:

  namespace: aic

  name: httpbin-external-domain

spec:

  ingressClassName: apisix

  externalNodes:

  - type: Domain

    name: httpbin.org

---

apiVersion: apisix.apache.org/v2

kind: ApisixRoute

metadata:

  namespace: aic

  name: prometheus-route

spec:

  ingressClassName: apisix

  http:

    - name: prometheus-route

      match:

        hosts:

          - "prometheus.example.com"

        paths:

          - /get

      upstreams:

      - name: httpbin-external-domain

      plugins:

      - name: prometheus

        enable: true

        config: {}
```

Apply the configuration to your cluster:

```
kubectl apply -f prometheus-labels-ic.yaml
```

Send a request to the route to verify:

```
curl -i "http://127.0.0.1:9080/get"
```

You should see an `HTTP/1.1 200 OK` response.

Send a request to the APISIX Prometheus metrics endpoint:

```
curl "http://127.0.0.1:9091/apisix/prometheus/metrics"
```

You should see an output similar to the following:

```
# HELP apisix_http_status HTTP status codes per service in APISIX

# TYPE apisix_http_status counter

apisix_http_status{code="200",route="1",matched_uri="/get",matched_host="",service="",consumer="",node="54.237.103.220",request_type="traditional_http",request_llm_model="",llm_model="",response_source="upstream",upstream_addr="54.237.103.220:80",route_name="extra-label"} 1
```

### Monitor TCP/UDP Traffic with Prometheus[​](#monitor-tcpudp-traffic-with-prometheus "Direct link to Monitor TCP/UDP Traffic with Prometheus")

The following example demonstrates how to collect TCP/UDP traffic metrics in APISIX.

To collect TCP/UDP metrics, enable stream proxy and add `prometheus` to the existing stream plugin list. Preserve the other stream plugins used by the deployment; the host/Docker example below shows the minimal list for this walkthrough.

* Host or Docker
* Kubernetes (Helm)

Add or update this section in the gateway configuration file:

config.yaml

```
apisix:

  proxy_mode: http&stream   # Enable both L4 & L7 proxies

  stream_proxy:             # Configure L4 proxy

    tcp:

      - 9100                # Set TCP proxy listening port

    udp:

      - 9200                # Set UDP proxy listening port



stream_plugins:

  - prometheus              # Enable prometheus for stream proxy
```

Reload the gateway for changes to take effect.

For the APISIX Helm chart, configure stream listeners and include the complete stream plugin list you want the gateway to load. The following example keeps the default stream plugins and adds `prometheus`:

values.yaml

```
service:

  stream:

    enabled: true

    tcp:

      - 9100

    udp:

      - 9200

apisix:

  stream_plugins:

    - ip-restriction

    - limit-conn

    - mqtt-proxy

    - prometheus

    - syslog
```

For the API7 Gateway Helm chart, set the following values:

values.yaml

```
gateway:

  stream:

    enabled: true

    tcp:

      - addr: 9100

    udp:

      - addr: 9200
```

Then apply the values file with the chart used for this gateway release:

```
helm upgrade <release-name> <chart-name> -n <namespace> -f values.yaml
```

In API7 Enterprise, the control plane delivers the stream plugin list. The values above are sufficient, and the `stream_plugins` entry shown for the other deployment forms is not required.

Create a [stream route](https://docs.api7.ai/apisix/key-concepts/stream-routes.md) with the `prometheus` plugin:

```
curl "http://127.0.0.1:9180/apisix/admin/stream_routes" -X PUT \

  -H "X-API-KEY: ${ADMIN_API_KEY}" \

  -d '{

    "id": "prometheus-route",

    "plugins": {

      "prometheus":{}

    },

    "upstream": {

      "type": "roundrobin",

      "nodes": {

        "httpbin.org:80": 1

      }

    }

  }'
```

Send a request to the stream route to verify:

```
curl -i "http://127.0.0.1:9100"
```

You should see an `HTTP/1.1 200 OK` response.

Send a request to the APISIX Prometheus metrics endpoint:

```
curl "http://127.0.0.1:9091/apisix/prometheus/metrics"
```

You should see an output similar to the following:

```
# HELP apisix_stream_connection_total Total number of connections handled per stream route in APISIX

# TYPE apisix_stream_connection_total counter

apisix_stream_connection_total{route="prometheus-route",service="",service_id=""} 1
```

APISIX also exports the termination status. When APISIX-Runtime provides the stream-metrics module, the scrape includes active connections and bandwidth:

```
# HELP apisix_stream_active_connections Number of stream sessions currently being proxied per listening address and service

# TYPE apisix_stream_active_connections gauge

apisix_stream_active_connections{listen_addr="0.0.0.0:9100",service="",service_id=""} 0

# HELP apisix_stream_status Stream sessions per termination status in APISIX

# TYPE apisix_stream_status counter

apisix_stream_status{code="200",listen_addr="0.0.0.0:9100",service="",service_id="",node="54.237.103.220:80"} 1

# HELP apisix_stream_bandwidth Total bandwidth in bytes proxied by the stream subsystem in APISIX

# TYPE apisix_stream_bandwidth counter

apisix_stream_bandwidth{listen_addr="0.0.0.0:9100",service="",service_id="",type="ingress",side="downstream"} 78

apisix_stream_bandwidth{listen_addr="0.0.0.0:9100",service="",service_id="",type="egress",side="downstream"} 219

apisix_stream_bandwidth{listen_addr="0.0.0.0:9100",service="",service_id="",type="egress",side="upstream"} 78

apisix_stream_bandwidth{listen_addr="0.0.0.0:9100",service="",service_id="",type="ingress",side="upstream"} 219
```

The route in this example does not belong to a service, so `service` and `service_id` are empty. The exact upstream address and byte counts depend on the request. The active-connections gauge is `0` above because the request completed before the scrape; scrape while a connection remains open to observe a positive value.

The active-connection and bandwidth metrics use a shared memory zone that defaults to `1m` and holds about 750 slots. Each stream listening address takes one slot, and each service seen on that address takes one more. Per-service slots were introduced in APISIX 3.20.0 and API7 Enterprise 3.9.22 and 3.10.9.

Increase the zone when a gateway exposes many stream listening addresses or serves many stream services:

config.yaml

```
nginx_config:

  stream:

    metrics_zone_size: 2m
```

Reload APISIX after changing the zone size. On a runtime without the stream-metrics module, APISIX continues to export the connection-total and status metrics but does not publish active-connection or bandwidth metrics.
