Prometheus

The prometheus plugin provides the capability to integrate APISIX with Prometheus.

After enabling the plugin, APISIX will start collecting relevant metrics, such as API requests and latencies, and exporting them in a text-based exposition format to Prometheus. You can then create event monitoring and alerting in Prometheus to monitor the health of your API gateway and APIs.

Metrics​

There are different types of metrics in Prometheus. To understand their differences, see metrics types.

The following metrics are exported by the prometheus plugin by default. See get APISIX metrics for an example. Note that some metrics, such as apisix_batch_process_entries, are not readily visible if there are no data.

Labels​

Labels are attributes of metrics that are used to differentiate metrics.

For example, the apisix_http_status metric can be labeled with route information to identify which route the HTTP status originates from.

The following are labels for a non-exhaustive list of APISIX metrics and their descriptions.

Labels for apisix_http_status​

The following labels are used to differentiate apisix_http_status metrics.

The response_source label is mandatory on apisix_http_status and can add up to three series per existing status-label combination. Update PromQL joins, recording rules, alerts, and dashboards that match the complete label set, and review the resulting cardinality before rollout.

Labels for apisix_bandwidth​

The following labels are used to differentiate apisix_bandwidth metrics.

Labels for apisix_http_latency​

The following labels are used to differentiate apisix_http_latency metrics.

Latency Types​

apisix_http_latency can be labeled with one of the three types:

• request represents the time elapsed between the first byte was read from the client and the log write after the last byte was sent to the client.

• upstream represents the time elapsed waiting on responses from the upstream service.

• apisix represents the difference between the request latency and upstream latency.

In other words, the APISIX latency is not only attributed to the Lua processing. It should be understood as follows:

APISIX latency
  = downstream request time - upstream response time
  = downstream traffic latency + NGINX latency

Labels for apisix_upstream_status​

The following labels are used to differentiate apisix_upstream_status metrics.

Labels for Stream Metrics​

The apisix_stream_active_connections metric uses the stream listening address as a label:

The apisix_stream_status counter records each stream session when it ends:

NGINX reports Stream $status as 200 for some failures that happen after the upstream connection is established. This metric uses the session termination reason to distinguish a clean close from a later timeout or reset. 200 is also used for worker shutdown and when no recognized reason is recorded.

The apisix_stream_bandwidth counter updates while a session remains open:

Labels for apisix_llm_latency​

The following labels are used to differentiate apisix_llm_latency metrics.

Labels for Other LLM Metrics​

The following labels are used to differentiate apisix_llm_prompt_tokens, apisix_llm_completion_tokens, apisix_llm_active_connections, apisix_llm_prompt_tokens_dist, and apisix_llm_completion_tokens_dist metrics.

The prompt- and completion-token counters and distribution histograms use different label sets by product. APISIX exports route_id, service_id, consumer, node, request_type, request_llm_model, and llm_model. API7 Enterprise additionally exports route, matched_uri, matched_host, and service, as well as its gateway-instance and API-product labels.

The request_llm_model and llm_model values originate from client and provider data. Their values are limited to 128 bytes. Introduced in API7 Enterprise 3.9.14 and 3.10.1, and APISIX 3.18.0. Disable the labels when per-model series are not required.

Labels for AI Cache Metrics​

The four AI Cache metrics share the following labels. Only apisix_ai_cache_hits_total includes layer.

API7 Enterprise also exports matched_uri, matched_host, and its gateway-instance and API-product labels for these metrics.

Examples​

The examples below demonstrate how you can work with the prometheus plugin for different scenarios.

Get APISIX Metrics​

The following example demonstrates how you can get metrics from APISIX.

The default Prometheus metrics endpoint and other Prometheus related configurations can be found in the static configuration. If you would like to customize these configurations, see configuration files.

If you deploy the gateway in a containerized environment and would like to access the Prometheus metrics endpoint externally, update the Prometheus export address in the gateway static configuration:

plugin_attr:
  prometheus:
    export_addr:
      ip: 0.0.0.0

apisix:
  prometheus:
    enabled: true

pluginAttrs:
  prometheus:
    export_addr:
      ip: 0.0.0.0
      port: 9091

helm upgrade <release-name> <chart-name> -n <namespace> -f values.yaml

Send a request to the APISIX Prometheus metrics endpoint:

curl "http://127.0.0.1:9091/apisix/prometheus/metrics"

You should see an output similar to the following:

# HELP apisix_bandwidth Total bandwidth in bytes consumed per service in Apisix
# TYPE apisix_bandwidth counter
apisix_bandwidth{type="egress",route="",service="",consumer="",node=""} 8417
apisix_bandwidth{type="egress",route="1",service="",consumer="",node="127.0.0.1"} 1420
apisix_bandwidth{type="egress",route="2",service="",consumer="",node="127.0.0.1"} 1420
apisix_bandwidth{type="ingress",route="",service="",consumer="",node=""} 189
apisix_bandwidth{type="ingress",route="1",service="",consumer="",node="127.0.0.1"} 332
apisix_bandwidth{type="ingress",route="2",service="",consumer="",node="127.0.0.1"} 332
# HELP apisix_etcd_modify_indexes Etcd modify index for APISIX keys
# TYPE apisix_etcd_modify_indexes gauge
apisix_etcd_modify_indexes{key="consumers"} 0
apisix_etcd_modify_indexes{key="global_rules"} 0
...

Reduce Metric Cardinality by Disabling Labels​

Plugin metadata can collapse selected label values to an empty string, reducing the number of time series while preserving the metric's label schema. APISIX and API7 Enterprise use different metadata keys for the HTTP status and latency metrics.

The accepted labels also differ where API7 Enterprise exports additional dimensions:

Structural labels are excluded from this table because they cannot be disabled.

In API7 Enterprise, the stream_status metadata key disables the node label on apisix_stream_status. Its code and listen_addr labels are structural. Introduced in API7 Enterprise 3.9.19 on the 3.9 line and 3.10.6 on the 3.10 line.

In APISIX, disable the node label on HTTP status and latency metrics:

curl "http://127.0.0.1:9180/apisix/admin/plugin_metadata/prometheus" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d '{
    "disabled_labels": {
      "http_status": ["node"],
      "http_latency": ["node"]
    }
  }'

For API7 Enterprise, use status and latency instead:

{
  "disabled_labels": {
    "status": ["node"],
    "latency": ["node"]
  }
}

Send a request through a route with the plugin enabled, then fetch the metrics endpoint. The affected series should keep the node label with an empty value:

apisix_http_status{code="200",route="1",matched_uri="/get",matched_host="",service="",consumer="",node="",request_type="traditional_http",request_llm_model="",llm_model="",response_source="upstream"} 1

The schema rejects structural labels that distinguish different measurements, including code for HTTP status, type for HTTP latency, bandwidth, and LLM latency, and layer for AI Cache hits. The exact optional-label set differs between APISIX and API7 Enterprise; use the plugin metadata reference for the gateway you are configuring.

Expose APISIX Metrics on Public API Endpoint​

The following example demonstrates how you can disable the Prometheus export server that, by default, exposes an endpoint on port 9091, and expose APISIX Prometheus metrics on a new public API endpoint on port 9080, which APISIX uses to listen to other client requests.

To expose metrics through public-api, first disable the default Prometheus export server:

plugin_attr:
  prometheus:
    enable_export_server: false

apisix:
  pluginAttrs:
    prometheus:
      enable_export_server: false

pluginAttrs:
  prometheus:
    enable_export_server: false

helm upgrade <release-name> <chart-name> -n <namespace> -f values.yaml

Next, create a route with public-api plugin and expose a public API endpoint for APISIX metrics:

curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d '{
    "id": "prometheus-metrics",
    "uri": "/prometheus_metrics",
    "plugins": {
      "public-api": {
        "uri": "/apisix/prometheus/metrics"
      }
    }
  }'

routes:
  - uri: /prometheus_metrics
    name: prometheus-metrics
    plugins:
      public-api:
        uri: /apisix/prometheus/metrics

adc sync -f adc.yaml

apiVersion: apisix.apache.org/v1alpha1
kind: PluginConfig
metadata:
  namespace: aic
  name: prometheus-public-api-config
spec:
  plugins:
    - name: public-api
      config:
        uri: /apisix/prometheus/metrics
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  namespace: aic
  name: prometheus-metrics-route
spec:
  parentRefs:
    - name: apisix
  rules:
    - matches:
        - path:
            type: Exact
            value: /prometheus_metrics
      filters:
        - type: ExtensionRef
          extensionRef:
            group: apisix.apache.org
            kind: PluginConfig
            name: prometheus-public-api-config

apiVersion: apisix.apache.org/v2
kind: ApisixRoute
metadata:
  namespace: aic
  name: prometheus-metrics-route
spec:
  ingressClassName: apisix
  http:
    - name: prometheus-metrics-route
      match:
        paths:
          - /prometheus_metrics
      plugins:
      - name: public-api
        enable: true
        config:
          uri: /apisix/prometheus/metrics

kubectl apply -f prometheus-ic.yaml

Send a request to the new metrics endpoint to verify:

curl "http://127.0.0.1:9080/prometheus_metrics"

You should see an output similar to the following:

# HELP apisix_http_requests_total The total number of client requests since APISIX started
# TYPE apisix_http_requests_total gauge
apisix_http_requests_total 1
# HELP apisix_nginx_http_current_connections Number of HTTP connections
# TYPE apisix_nginx_http_current_connections gauge
apisix_nginx_http_current_connections{state="accepted"} 1
apisix_nginx_http_current_connections{state="active"} 1
apisix_nginx_http_current_connections{state="handled"} 1
apisix_nginx_http_current_connections{state="reading"} 0
apisix_nginx_http_current_connections{state="waiting"} 0
apisix_nginx_http_current_connections{state="writing"} 1
...

Integrate APISIX with Prometheus and Grafana​

To learn about how to collect APISIX metrics with Prometheus and visualize them in Grafana, see how-to guide.

Monitor Upstream Health Statuses​

The following example demonstrates how to monitor the health status of upstream nodes.

Create a route with the prometheus plugin and configure upstream active health checks:

curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d '{
    "id": "prometheus-route",
    "uri": "/get",
    "plugins": {
      "prometheus": {}
    },
    "upstream": {
      "type": "roundrobin",
      "nodes": {
        "httpbin.org:80": 1,
        "127.0.0.1:20001": 1
      },
      "checks": {
        "active": {
          "timeout": 5,
          "http_path": "/status",
          "healthy": {
            "interval": 2,
            "successes": 1
          },
          "unhealthy": {
            "interval": 1,
            "http_failures": 2
          }
        },
        "passive": {
          "healthy": {
            "http_statuses": [200, 201],
            "successes": 3
          },
          "unhealthy": {
            "http_statuses": [500],
            "http_failures": 3,
            "tcp_failures": 3
          }
        }
      }
    }
  }'

services:
  - name: httpbin
    routes:
      - uris:
          - /get
        name: prometheus-route
        plugins:
          prometheus: {}
    upstream:
      type: roundrobin
      nodes:
        - host: httpbin.org
          port: 80
          weight: 1
        - host: 127.0.0.1
          port: 20001
          weight: 1
      checks:
        active:
          timeout: 5
          http_path: /status
          healthy:
            interval: 2
            successes: 1
          unhealthy:
            interval: 1
            http_failures: 2
        passive:
          healthy:
            http_statuses:
              - 200
              - 201
            successes: 3
          unhealthy:
            http_statuses:
              - 500
            http_failures: 3
            tcp_failures: 3

adc sync -f adc.yaml

apiVersion: v1
kind: Service
metadata:
  namespace: aic
  name: healthy-httpbin
spec:
  type: ExternalName
  externalName: httpbin.org
---
apiVersion: v1
kind: Service
metadata:
  namespace: aic
  name: unhealthy-httpbin
spec:
  type: ExternalName
  externalName: example.com
---
apiVersion: apisix.apache.org/v1alpha1
kind: BackendTrafficPolicy
metadata:
  namespace: aic
  name: healthy-httpbin-health
spec:
  targetRefs:
    - group: ""
      kind: Service
      name: healthy-httpbin
  healthCheck:
    active:
      type: http
      httpPath: /status/200
      timeout: 5s
      healthy:
        interval: 2s
        successes: 1
      unhealthy:
        interval: 1s
        httpFailures: 2
    passive:
      type: http
      healthy:
        httpCodes:
          - 200
          - 201
        successes: 3
      unhealthy:
        httpCodes:
          - 500
        httpFailures: 3
        tcpFailures: 3
---
apiVersion: apisix.apache.org/v1alpha1
kind: BackendTrafficPolicy
metadata:
  namespace: aic
  name: unhealthy-httpbin-health
spec:
  targetRefs:
    - group: ""
      kind: Service
      name: unhealthy-httpbin
  healthCheck:
    active:
      type: http
      httpPath: /status/200
      timeout: 5s
      healthy:
        interval: 2s
        successes: 1
      unhealthy:
        interval: 1s
        httpFailures: 2
    passive:
      type: http
      healthy:
        httpCodes:
          - 200
          - 201
        successes: 3
      unhealthy:
        httpCodes:
          - 500
        httpFailures: 3
        tcpFailures: 3
---
apiVersion: apisix.apache.org/v1alpha1
kind: PluginConfig
metadata:
  namespace: aic
  name: prometheus-plugin-config
spec:
  plugins:
    - name: prometheus
      config: {}
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  namespace: aic
  name: prometheus-route
spec:
  parentRefs:
    - name: apisix
  rules:
    - matches:
        - path:
            type: Exact
            value: /status/200
      filters:
        - type: ExtensionRef
          extensionRef:
            group: apisix.apache.org
            kind: PluginConfig
            name: prometheus-plugin-config
      backendRefs:
        - name: healthy-httpbin
          port: 80
          weight: 1
        - name: unhealthy-httpbin
          port: 80
          weight: 1

apiVersion: apisix.apache.org/v2
kind: ApisixUpstream
metadata:
  namespace: aic
  name: httpbin-external-domain
spec:
  ingressClassName: apisix
  externalNodes:
  - type: Service
    name: healthy-httpbin
    port: 80
  - type: Service
    name: unhealthy-httpbin
    port: 80
  healthCheck:
    active:
      type: http
      httpPath: /status
      timeout: 5
      healthy:
        interval: 2s
        successes: 1
      unhealthy:
        interval: 1s
        httpFailures: 2
    passive:
      type: http
      healthy:
        httpCodes:
          - 200
          - 201
        successes: 3
      unhealthy:
        httpCodes:
          - 500
        httpFailures: 3
        tcpFailures: 3
---
apiVersion: v1
kind: Service
metadata:
  namespace: aic
  name: healthy-httpbin
spec:
  type: ExternalName
  externalName: httpbin.org
---
apiVersion: v1
kind: Service
metadata:
  namespace: aic
  name: unhealthy-httpbin
spec:
  type: ExternalName
  externalName: example.com
---
apiVersion: apisix.apache.org/v2
kind: ApisixRoute
metadata:
  namespace: aic
  name: prometheus-route
spec:
  ingressClassName: apisix
  http:
    - name: prometheus-route
      match:
        paths:
          - /get
      upstreams:
      - name: httpbin-external-domain
      plugins:
      - name: prometheus
        enable: true
        config: {}

kubectl apply -f prometheus-health-ic.yaml

Send a request to the APISIX Prometheus metrics endpoint:

curl "http://127.0.0.1:9091/apisix/prometheus/metrics"

You should see an output similar to the following:

# HELP apisix_upstream_status upstream status from health check
# TYPE apisix_upstream_status gauge
apisix_upstream_status{name="/upstreams/<id>",ip="<healthy-node-ip>",port="80"} 1
apisix_upstream_status{name="/upstreams/<id>",ip="<unhealthy-node-ip>",port="80"} 0

In that sample output, one upstream node is healthy and another upstream node is unhealthy.

To learn more about how to configure active and passive health checks, see health checks.

Add Extra Labels for Metrics​

The following example demonstrates how to add additional labels to metrics and use built-in variables in label values.

Currently, extra labels are supported on:

• apisix_http_status
• apisix_http_latency
• apisix_bandwidth
• all apisix_llm_* metrics listed above
• all four apisix_ai_cache_* metrics

Both APISIX and API7 Gateway apply extra labels the same way.

Add extra labels to the Prometheus static configuration:

plugin_attr:
  prometheus:                                # Plugin: prometheus
    metrics:                                 # Create extra labels from built-in variables.
      http_status:
        extra_labels:                        # Set the extra labels for http_status metrics.
          - upstream_addr: $upstream_addr    # Add an extra upstream_addr label with value being the NGINX variable $upstream_addr.
          - route_name: $route_name          # Add an extra route_name label with value being the APISIX variable $route_name.

apisix:
  pluginAttrs:
    prometheus:
      metrics:
        http_status:
          extra_labels:
            - upstream_addr: $upstream_addr
            - route_name: $route_name

pluginAttrs:
  prometheus:
    metrics:
      http_status:
        extra_labels:
          - upstream_addr: $upstream_addr
          - route_name: $route_name

helm upgrade <release-name> <chart-name> -n <namespace> -f values.yaml

Note that if you define a variable in the label value but it does not correspond to any existing built-in variables, the label value will default to an empty string.

Create a route with the prometheus plugin:

curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d '{
    "id": "prometheus-route",
    "uri": "/get",
    "name": "extra-label",
    "plugins": {
      "prometheus": {}
    },
    "upstream": {
      "nodes": {
        "httpbin.org:80": 1
      }
    }
  }'

services:
  - name: httpbin
    routes:
      - uris:
          - /get
        name: extra-label
        plugins:
          prometheus: {}
    upstream:
      type: roundrobin
      nodes:
        - host: httpbin.org
          port: 80
          weight: 1

adc sync -f adc.yaml

apiVersion: v1
kind: Service
metadata:
  namespace: aic
  name: httpbin-external-domain
spec:
  type: ExternalName
  externalName: httpbin.org
---
apiVersion: apisix.apache.org/v1alpha1
kind: PluginConfig
metadata:
  namespace: aic
  name: prometheus-plugin-config
spec:
  plugins:
    - name: prometheus
      config: {}
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  namespace: aic
  name: prometheus-route
spec:
  parentRefs:
    - name: apisix
  hostnames:
    - "prometheus.example.com"
  rules:
    - matches:
        - path:
            type: Exact
            value: /get
      filters:
        - type: ExtensionRef
          extensionRef:
            group: apisix.apache.org
            kind: PluginConfig
            name: prometheus-plugin-config
      backendRefs:
        - name: httpbin-external-domain
          port: 80

apiVersion: apisix.apache.org/v2
kind: ApisixUpstream
metadata:
  namespace: aic
  name: httpbin-external-domain
spec:
  ingressClassName: apisix
  externalNodes:
  - type: Domain
    name: httpbin.org
---
apiVersion: apisix.apache.org/v2
kind: ApisixRoute
metadata:
  namespace: aic
  name: prometheus-route
spec:
  ingressClassName: apisix
  http:
    - name: prometheus-route
      match:
        hosts:
          - "prometheus.example.com"
        paths:
          - /get
      upstreams:
      - name: httpbin-external-domain
      plugins:
      - name: prometheus
        enable: true
        config: {}

kubectl apply -f prometheus-labels-ic.yaml

Send a request to the route to verify:

curl -i "http://127.0.0.1:9080/get"

You should see an HTTP/1.1 200 OK response.

Send a request to the APISIX Prometheus metrics endpoint:

curl "http://127.0.0.1:9091/apisix/prometheus/metrics"

You should see an output similar to the following:

# HELP apisix_http_status HTTP status codes per service in APISIX
# TYPE apisix_http_status counter
apisix_http_status{code="200",route="1",matched_uri="/get",matched_host="",service="",consumer="",node="54.237.103.220",request_type="traditional_http",request_llm_model="",llm_model="",response_source="upstream",upstream_addr="54.237.103.220:80",route_name="extra-label"} 1

Monitor TCP/UDP Traffic with Prometheus​

The following example demonstrates how to collect TCP/UDP traffic metrics in APISIX.

To collect TCP/UDP metrics, enable stream proxy and add prometheus to the existing stream plugin list. Preserve the other stream plugins used by the deployment; the host/Docker example below shows the minimal list for this walkthrough.

apisix:
  proxy_mode: http&stream   # Enable both L4 & L7 proxies
  stream_proxy:             # Configure L4 proxy
    tcp:
      - 9100                # Set TCP proxy listening port
    udp:
      - 9200                # Set UDP proxy listening port

stream_plugins:
  - prometheus              # Enable prometheus for stream proxy

service:
  stream:
    enabled: true
    tcp:
      - 9100
    udp:
      - 9200
apisix:
  stream_plugins:
    - ip-restriction
    - limit-conn
    - mqtt-proxy
    - prometheus
    - syslog

gateway:
  stream:
    enabled: true
    tcp:
      - addr: 9100
    udp:
      - addr: 9200

helm upgrade <release-name> <chart-name> -n <namespace> -f values.yaml

Create a stream route with the prometheus plugin:

curl "http://127.0.0.1:9180/apisix/admin/stream_routes" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d '{
    "id": "prometheus-route",
    "plugins": {
      "prometheus":{}
    },
    "upstream": {
      "type": "roundrobin",
      "nodes": {
        "httpbin.org:80": 1
      }
    }
  }'

Send a request to the stream route to verify:

curl -i "http://127.0.0.1:9100"

You should see an HTTP/1.1 200 OK response.

Send a request to the APISIX Prometheus metrics endpoint:

curl "http://127.0.0.1:9091/apisix/prometheus/metrics"

You should see an output similar to the following:

# HELP apisix_stream_connection_total Total number of connections handled per stream route in APISIX
# TYPE apisix_stream_connection_total counter
apisix_stream_connection_total{route="prometheus-route"} 1

APISIX also exports the termination status. When APISIX-Runtime provides the stream-metrics module, the scrape includes active connections and bandwidth:

# HELP apisix_stream_active_connections Number of stream sessions currently being proxied per listening address
# TYPE apisix_stream_active_connections gauge
apisix_stream_active_connections{listen_addr="0.0.0.0:9100"} 0
# HELP apisix_stream_status Stream sessions per termination status in APISIX
# TYPE apisix_stream_status counter
apisix_stream_status{code="200",listen_addr="0.0.0.0:9100",node="54.237.103.220:80"} 1
# HELP apisix_stream_bandwidth Total bandwidth in bytes proxied by the stream subsystem in APISIX
# TYPE apisix_stream_bandwidth counter
apisix_stream_bandwidth{listen_addr="0.0.0.0:9100",type="ingress",side="downstream"} 78
apisix_stream_bandwidth{listen_addr="0.0.0.0:9100",type="egress",side="downstream"} 219
apisix_stream_bandwidth{listen_addr="0.0.0.0:9100",type="egress",side="upstream"} 78
apisix_stream_bandwidth{listen_addr="0.0.0.0:9100",type="ingress",side="upstream"} 219

The exact upstream address and byte counts depend on the request. The active-connections gauge is 0 above because the request completed before the scrape; scrape while a connection remains open to observe a positive value.

The active-connection and bandwidth metrics use a shared memory zone that defaults to 1m. Increase the zone when a gateway exposes many stream listening addresses:

nginx_config:
  stream:
    metrics_zone_size: 2m

Reload APISIX after changing the zone size. On a runtime without the stream-metrics module, APISIX continues to export the connection-total and status metrics but does not publish active-connection or bandwidth metrics.