Skip to main content
API7.ai Logo

Monitor APISIX Metrics with Prometheus

Prometheus is a popular systems monitoring and alerting toolkit. It collects and stores multi-dimensional time series data like metrics with key-value paired labels.

APISIX offers the capability to expose a significant number of metrics to Prometheus with low latency, allowing for continuous monitoring and diagnostics.

This guide will show you how to enable the prometheus plugin to integrate with Prometheus and Grafana services, where APISIX HTTP metrics are collected and visualized.


APISIX Prometheus Grafana flow diagram

Prerequisite(s)​

Enable Prometheus Plugin​

Enable the prometheus plugin globally. Alternatively, you can enable the plugin on a route.

curl "http://127.0.0.1:9180/apisix/admin/global_rules/prometheus" -X PUT \
  -H "Content-Type: application/json" \
  -d '{
    "plugins": {
      "prometheus": {}
    }
  }'

APISIX gathers internal runtime metrics and exposes them through port 9091 and path /apisix/prometheus/metrics by default. The port and the path can be customized in the configuration file.

Send a request to the route /apisix/prometheus/metrics to fetch metrics from APISIX:

curl "http://127.0.0.1:9091/apisix/prometheus/metrics"

You should see a list of metrics similar to the following:

# HELP apisix_etcd_modify_indexes Etcd modify index for APISIX keys
# TYPE apisix_etcd_modify_indexes gauge
apisix_etcd_modify_indexes{key="consumers"} 0
apisix_etcd_modify_indexes{key="global_rules"} 0
apisix_etcd_modify_indexes{key="max_modify_index"} 16
apisix_etcd_modify_indexes{key="prev_index"} 15
apisix_etcd_modify_indexes{key="protos"} 0
apisix_etcd_modify_indexes{key="routes"} 16
apisix_etcd_modify_indexes{key="services"} 0
apisix_etcd_modify_indexes{key="ssls"} 0
apisix_etcd_modify_indexes{key="stream_routes"} 0
apisix_etcd_modify_indexes{key="upstreams"} 0
apisix_etcd_modify_indexes{key="x_etcd_index"} 16
# HELP apisix_etcd_reachable Config server etcd reachable from APISIX, 0 is unreachable
# TYPE apisix_etcd_reachable gauge
apisix_etcd_reachable 1
...
# HELP apisix_http_status HTTP status codes per service in APISIX
# TYPE apisix_http_status counter
...

Configure Prometheus​

In Prometheus, targets are the endpoints that Prometheus scrapes for metrics. You can configure the APISIX metrics endpoint as a target in Prometheus to collect metrics from it.

Create a configuration file prometheus.yml:

cat > prometheus.yml <<'EOF'
global:
  scrape_interval: 15s

scrape_configs:
  - job_name: apisix
    metrics_path: /apisix/prometheus/metrics
    static_configs:
      - targets:
          - apisix-quickstart:9091
EOF

Create a data volume and start Prometheus. Port 9092 on the host maps to the Prometheus web interface on port 9090 in the container:

docker volume create apisix-quickstart-prometheus-data

docker run -d --name apisix-quickstart-prometheus \
  --network apisix-quickstart-net \
  -p 9092:9090 \
  -v "$(pwd)/prometheus.yml:/etc/prometheus/prometheus.yml:ro" \
  -v apisix-quickstart-prometheus-data:/prometheus \
  prom/prometheus:v3.7.3

Open the Prometheus targets page. The apisix target should have the UP state.

Prometheus targets page showing the APISIX target in the UP state

Configure Grafana​

Grafana can visualize metrics stored in Prometheus.

Create a data volume and start Grafana on the same Docker network:

docker volume create apisix-quickstart-grafana-data

docker run -d --name apisix-quickstart-grafana \
  --network apisix-quickstart-net \
  -p 3000:3000 \
  -v apisix-quickstart-grafana-data:/var/lib/grafana \
  grafana/grafana:13.1.0

Open Grafana, sign in with the default username and password admin, and change the password when prompted.

Go to Connections > Data sources, add a Prometheus data source, and set its URL to http://apisix-quickstart-prometheus:9090. This URL uses the Prometheus container name and container port because Grafana makes the request from inside the Docker network. Select Save & test and verify that Grafana can query Prometheus.

Grafana Prometheus data source configured with the Prometheus container URL

Generate gateway traffic and wait for Prometheus to complete a scrape:

for i in {1..100}; do
  curl -sS "http://127.0.0.1:9080/ip" > /dev/null
done

sleep 15

Download the dashboard maintained in the Apache APISIX source tree for this version:

curl -fL "https://raw.githubusercontent.com/apache/apisix/3.18.0/docs/assets/other/json/apisix-grafana-dashboard.json" \
  -o apisix-grafana-dashboard.json
note

The Grafana catalog page for dashboard 11719 lists compatibility with APISIX 2.10.x and depends on legacy Grafana panels. Use the versioned dashboard JSON above for the metrics exposed in this guide.

In Grafana, go to Dashboards > New > Import, upload apisix-grafana-dashboard.json, select the Prometheus data source, and select Import.

Grafana dashboard import page with the Apache APISIX dashboard JSON loaded

The dashboard should populate with the requests generated above and show APISIX connection, request, latency, bandwidth, etcd, and shared dictionary metrics.

If you already scrape apisix_llm_latency, update PromQL, recording rules, and Grafana panels that omit the type label. That histogram now distinguishes type="total" (full response latency) from type="ttft" (time to first token on streaming requests). A selector without type matches both. Use type="total" when you need the previous total-latency meaning:

histogram_quantile(0.99, sum by (le, route_id) (rate(apisix_llm_latency_bucket{type="total"}[5m])))

Each streaming request records one total observation and one ttft observation, so sample counts and time-series cardinality increase for streaming traffic.

Apache APISIX Grafana dashboard populated with live gateway metrics

Reduce Metric Cardinality​

High-cardinality labels, such as an upstream node address that changes frequently, can create many time series. Configure the prometheus plugin metadata to replace selected label values with an empty string while keeping the labels in the metric schema:

curl "http://127.0.0.1:9180/apisix/admin/plugin_metadata/prometheus" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d '{
    "disabled_labels": {
      "http_status": ["node"],
      "http_latency": ["node"]
    }
  }'

Generate traffic and fetch the metrics endpoint again. The affected series should retain the node label with an empty value:

apisix_http_status{code="200",route="1",matched_uri="/ip",matched_host="",service="",consumer="",node="",request_type="traditional_http",request_llm_model="",llm_model="",response_source="upstream"} 100

APISIX rejects attempts to disable structural labels that distinguish different measurements, including code for HTTP status and type for latency and bandwidth. See the prometheus plugin metadata reference for the labels supported by each metric.

Monitor TCP and UDP Traffic​

Enable the stream proxy and add prometheus to the existing stream_plugins list. The following minimal list is sufficient for this example; preserve any other stream plugins your deployment uses:

config.yaml
apisix:
  proxy_mode: http&stream
  stream_proxy:
    tcp:
      - 9100

stream_plugins:
  - prometheus

Reload APISIX, then create a stream route:

curl "http://127.0.0.1:9180/apisix/admin/stream_routes" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d '{
    "id": "prometheus-stream-route",
    "plugins": {
      "prometheus": {}
    },
    "upstream": {
      "type": "roundrobin",
      "nodes": {
        "httpbin.org:80": 1
      }
    }
  }'

Send traffic through the stream listener and scrape the metrics endpoint:

curl -i "http://127.0.0.1:9100"
curl "http://127.0.0.1:9091/apisix/prometheus/metrics" | grep apisix_stream

The output should include the connection total, active sessions, termination status, and bandwidth counters:

apisix_stream_connection_total{route="prometheus-stream-route"} 1
apisix_stream_active_connections{listen_addr="0.0.0.0:9100"} 0
apisix_stream_status{code="200",listen_addr="0.0.0.0:9100",node="54.237.103.220:80"} 1
apisix_stream_bandwidth{listen_addr="0.0.0.0:9100",type="ingress",side="downstream"} 78

The upstream address and byte count depend on the request. The active-session value is 0 after the request closes. Active-session and bandwidth metrics require APISIX-Runtime; APISIX continues to export the connection-total and status metrics when that runtime module is unavailable. The runtime-backed metrics use the shared memory size configured by nginx_config.stream.metrics_zone_size, which defaults to 1m.

Next Steps​

You have now learned how to monitor APISIX metrics with Prometheus and visualize them in Grafana. See the prometheus plugin documentation for more configuration options.