Monitor APISIX Metrics with Prometheus

Prometheus is a popular systems monitoring and alerting toolkit. It collects and stores multi-dimensional time series data like metrics with key-value paired labels.

APISIX offers the capability to expose a significant number of metrics to Prometheus with low latency, allowing for continuous monitoring and diagnostics.

This guide will show you how to enable the prometheus plugin to integrate with Prometheus and Grafana services, where APISIX HTTP metrics are collected and visualized.

Prerequisite(s)

• Install Docker.
• Install cURL to send requests to the services for validation.
• Follow the Getting Started tutorial to start a new APISIX instance in Docker.

Enable Prometheus Plugin

Enable the prometheus plugin globally. Alternatively, you can enable the plugin on a route.

Admin API

curl -i "http://127.0.0.1:9180/apisix/admin/global_rules" -X PUT -d '{
  "id": "rule-for-metrics",
  "plugins": {
    "prometheus":{}
  }
}'

APISIX gathers internal runtime metrics and exposes them through port 9091 and path /apisix/prometheus/metrics by default. The port and the path can be customized in the configuration file.

ADC

adc.yaml

global_rules:
  prometheus: {}

Synchronize the configuration to APISIX:

adc sync -f adc.yaml

APISIX gathers internal runtime metrics and exposes them through port 9091 and path /apisix/prometheus/metrics by default. The port and the path can be customized in the configuration file.

Ingress Controller

Gateway API

global-prometheus.yaml

apiVersion: apisix.apache.org/v1alpha1
kind: GatewayProxy
metadata:
  name: apisix-config
spec:
  plugins:
  - name: prometheus
    enabled: true

APISIX CRD

global-prometheus.yaml

apiVersion: apisix.apache.org/v2
kind: ApisixGlobalRule
metadata:
  name: global-prometheus
spec:
  ingressClassName: apisix
  plugins:
    - name: prometheus
      enable: true

Apply the configuration to your cluster:

kubectl apply -f global-prometheus.yaml

To expose Prometheus metrics on the specified path and port, update your Helm installation:

helm upgrade apisix apisix/apisix \
  --namespace ingress-apisix \
  --set apisix.deployment.role=traditional \
  --set apisix.deployment.role_traditional.config_provider=yaml \
  --set etcd.enabled=false \
  --set ingress-controller.enabled=true \
  --set ingress-controller.config.provider.type=apisix-standalone \
  --set ingress-controller.apisix.adminService.namespace=ingress-apisix \
  --set ingress-controller.gatewayProxy.createDefault=true \
  --set apisix.prometheus.enabled=true \
  --set apisix.prometheus.path="/apisix/prometheus/metrics" \
  --set apisix.prometheus.containerPort=9091

Port forward the Prometheus metrics service port to your local machine's port:

kubectl port-forward service/apisix-prometheus-metrics 9091:9091 &

Send a request to the route /apisix/prometheus/metrics to fetch metrics from APISIX:

curl "http://127.0.0.1:9091/apisix/prometheus/metrics"

You should see a list of metrics similar to the following:

# HELP apisix_etcd_modify_indexes Etcd modify index for APISIX keys
# TYPE apisix_etcd_modify_indexes gauge
apisix_etcd_modify_indexes{key="consumers"} 0
apisix_etcd_modify_indexes{key="global_rules"} 0
apisix_etcd_modify_indexes{key="max_modify_index"} 16
apisix_etcd_modify_indexes{key="prev_index"} 15
apisix_etcd_modify_indexes{key="protos"} 0
apisix_etcd_modify_indexes{key="routes"} 16
apisix_etcd_modify_indexes{key="services"} 0
apisix_etcd_modify_indexes{key="ssls"} 0
apisix_etcd_modify_indexes{key="stream_routes"} 0
apisix_etcd_modify_indexes{key="upstreams"} 0
apisix_etcd_modify_indexes{key="x_etcd_index"} 16
# HELP apisix_etcd_reachable Config server etcd reachable from APISIX, 0 is unreachable
# TYPE apisix_etcd_reachable gauge
apisix_etcd_reachable 1
...
# HELP apisix_http_status HTTP status codes per service in APISIX
# TYPE apisix_http_status counter
apisix_http_status{code="200",route="ip",matched_uri="/ip",matched_host="",service="",consumer="",node="52.20.124.211"} 1
...

Configure Prometheus

In Prometheus, targets are the endpoints that Prometheus scrapes for metrics. You can configure the APISIX metrics endpoint as a target in Prometheus to collect metrics from it.

Docker

Create a configuration file prometheus.yml:

echo 'scrape_configs:
  - job_name: "apisix"
    scrape_interval: 15s
    metrics_path: "/apisix/prometheus/metrics"
    static_configs:
      - targets: ["apisix-quickstart:9091"]
' > prometheus.yml

Start a Prometheus instance in Docker. The exposed port is mapped to 9092 on the host because 9090 is reserved for APISIX. The local configuration file prometheus.yml is mounted to the Prometheus container.

docker run -d --name apisix-quickstart-prometheus \
  -p 9092:9090 \
  --network=apisix-quickstart-net \
  -v $(pwd)/prometheus.yml:/etc/prometheus/prometheus.yml \
  prom/prometheus:latest

Kubernetes

Create a ConfigMap:

cm-prometheus.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-config
data:
  prometheus.yml: |
    scrape_configs:
      - job_name: "apisix"
        scrape_interval: 15s
        metrics_path: "/apisix/prometheus/metrics"
        static_configs:
          - targets: ["apisix-prometheus-metrics:9091"]

Create a deployment file for Prometheus:

deployment-prometheus.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: apisix-quickstart-prometheus
spec:
  replicas: 1
  selector:
    matchLabels:
      app: prometheus
  template:
    metadata:
      labels:
        app: prometheus
    spec:
      containers:
        - name: prometheus
          image: prom/prometheus:latest
          ports:
            - containerPort: 9090
          volumeMounts:
            - name: config-volume
              mountPath: /etc/prometheus/
      volumes:
        - name: config-volume
          configMap:
            name: prometheus-config

Create a service file to expose Prometheus:

service-prometheus.yaml

apiVersion: v1
kind: Service
metadata:
  name: apisix-quickstart-prometheus
spec:
  type: ClusterIP
  selector:
    app: prometheus
  ports:
    - protocol: TCP
      port: 9090
      targetPort: 9090

Apply the configuration to your cluster:

kubectl apply -f cm-prometheus.yaml -f deployment-prometheus.yaml -f service-prometheus.yaml

Port forward the Prometheus targets page port to your local machine's port:

kubectl port-forward svc/apisix-quickstart-prometheus 9092:9090 &

You can now check if the APISIX metric endpoint state is UP on the Prometheus targets page. Prometheus will collect metrics from APISIX by scraping this endpoint.

Configure Grafana

Grafana can visualize metrics stored in Prometheus.

Docker

Start a Grafana instance on port 3000 in Docker:

docker run -d --name=apisix-quickstart-grafana \
  -p 3000:3000 \
  --network=apisix-quickstart-net \
  grafana/grafana-oss

Kubernetes

Create a deployment file for Grafana:

deployment-grafana.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: apisix-quickstart-grafana
spec:
  replicas: 1
  selector:
    matchLabels:
      app: grafana
  template:
    metadata:
      labels:
        app: grafana
    spec:
      containers:
      - name: grafana
        image: grafana/grafana-oss:latest
        ports:
        - containerPort: 3000

service-grafana.yaml

apiVersion: v1
kind: Service
metadata:
  name: apisix-quickstart-grafana
spec:
  type: ClusterIP
  selector:
    app: grafana
  ports:
  - protocol: TCP
    port: 3000
    targetPort: 3000

Apply the configuration to your cluster:

kubectl apply -f deployment-grafana.yaml -f service-grafana.yaml

Port forward the Grafana console port to your local machine's port:

kubectl port-forward svc/apisix-quickstart-grafana 3000:3000 &

Visit Grafana console and add the Prometheus instance created above to Grafana as a data source. Use http://127.0.0.1:9092 if your Prometheus instance is running in Docker, and use http://apisix-quickstart-prometheus:9090 if your Prometheus instance is deployed on Kubernetes.

The official APISIX metric dashboard is published to Grafana dashboards with ID 11719. You can then import the dashboard into Grafana with the ID.

If everything is OK, the dashboard will automatically visualize metrics in real time.

Next Steps

You have now learned how to monitor APISIX metrics with Prometheus and visualize them in Grafana. See the prometheus plugin documentation for more configuration options.