Prometheus

The prometheus plugin provides the capability to integrate APISIX with Prometheus.

After enabling the plugin, APISIX will start collecting relevant metrics, such as API requests and latencies, and exporting them in a text-based exposition format to Prometheus. You can then create event monitoring and alerting in Prometheus to monitor the health of your API gateway and APIs.

Metrics​

There are different types of metrics in Prometheus. To understand their differences, see metrics types.

The following metrics are exported by the prometheus plugin by default. See get APISIX metrics for an example. Note that some metrics, such as apisix_batch_process_entries, are not readily visible if there are no data.

Labels​

Labels are attributes of metrics that are used to differentiate metrics.

For example, the apisix_http_status metric can be labeled with route information to identify which route the HTTP status originates from.

The following are labels for a non-exhaustive list of APISIX metrics and their descriptions.

Labels for apisix_http_status​

The following labels are used to differentiate apisix_http_status metrics.

Labels for apisix_bandwidth​

The following labels are used to differentiate apisix_bandwidth metrics.

Labels for apisix_http_latency​

The following labels are used to differentiate apisix_http_latency metrics.

Latency Types​

apisix_http_latency can be labeled with one of the three types:

• request represents the time elapsed between the first byte was read from the client and the log write after the last byte was sent to the client.

• upstream represents the time elapsed waiting on responses from the upstream service.

• apisix represents the difference between the request latency and upstream latency.

In other words, the APISIX latency is not only attributed to the Lua processing. It should be understood as follows:

APISIX latency
  = downstream request time - upstream response time
  = downstream traffic latency + NGINX latency

Labels for apisix_upstream_status​

The following labels are used to differentiate apisix_upstream_status metrics.

Examples​

The examples below demonstrate how you can work with the prometheus plugin for different scenarios.

Get APISIX Metrics​

The following example demonstrates how you can get metrics from APISIX.

The default Prometheus metrics endpoint and other Prometheus related configurations can be found in the static configuration. If you would like to customize these configuration, see configuration files.

If you deploy APISIX In a containerized environment and would like to access the Prometheus metrics endpoint externally, update the configuration file as follows and reload APISIX:

plugin_attr:
  prometheus:
    export_addr:
      ip: 0.0.0.0 

Send a request to the APISIX Prometheus metrics endpoint:

curl "http://127.0.0.1:9091/apisix/prometheus/metrics"

You should see an output similar to the following:

# HELP apisix_bandwidth Total bandwidth in bytes consumed per service in Apisix
# TYPE apisix_bandwidth counter
apisix_bandwidth{type="egress",route="",service="",consumer="",node=""} 8417
apisix_bandwidth{type="egress",route="1",service="",consumer="",node="127.0.0.1"} 1420
apisix_bandwidth{type="egress",route="2",service="",consumer="",node="127.0.0.1"} 1420
apisix_bandwidth{type="ingress",route="",service="",consumer="",node=""} 189
apisix_bandwidth{type="ingress",route="1",service="",consumer="",node="127.0.0.1"} 332
apisix_bandwidth{type="ingress",route="2",service="",consumer="",node="127.0.0.1"} 332
# HELP apisix_etcd_modify_indexes Etcd modify index for APISIX keys
# TYPE apisix_etcd_modify_indexes gauge
apisix_etcd_modify_indexes{key="consumers"} 0
apisix_etcd_modify_indexes{key="global_rules"} 0
...

Expose APISIX Metrics on Public API Endpoint​

The following example demonstrates how you can disable the Prometheus export server that, by default, exposes an endpoint on port 9091, and expose APISIX Prometheus metrics on a new public API endpoint on port 9080, which APISIX uses to listen to other client requests.

Disable the Prometheus export server in the configuration file and reload APISIX for changes to take effect:

plugin_attr:
  prometheus:
    enable_export_server: false

Next, create a route with public-api plugin and expose a public API endpoint for APISIX metrics:

curl "http://127.0.0.1:9180/apisix/admin/routes/prometheus-metrics" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d '{
    "uri": "/apisix/prometheus/metrics",
    "plugins": {
      "public-api": {}
    }
  }'

Send a request to the new metrics endpoint to verify:

curl "http://127.0.0.1:9080/apisix/prometheus/metrics"

You should see an output similar to the following:

# HELP apisix_http_requests_total The total number of client requests since APISIX started
# TYPE apisix_http_requests_total gauge
apisix_http_requests_total 1
# HELP apisix_nginx_http_current_connections Number of HTTP connections
# TYPE apisix_nginx_http_current_connections gauge
apisix_nginx_http_current_connections{state="accepted"} 1
apisix_nginx_http_current_connections{state="active"} 1
apisix_nginx_http_current_connections{state="handled"} 1
apisix_nginx_http_current_connections{state="reading"} 0
apisix_nginx_http_current_connections{state="waiting"} 0
apisix_nginx_http_current_connections{state="writing"} 1
...

Integrate APISIX with Prometheus and Grafana​

To learn about how to collect APISIX metrics with Prometheus and visualize them in Grafana, see how-to guide.

Monitor Upstream Health Statuses​

The following example demonstrates how to monitor the health status of upstream nodes.

Create a route with the prometheus plugin and configure upstream active health checks:

curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d '{
    "id": "prometheus-route",
    "uri": "/get",
    "plugins": {
      "prometheus": {}
    },
    "upstream": {
      "type": "roundrobin",
      "nodes": {
        "httpbin.org:80": 1,
        "127.0.0.1:20001": 1
      },
      "checks": {
        "active": {
          "timeout": 5,
          "http_path": "/status",
          "healthy": {
            "interval": 2,
            "successes": 1
          },
          "unhealthy": {
            "interval": 1,
            "http_failures": 2
          }
        },
        "passive": {
          "healthy": {
            "http_statuses": [200, 201],
            "successes": 3
          },
          "unhealthy": {
            "http_statuses": [500],
            "http_failures": 3,
            "tcp_failures": 3
          }
        }
      }
    }
  }'

Send a request to the APISIX Prometheus metrics endpoint:

curl "http://127.0.0.1:9091/apisix/prometheus/metrics"

You should see an output similar to the following:

# HELP apisix_upstream_status upstream status from health check
# TYPE apisix_upstream_status gauge
apisix_upstream_status{name="/apisix/routes/1",ip="54.237.103.220",port="80"} 1
apisix_upstream_status{name="/apisix/routes/1",ip="127.0.0.1",port="20001"} 0

This shows that the upstream node httpbin.org:80 is healthy and the upstream node 127.0.0.1:20001 is unhealthy.

To learn more about how to configure active and passive health checks, see health checks.

Add Extra Labels for Metrics​

The following example demonstrates how to add additional labels to metrics and use built-in variables in label values.

Currently, only the following metrics support extra labels:

• apisix_http_status
• apisix_http_latency
• apisix_bandwidth

Include the following configurations in the configuration file to add labels for metrics and reload APISIX for changes to take effect:

plugin_attr:
  prometheus:                                # Plugin: prometheus
    metrics:                                 # Create extra labels from built-in variables.
      http_status:
        extra_labels:                        # Set the extra labels for http_status metrics.
          - upstream_addr: $upstream_addr    # Add an extra upstream_addr label with value being the NGINX variable $upstream_addr.
          - route_name: $route_name          # Add an extra route_name label with value being the APISIX variable $route_name.

Note that if you define a variable in the label value but it does not correspond to any existing built-in variables, the label value will default to an empty string.

Create a route with the prometheus plugin:

curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d '{
    "id": "prometheus-route",
    "uri": "/get",
    "name": "extra-label",
    "plugins": {
      "prometheus": {}
    },
    "upstream": {
      "nodes": {
        "httpbin.org:80": 1
      }
    }
  }'

Send a request to the route to verify:

curl -i "http://127.0.0.1:9080/get"

You should see an HTTP/1.1 200 OK response.

Send a request to the APISIX Prometheus metrics endpoint:

curl "http://127.0.0.1:9091/apisix/prometheus/metrics"

You should see an output similar to the following:

# HELP apisix_http_status HTTP status codes per service in APISIX
# TYPE apisix_http_status counter
apisix_http_status{code="200",route="1",matched_uri="/get",matched_host="",service="",consumer="",node="54.237.103.220",upstream_addr="54.237.103.220:80",route_name="extra-label"} 1

Monitor TCP/UDP Traffic with Prometheus​

The following example demonstrates how to collect TCP/UDP traffic metrics in APISIX.

Include the following configurations in the configuration file to enable stream proxy and prometheus plugin for stream proxy. Reload APISIX for changes to take effect:

apisix:
  proxy_mode: http&stream   # Enable both L4 & L7 proxies
  stream_proxy:             # Configure L4 proxy
    tcp:
      - 9100                # Set TCP proxy listening port
    udp:
      - 9200                # Set UDP proxy listening port

stream_plugins:
  - prometheus              # Enable prometheus for stream proxy

Create a stream route with the prometheus plugin:

curl "http://127.0.0.1:9180/apisix/admin/stream_routes" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d '{
    "id": "prometheus-route",
    "plugins": {
      "prometheus":{}
    },
    "upstream": {
      "type": "roundrobin",
      "nodes": {
        "httpbin.org:80": 1
      }
    }
  }'

Send a request to the stream route to verify:

curl -i "http://127.0.0.1:9100"

You should see an HTTP/1.1 200 OK response.

Send a request to the APISIX Prometheus metrics endpoint:

curl "http://127.0.0.1:9091/apisix/prometheus/metrics"

You should see an output similar to the following:

# HELP apisix_stream_connection_total Total number of connections handled per stream route in APISIX
# TYPE apisix_stream_connection_total counter
apisix_stream_connection_total{route="1"} 1