Skip to main content
API7.ai Logo
Version: 3.10.x

Use an Existing Prometheus

The Docker Compose quickstart and the api7ee3 Helm chart ship a bundled Prometheus. If you already run Prometheus, you can replace the bundled instance with your external deployment. After the migration, the Dashboard Monitoring page queries the external Prometheus.

How Gateway Metrics Reach Prometheus​

API7 Gateway metrics can reach Prometheus through two independent paths.

PathFlowUsed by
Remote writeData plane → DP Manager → Prometheus (/api/v1/write)The Dashboard Monitoring page, whose queries run against this Prometheus.
ScrapePrometheus → data plane :9091 (see Monitor Metrics)Your own Grafana dashboards or alerting stack.

The bundled Prometheus receives remote write for the Dashboard. Both paths carry the same gateway-emitted metric names and labels, including gateway_group_id and instance_id. Prometheus also adds the target labels job and instance when it scrapes a data plane. Dashboard queries aggregate with sum(...) by (...) and do not filter on job, so the Monitoring page can query metrics delivered through either path.

Do not keep both paths to the same Prometheus

If one Prometheus both scrapes the data plane and receives remote write, the same gateway metric can arrive through both paths. The scraped copy carries the additional job and instance labels, so Prometheus stores both copies as separate series and the Dashboard double-counts requests and bandwidth. Keep exactly one path per Prometheus instance.

Prerequisites​

Before migrating, make sure the external instance and the control-plane deployment configuration are available.

  • An external Prometheus instance reachable from the control plane containers or pods.
  • Access to either the Docker Compose deployment directory or the installed api7/api7ee3 Helm release and its values.

Choose a Migration Path​

Pick the path that matches how your external Prometheus already gets gateway metrics.

Compare the changes required by each path:

ConsiderationRemote writeDirect scrape
WhenExternal Prometheus can accept DP Manager remote write, or you do not want to expose the data plane metrics listenerExternal Prometheus already scrapes the data plane :9091
External Prometheus changesAdd --web.enable-remote-write-receiver; the control plane writes metrics to itNone; the scrape configuration is unchanged
Permission neededDashboard needs read access; DP Manager needs write accessRead access
Existing Grafana and alertsRemote-written series omit job and instance; update queries that filter on themNo query changes

Complete the configuration for one path, then configure authentication if needed, remove the bundled instance, apply the migration, and verify the result.

Back Up the Current Configuration​

Create a backup that can restore the original Prometheus configuration and bundled service.

cd <control-plane-directory>
BAK="/var/tmp/api7-prom-backup-$(date +%F)"
mkdir -p "$BAK"
cp -a docker-compose.yaml dashboard_conf dp_manager_conf "$BAK"/

Configure Remote Write​

DP Manager sends gateway metrics to the external Prometheus, and the Dashboard queries the same instance.

Enable the Remote-Write Receiver​

--web.enable-remote-write-receiver is a Prometheus startup flag, not a prometheus.yml setting. Add the flag and restart or recreate Prometheus for the change to take effect:

  • systemd: add --web.enable-remote-write-receiver to ExecStart, then systemctl daemon-reload && systemctl restart prometheus.
  • Docker / Docker Compose: add --web.enable-remote-write-receiver to the container's command, then recreate the container.
  • Kubernetes: add it to the Prometheus container args, then roll the pod.

Confirm that the receiver is enabled:

curl -s -o /dev/null -w '%{http_code}\n' -X POST "http://<external-prometheus>:9090/api/v1/write"
# 400 = enabled (invalid body, expected)   404 = still disabled

If the endpoint requires authentication, add the matching scheme, such as -u <user>:<password> for Basic authentication, and use the https:// URL. A 401 or 403 response indicates an authentication failure, not a disabled receiver.

Point the Dashboard and DP Manager at the External Prometheus​

Point the Dashboard and DP Manager at the external Prometheus. Keep telemetry enabled so DP Manager continues reporting metrics.

dashboard_conf/conf.yaml
prometheus:
  addr: "http://<external-prometheus>:9090"   # must be reachable from inside the dashboard container
  query_path_prefix: ""
  whitelist:
    - "/api/v1/query_range"
    - "/api/v1/query"
    # keep the rest unchanged

telemetry:
  enable: true   # keep enabled for remote write

Then point DP Manager at the same instance:

dp_manager_conf/conf.yaml
prometheus:
  addr: "http://<external-prometheus>:9090"
  remote_write_path: "/api/v1/write"

Remote-write users can skip the direct-scrape section and continue with Configure Authentication.

Keep Direct Scraping​

Before continuing, configure the data plane metrics listener to bind to an address Prometheus can reach. Its default is 127.0.0.1; publishing or exposing port 9091 alone does not change that address. See Monitor Metrics for the listener configuration.

Confirm the External Prometheus Has Gateway Metrics​

Send a few requests through a route, then check that the external Prometheus has ingested the resulting gateway metrics. This read-only request should return a result greater than 0:

curl -sG --data-urlencode 'query=count(apisix_http_status)' \
  "http://<external-prometheus>:9090/api/v1/query"

An empty result can also mean that no traffic has reached the gateway or that the prometheus plugin is not enabled as a global rule. If the count stays 0 after traffic, fix the scrape job or enable the plugin before continuing.

If the external Prometheus is authenticated, add credentials to this curl, such as -u <user>:<password>, and use the https:// URL. A 401 or 403 response indicates an authentication failure rather than missing metrics.

Point the Dashboard at the External Prometheus and Stop Remote Write​

Point the Dashboard at the external Prometheus, and disable telemetry so the control plane stops writing metrics that would duplicate your scrape.

dashboard_conf/conf.yaml
prometheus:
  addr: "http://<external-prometheus>:9090"   # must be reachable from inside the dashboard container
  query_path_prefix: ""
  whitelist:
    - "/api/v1/query_range"
    - "/api/v1/query"
    # keep the rest unchanged

telemetry:
  enable: false
Disable remote write when using direct scraping

If telemetry.enable stays true, the control plane keeps remote-writing metrics that collide with your scraped series, and the Dashboard counts them twice.

This telemetry block is a top-level key in dashboard_conf/conf.yaml and maps to dashboard_configuration.telemetry in the Helm chart. It is not the data plane api7ee.telemetry opt-out described in Optimize Telemetry Data Transfer. Disabling it stops DP Manager from remote-writing to Prometheus. Apply the control-plane configuration to make the change; data planes continue serving traffic.

dp_manager_conf/conf.yaml, or dp_manager_configuration in Helm values, needs no change when using direct scraping. With telemetry off, DP Manager stops contacting Prometheus, so its addr value is no longer used.

Configure Authentication​

If the external Prometheus does not require authentication, continue to Remove the Bundled Prometheus. Otherwise, configure the authentication method used by that instance.

Basic Authentication​

Add a nested basic_auth block. The control plane ignores username and password set directly under prometheus.

Send Basic credentials over HTTPS or a trusted network because plain http:// sends them in cleartext. Point addr at an https:// endpoint, or terminate TLS on a co-located reverse proxy.

dashboard_conf/conf.yaml
prometheus:
  addr: "https://<external-prometheus>:9090"   # use https so credentials are not sent in cleartext
  basic_auth:
    username: "api7-readonly"
    password: "<password>"

When using remote write, add the same basic_auth block to dp_manager_conf/conf.yaml. The DP Manager credential needs write access to /api/v1/write.

For Helm deployments, configure these fields under dashboard_configuration.prometheus.basic_auth and, with remote write, dp_manager_configuration.prometheus.basic_auth.

Mutual TLS​

The control plane supports client certificates through a tls block without a proxy. Point addr at the https:// endpoint and reference the client key pair and CA:

dashboard_conf/conf.yaml
prometheus:
  addr: "https://<external-prometheus>:9090"
  tls:
    enable_client_cert: true
    cert_file: /path/to/client.crt
    key_file: /path/to/client.key
    ca_file: /path/to/ca.crt

When using remote write, add the same tls block to dp_manager_conf/conf.yaml and mount the certificate files into the control-plane containers.

For Helm deployments, configure the TLS fields under dashboard_configuration.prometheus.tls and, with remote write, dp_manager_configuration.prometheus.tls. Store the client certificate, key, and CA in one Kubernetes Secret, then set prometheus.server.existingSecret to its name. The chart mounts that Secret at /app/prometheus/certs in both components. Point cert_file, key_file, and ca_file at files under that directory.

Other Authentication Schemes​

For Bearer tokens or any scheme the control plane does not support natively, put a reverse proxy in front of Prometheus to inject the credential. Point addr at the proxy, and set query_path_prefix if the proxy serves the API under a path prefix.

With remote write, the same prefix applies to writes. Set remote_write_path in dp_manager_conf/conf.yaml to include it, for example /prom-proxy/api/v1/write, or have the proxy preserve /api/v1/write. The Helm equivalents are dashboard_configuration.prometheus.query_path_prefix and dp_manager_configuration.prometheus.remote_write_path.

Enforce Least Privilege​

With remote write, the Dashboard and DP Manager can use separate accounts because they read credentials from different files. Native Prometheus authorization gives every valid account the same read and write access, so separate accounts improve credential hygiene but do not make the Dashboard account read-only.

To enforce endpoint-specific permissions, put a reverse proxy in front of Prometheus. Allow the Dashboard account only /api/v1/query and /api/v1/query_range, and allow the DP Manager account /api/v1/write. Blocking the write endpoint causes remote write to fail.

Remove the Bundled Prometheus​

After configuring one metric path and any required authentication, disable the bundled instance.

Delete the prometheus service block, the depends_on entry that references it, and its data volume from docker-compose.yaml:

docker-compose.yaml
services:
  # remove the whole prometheus service block
  # prometheus:
  #   image: api7/prometheus:...

  dashboard:
    depends_on:
      postgresql: {condition: service_healthy}
      # remove the line below
      # prometheus: {condition: service_healthy}

volumes:
  # remove the line below
  # prometheus_data:

If the depends_on entry is left in, the stack fails to start with service "dashboard" depends on undefined service "prometheus": invalid compose project. Run docker compose config >/dev/null to catch it before restarting.

Apply the Migration​

The docker compose commands in this section, Verify, and Roll Back use the service names dashboard and dp-manager. If your Compose file names them api7-ee-dashboard and api7-ee-dp-manager, substitute those names everywhere.

docker compose config >/dev/null && echo "compose OK"

# recreate the control plane components
docker compose up -d --force-recreate dashboard dp-manager

# remove the now-orphaned bundled Prometheus container by name.
# Its service is no longer in the Compose file, so `docker compose rm prometheus`
# reports "no such service". Delete the container directly instead:
docker rm -f <compose-project>-prometheus-1

docker compose logs --tail=100 dp-manager | grep -i prometheus

The DP Manager logs should not contain failed to write prometheus metrics. Data planes continue serving traffic because this procedure does not restart them.

Avoid removing unrelated containers

docker compose up -d --remove-orphans removes every undefined container in the Compose project. If a data plane or another container shares the project without a service entry, it is removed too. Delete Prometheus by name to limit the operation to the intended container.

Remove the Existing Scrape Job​

If you kept direct scraping, skip this section. Otherwise, wait until the updated control plane begins remote-writing metrics, then remove any existing data-plane scrape job. Reload Prometheus by sending SIGHUP to the process or posting to /-/reload when Prometheus runs with --web.enable-lifecycle.

Remote-written series do not carry the scrape-added job and instance labels. Update any Grafana panels or alert rules that filter on those labels.

Verify​

Send some traffic, then wait five minutes before checking. Prometheus keeps returning a series for up to five minutes after its last sample, so stale series from the path you disabled can still appear until then.

  • No duplicate series: Expect one apisix_http_status{code="200"} series for each unique set of gateway-emitted labels, including route_id, service_id, and instance_id. When both paths are active, the scraped copy also carries target labels that are absent from the remote-written copy. The following query assumes that job and instance are the only scrape-added labels and returns no results when only one path is active:

    curl -sG --data-urlencode \
      'query=count without (job, instance) (apisix_http_status{code="200"}) > 1' \
      "http://<external-prometheus>:9090/api/v1/query"

    If a ServiceMonitor or custom relabeling adds other target labels, add every scrape-only label to the without (...) clause before using this check.

  • Values match the data plane: compare sum(apisix_http_status{code="200"}) from the external Prometheus with the same metric summed from the data plane endpoint. For Docker Compose, request http://127.0.0.1:9091/apisix/prometheus/metrics on the data plane host. On Kubernetes, request the configured metrics listener from a pod or through a temporary port-forward. The two values should match.

  • Dashboard Monitoring page: Open a 30-minute range and confirm that every panel contains data.

  • DP Manager with remote write logs no write errors. Use the command for your deployment method.

    Docker Compose

    docker compose logs --tail=200 dp-manager | grep -c 'failed to write prometheus'

    Kubernetes

    kubectl -n <namespace> logs \
      -l app.kubernetes.io/instance=<release-name>,app.kubernetes.io/component=dp-manager \
      --tail=200 | grep -c 'failed to write prometheus'

    The command returns 0 when no write failures are present.

Roll Back​

Restore the backup and bring the bundled Prometheus back. The data plane does not need to restart.

cd <control-plane-directory>
cp -a /var/tmp/api7-prom-backup-<date>/* .
docker compose up -d prometheus
docker compose up -d --force-recreate dashboard dp-manager

Metrics produced while the bundled Prometheus was offline are not copied to it later.

If you used direct scraping, the external Prometheus needs no changes. If you used remote write:

  1. Re-add any scrape job you removed.
  2. Remove --web.enable-remote-write-receiver once no sender needs it. Apply the change the same way you enabled the flag in Enable the Remote-Write Receiver.

Under Docker Compose, recreate the Prometheus container because a plain restart reuses the old command. Under systemd, edit ExecStart and restart the service.

The receiver flag does not add authentication. If you leave it enabled, protect /api/v1/write with authentication and network controls.

Caveats​

  • The gateway emits a repeated metric sample. API7 Gateway 3.10.8 emits apisix_nginx_metric_errors_total twice. Prometheus ingests the other samples and drops the repeated sample. Prometheus 2.52.0 and later also logs a warning; earlier versions ignore the repeated sample silently. This gateway-output issue is separate from the duplicate series caused by enabling both collection paths.
  • History does not migrate. Before the migration, create a restorable TSDB snapshot or storage-volume snapshot and copy it to separate durable storage. Prometheus recommends snapshots for backups. Docker Compose leaves the original volume in place. Disabling the Helm dependency removes its PersistentVolumeClaim manifest, so Kubernetes recovery depends on your storage policy and backups.
  • New data planes need a scrape target with direct scraping. Add every new data plane to Prometheus scrape_configs; otherwise, that instance is absent from Prometheus. Remote write uses the existing data plane-to-DP Manager channel and does not require a scrape target for each instance.
  • adc sync can stop most traffic metrics. HTTP traffic metrics and most AI metric families depend on the default prometheus global rule. If the ADC configuration omits global_rules.prometheus, synchronization deletes that rule. Those families stop updating on routes that do not enable the plugin directly. Node metrics and apisix_llm_active_connections can remain present. See Monitor Metrics for enabling the plugin.
  • Control plane self-metrics are optional. The bundled Prometheus also scraped the control plane's own metrics (api7_dashboard_*, etcd_*). The Dashboard does not use them; add a scrape target for the control plane port 7081 only if you want them for troubleshooting.

Additional Resources​