Use an Existing Prometheus

The Docker Compose quickstart ships a bundled Prometheus. If you already run Prometheus, you can point API7 Gateway at your existing instance and remove the bundled one. This page explains the two metric paths involved and gives a safe migration that keeps the Dashboard Monitoring page working.

How Gateway Metrics Reach Prometheus

API7 Gateway metrics can reach a Prometheus instance through two independent paths. Understanding both is the key to a clean migration, because the same metrics can arrive on either path.

Path	Flow	Serves
Remote write	Data plane → DP Manager → Prometheus (/api/v1/write)	The Dashboard Monitoring page, which the control plane populates by querying this Prometheus.
Scrape	Prometheus → data plane :9091 (see Monitor Metrics)	Your own Grafana dashboards or alerting stack.

The bundled Prometheus is the remote-write sink for the Dashboard. The metric names and labels are identical on both paths — the scraped series only carry the extra job and instance labels that Prometheus adds. Because the Dashboard's queries aggregate with sum(...) by (...) and do not filter on job, the control plane works the same whether its data arrived by remote write or by scrape.

Do not keep both paths to the same Prometheus

If one Prometheus both scrapes the data plane and receives remote write, every metric exists as two series. The Dashboard then counts them twice, so request counts and bandwidth read double. Keep exactly one path per Prometheus instance.

Prerequisites

• An external Prometheus instance reachable from the control plane containers.
• The control plane deployment directory containing docker-compose.yaml, dashboard_conf/conf.yaml, and dp_manager_conf/conf.yaml. Service names may be api7-ee-dashboard and api7-ee-dp-manager in some releases.

Choose a Migration Path

Pick the path that matches how your external Prometheus already gets gateway metrics.

	Path A — Keep scraping	Path B — Remote write
When	External Prometheus already scrapes the data plane :9091	External Prometheus does not scrape the data plane yet
External Prometheus changes	None — the scrape config is untouched	Add --web.enable-remote-write-receiver; the control plane writes metrics to it
Permission needed	Read only	Read (Dashboard queries) + write (DP Manager remote write)
Existing Grafana / alerts	Unaffected	Series lose the job and instance labels; queries filtering on them must be updated

Path A is the lighter change when your Prometheus already scrapes the data plane. Path A is the main line below; Path B follows as its own complete sequence. Verify and Roll Back apply to both.

Path A: Keep Scraping and Drop the Bundled Prometheus

Step 1: Back Up the Current Configuration

cd <control-plane-directory>
BAK="/var/tmp/api7-prom-backup-$(date +%F)"
mkdir -p "$BAK"
cp -a docker-compose.yaml dashboard_conf dp_manager_conf "$BAK"/

Step 2: Confirm the External Prometheus Already Has Gateway Metrics

This is a read-only check — it changes nothing.

curl -sG --data-urlencode 'query=count(apisix_http_status)' \
  "http://<external-prometheus>:9090/api/v1/query"

Send a few requests through a route first, then run the check — the result should be greater than 0. An empty result is not always a broken scrape: it is also empty before any traffic reaches the gateway, or when the prometheus plugin is not enabled as a global rule. If the count stays 0 after traffic, fix the scrape job or enable the plugin before continuing. If the external Prometheus is authenticated, add credentials to this curl (for example -u <user>:<password>) and use the https:// URL — otherwise it returns 401 or 403, which is not a metrics problem.

Step 3: Point the Dashboard at the External Prometheus and Stop Remote Write

Edit dashboard_conf/conf.yaml. Change addr to the external Prometheus, and disable telemetry so the control plane stops writing metrics that would duplicate your scrape.

dashboard_conf/conf.yaml

prometheus:
  addr: "http://<external-prometheus>:9090"   # must be reachable from inside the dashboard container
  query_path_prefix: ""
  whitelist:
    - "/api/v1/query_range"
    - "/api/v1/query"
    # keep the rest unchanged

telemetry:
  enable: false

Disabling telemetry is required on Path A

If telemetry.enable stays true, the control plane keeps remote-writing metrics that collide with your scraped series, and the Dashboard counts them twice.

Which telemetry setting this is

This telemetry block is a top-level key in the control plane's dashboard_conf/conf.yaml. It is not the data plane api7ee.telemetry opt-out described in Optimize Telemetry Data Transfer. Setting it here stops DP Manager from remote-writing to Prometheus. Recreating the control plane containers applies the change, and the data plane keeps running and serving traffic — no data plane restart is needed.

dp_manager_conf/conf.yaml needs no change on Path A — with telemetry off, DP Manager stops contacting Prometheus, so its addr value is no longer used.

Step 4: Remove the Bundled Prometheus from Docker Compose

Delete three things from docker-compose.yaml: the prometheus service block, the depends_on entry that references it, and its data volume.

docker-compose.yaml

services:
  # remove the whole prometheus service block
  # prometheus:
  #   image: api7/prometheus:...

  dashboard:
    depends_on:
      postgresql: {condition: service_healthy}
      # remove the line below
      # prometheus: {condition: service_healthy}

volumes:
  # remove the line below
  # prometheus_data:

If the depends_on entry is left in, the stack fails to start with service "dashboard" depends on undefined service "prometheus": invalid compose project. Run docker compose config >/dev/null to catch it before restarting.

Step 5: Restart the Control Plane

The docker compose commands throughout this guide — in this step, in Verify, and in Roll Back — use the service names dashboard and dp-manager. If your Compose file names them api7-ee-dashboard and api7-ee-dp-manager, substitute those names everywhere.

docker compose config >/dev/null && echo "compose OK"

# recreate the control plane components
docker compose up -d --force-recreate dashboard dp-manager

# remove the now-orphaned bundled Prometheus container by name.
# Step 4 deleted its service from the file, so `docker compose rm prometheus`
# reports "no such service". Delete the container directly instead:
docker rm -f <compose-project>-prometheus-1

docker compose logs --tail=100 dp-manager | grep -i prometheus

The DP Manager logs should not contain failed to write prometheus metrics. The data plane does not need to restart, and traffic is not interrupted.

Why not --remove-orphans

docker compose up -d --remove-orphans also works, but it removes every container in the Compose project that is no longer defined in the file — not just Prometheus. If your data plane or any other container runs in the same project without a service entry, that flag deletes it too. Removing the container by name is safer.

Path B: Switch to Remote Write

Use Path B when the external Prometheus does not scrape the data plane. Back up your configuration first (Step 1), then follow the steps below in order.

Step B-1: Enable the Remote-Write Receiver on the External Prometheus

This is a start-up flag on the Prometheus process, not a prometheus.yml setting, so add it to how Prometheus is launched and then restart Prometheus for it to take effect:

• systemd: add --web.enable-remote-write-receiver to ExecStart, then systemctl daemon-reload && systemctl restart prometheus.
• Docker / Docker Compose: add --web.enable-remote-write-receiver to the container's command, then recreate the container.
• Kubernetes: add it to the Prometheus container args, then roll the pod.

After the restart, confirm the receiver is on:

curl -s -o /dev/null -w '%{http_code}\n' -X POST "http://<external-prometheus>:9090/api/v1/write"
# 400 = enabled (invalid body, expected)   404 = still disabled

If the endpoint requires authentication, add the matching scheme (for example -u <user>:<password> for Basic auth) and use the https:// URL. Without credentials the probe returns 401 or 403 before it reaches the receiver, which is not the same as the receiver being off.

Step B-2: Point the Dashboard and DP Manager at the External Prometheus

Change addr in both config files. Unlike Path A, keep telemetry.enable: true so DP Manager keeps reporting metrics — now remote-written to the external Prometheus.

dashboard_conf/conf.yaml

prometheus:
  addr: "http://<external-prometheus>:9090"   # must be reachable from inside the dashboard container
  query_path_prefix: ""
  whitelist:
    - "/api/v1/query_range"
    - "/api/v1/query"
    # keep the rest unchanged

telemetry:
  enable: true   # keep enabled on Path B

dp_manager_conf/conf.yaml

prometheus:
  addr: "http://<external-prometheus>:9090"
  remote_write_path: "/api/v1/write"

Step B-3: Remove the Bundled Prometheus and Restart the Control Plane

Remove the bundled Prometheus from docker-compose.yaml exactly as in Step 4, then restart the control plane exactly as in Step 5. Recreating the dashboard and dp-manager containers applies both changes from Step B-2. The data plane keeps running and serving traffic — no data plane restart is needed.

Step B-4: Remove an Existing Scrape Job

Only if this Prometheus was already scraping the data plane: remove that scrape job so metrics do not arrive twice, then reload Prometheus (send SIGHUP to the process, or POST /-/reload when Prometheus runs with --web.enable-lifecycle). Update any Grafana panels or alert rules that filter on job or instance, since remote-written series do not carry those labels.

Authentication

If the external Prometheus requires Basic authentication, add a basic_auth block. It must be nested — username and password set directly under prometheus are ignored. Send Basic credentials over HTTPS or a trusted network only; over plain http:// they travel in cleartext. Point addr at an https:// endpoint, or terminate TLS on a co-located reverse proxy.

dashboard_conf/conf.yaml

prometheus:
  addr: "https://<external-prometheus>:9090"   # use https so credentials are not sent in cleartext
  basic_auth:
    username: "api7-readonly"
    password: "<password>"

Use the same basic_auth block in dp_manager_conf/conf.yaml on Path B. Because DP Manager writes to /api/v1/write, its credential must permit writes.

For mutual TLS, the control plane supports client certificates natively through a tls block — no proxy needed. Point addr at the https:// endpoint and reference the client key pair and CA:

dashboard_conf/conf.yaml

prometheus:
  addr: "https://<external-prometheus>:9090"
  tls:
    enable_client_cert: true
    cert_file: /path/to/client.crt
    key_file: /path/to/client.key
    ca_file: /path/to/ca.crt

Use the same tls block in dp_manager_conf/conf.yaml on Path B, and mount the certificate files into the control plane containers.

For Bearer tokens or any scheme the control plane does not support natively, put a reverse proxy in front of Prometheus to inject the credential. Point addr at the proxy, and set query_path_prefix if the proxy serves the API under a path prefix. On Path B the same prefix applies to writes. Set remote_write_path in dp_manager_conf/conf.yaml to include it (for example /prom-proxy/api/v1/write), or have the proxy preserve /api/v1/write.

Separate accounts do not mean least privilege

The Dashboard and DP Manager read their credentials from different files, so you can give them separate accounts — but that separates credentials, not permissions. Native Prometheus authorization is all-or-nothing: any valid account can both read and write, so separate accounts alone do not make the Dashboard's credential read-only. A separate account is enough for credential hygiene and needs no special role. To enforce least privilege — for example so a leaked query credential cannot write to Prometheus — put a reverse proxy in front that limits each account to its own endpoints. Allow the Dashboard's account only /api/v1/query and /api/v1/query_range, and, on Path B, allow DP Manager's account /api/v1/write. Do not block the write endpoint for DP Manager on Path B, or its remote write fails.

Verify

Send some traffic, then wait five minutes before checking. Prometheus keeps returning a series for up to five minutes after its last sample, so stale series from the path you disabled can still appear until then.

• No duplicate series — apisix_http_status{code="200"} returns one series per route, service, and instance, which is expected. A duplicate is the same label set arriving from both paths, distinguished only by the scrape-added job and instance labels. Detect it by grouping those two labels away: if any group has more than one series, both paths are still active. With test traffic that produced a 200, this returns nothing when the migration is clean:

  curl -sG --data-urlencode \
    'query=count without (job, instance) (apisix_http_status{code="200"}) > 1' \
    "http://<external-prometheus>:9090/api/v1/query"

• Values match the data plane — compare sum(apisix_http_status{code="200"}) from the external Prometheus with the same metric summed from the data plane endpoint (curl "http://127.0.0.1:9091/apisix/prometheus/metrics"); the two should match.

• Dashboard Monitoring page shows data across all panels for a 30-minute range.

• DP Manager logs no write errors: docker compose logs --tail=200 dp-manager | grep -c 'failed to write prometheus' returns 0.

Roll Back

Restore the backup and bring the bundled Prometheus back. The data plane does not need to restart, and on Path A the external Prometheus needs no changes.

cd <control-plane-directory>
cp -a /var/tmp/api7-prom-backup-<date>/* .
docker compose up -d prometheus
docker compose up -d --force-recreate dashboard dp-manager

The bundled Prometheus does not receive metrics for the switchover window retroactively. On Path B, also revert the external Prometheus: re-add any scrape job you removed, and remove the --web.enable-remote-write-receiver flag once no sender needs it. This flag is part of the launch command, so apply the removal the way you enabled it in Step B-1. Under Docker Compose, recreate the container — a plain restart reuses the old command. Under systemd, edit ExecStart and restart the service. An enabled receiver accepts writes without authentication — the flag adds none. If you leave it on, protect /api/v1/write with authentication and network controls.

Caveats

• History does not migrate. Data in the bundled Prometheus stays in its volume. Export it first (promtool tsdb dump) if you need it.
• New data planes need a scrape target (Path A). Each new data plane must be added to your Prometheus scrape_configs, and a data plane that Prometheus cannot reach has no metrics. Path B avoids this, since metrics flow over the existing data plane to DP Manager channel.
• adc sync can zero out metrics. Gateway metrics depend on the default prometheus global rule. If your ADC configuration file omits global_rules.prometheus, a sync deletes that rule and all gateway metrics stop. See Monitor Metrics for enabling the plugin.
• Control plane self-metrics are optional. The bundled Prometheus also scraped the control plane's own metrics (api7_dashboard_*, etcd_*). The Dashboard does not use them; add a scrape target for the control plane port 7081 only if you want them for troubleshooting.

Additional Resources

• Monitor Metrics — the data plane scrape endpoint and key metric names.
• Configure Alerts — alerting on metric thresholds.