Use an Existing Prometheus

The Docker Compose quickstart ships a bundled Prometheus. If you already run Prometheus, you can replace the bundled instance with your external deployment. After the migration, the Dashboard Monitoring page queries the external Prometheus.

How Gateway Metrics Reach Prometheus

API7 Gateway metrics can reach Prometheus through two independent paths.

Path	Flow	Used by
Remote write	Data plane → DP Manager → Prometheus (/api/v1/write)	The Dashboard Monitoring page, whose queries run against this Prometheus.
Scrape	Prometheus → data plane :9091 (see Monitor Metrics)	Your own Grafana dashboards or alerting stack after verifying that the gateway endpoint does not emit duplicate families.

The bundled Prometheus receives remote write for the Dashboard. When the metrics endpoint does not emit duplicate metric families, both paths use identical metric names and labels. Prometheus adds job and instance only to scraped series. Dashboard queries aggregate with sum(...) by (...) and do not filter on job, so the Monitoring page can query metrics delivered through either path.

Do not keep both paths to the same Prometheus

If one Prometheus both scrapes the data plane and receives remote write, the same gateway metric can arrive through both paths. The Dashboard then double-counts request and bandwidth series. Keep exactly one path per Prometheus instance.

Prerequisites

Before migrating, make sure the external instance and the control-plane configuration files are available.

• An external Prometheus instance reachable from the control plane containers.
• The control plane deployment directory containing docker-compose.yaml, dashboard_conf/conf.yaml, and dp_manager_conf/conf.yaml. Service names may be api7-ee-dashboard and api7-ee-dp-manager in some releases.

Choose a Migration Path

For the current public release, use remote write. Use direct scraping only with a later fixed release or a build that API7 Support confirms contains the fix.

Direct scraping is unavailable in this release

This release emits apisix_nginx_metric_errors_total twice from the data-plane endpoint, and Prometheus rejects the entire scrape. No fixed version has been released.

Compare the changes required by each path:

Consideration	Remote write	Direct scrape
Availability	Available in the current public release	Not available in the current public release; requires a later fixed release or API7 Support-confirmed build
When	External Prometheus can accept DP Manager remote write, including when it previously scraped the data plane	External Prometheus already scrapes a verified gateway endpoint without duplicate metric families
External Prometheus changes	Add --web.enable-remote-write-receiver; the control plane writes metrics to it	None; the scrape configuration is unchanged
Permission needed	Dashboard needs read access; DP Manager needs write access	Read access
Existing Grafana and alerts	Remote-written series omit job and instance; update queries that filter on them	No query changes

Complete the configuration for one path, then continue with the shared authentication, removal, restart, verification, and rollback sections.

Back Up the Current Configuration

Create a backup that can restore the original Prometheus configuration and Docker Compose services.

cd <control-plane-directory>
BAK="/var/tmp/api7-prom-backup-$(date +%F)"
mkdir -p "$BAK"
cp -a docker-compose.yaml dashboard_conf dp_manager_conf "$BAK"/

Configure Remote Write

DP Manager sends gateway metrics to the external Prometheus, and the Dashboard queries the same instance.

Enable the Remote-Write Receiver

--web.enable-remote-write-receiver is a Prometheus startup flag, not a prometheus.yml setting. Add the flag and restart or recreate Prometheus for the change to take effect:

• systemd: add --web.enable-remote-write-receiver to ExecStart, then systemctl daemon-reload && systemctl restart prometheus.
• Docker / Docker Compose: add --web.enable-remote-write-receiver to the container's command, then recreate the container.
• Kubernetes: add it to the Prometheus container args, then roll the pod.

Confirm that the receiver is enabled:

curl -s -o /dev/null -w '%{http_code}\n' -X POST "http://<external-prometheus>:9090/api/v1/write"
# 400 = enabled (invalid body, expected)   404 = still disabled

If the endpoint requires authentication, add the matching scheme, such as -u <user>:<password> for Basic authentication, and use the https:// URL. A 401 or 403 response indicates an authentication failure, not a disabled receiver.

Point the Dashboard and DP Manager at the External Prometheus

Set addr in both config files to the external Prometheus. Keep telemetry.enable: true so DP Manager continues reporting metrics.

dashboard_conf/conf.yaml

prometheus:
  addr: "http://<external-prometheus>:9090"   # must be reachable from inside the dashboard container
  query_path_prefix: ""
  whitelist:
    - "/api/v1/query_range"
    - "/api/v1/query"
    # keep the rest unchanged

telemetry:
  enable: true   # keep enabled for remote write

dp_manager_conf/conf.yaml

prometheus:
  addr: "http://<external-prometheus>:9090"
  remote_write_path: "/api/v1/write"

Remote-write users can skip the direct-scrape section and continue with Configure Authentication.

Keep Direct Scraping on a Verified Fixed Build

Use direct scraping only with a later release that fixes the duplicate exposition or a build that API7 Support has confirmed contains the fix.

Before continuing, configure the data plane metrics listener to bind to an address Prometheus can reach. Its default is 127.0.0.1; publishing or exposing port 9091 alone does not change that address. See Monitor Metrics for the listener configuration.

Confirm the External Prometheus Has Gateway Metrics

Send a few requests through a route, then check that the external Prometheus has ingested the resulting gateway metrics. This read-only request should return a result greater than 0:

curl -sG --data-urlencode 'query=count(apisix_http_status)' \
  "http://<external-prometheus>:9090/api/v1/query"

An empty result can also mean that no traffic has reached the gateway or that the prometheus plugin is not enabled as a global rule. If the count stays 0 after traffic, fix the scrape job or enable the plugin before continuing.

If the external Prometheus is authenticated, add credentials to this curl, such as -u <user>:<password>, and use the https:// URL. A 401 or 403 response indicates an authentication failure rather than missing metrics.

Point the Dashboard at the External Prometheus and Stop Remote Write

Edit dashboard_conf/conf.yaml. Change addr to the external Prometheus, and disable telemetry so the control plane stops writing metrics that would duplicate your scrape.

dashboard_conf/conf.yaml

prometheus:
  addr: "http://<external-prometheus>:9090"   # must be reachable from inside the dashboard container
  query_path_prefix: ""
  whitelist:
    - "/api/v1/query_range"
    - "/api/v1/query"
    # keep the rest unchanged

telemetry:
  enable: false

Disable remote write when using direct scraping

If telemetry.enable stays true, the control plane keeps remote-writing metrics that collide with your scraped series, and the Dashboard counts them twice.

Control-plane telemetry setting

This telemetry block is a top-level key in the control plane's dashboard_conf/conf.yaml, not the data plane api7ee.telemetry opt-out described in Optimize Telemetry Data Transfer. Disabling it stops DP Manager from remote-writing to Prometheus. Recreate the control-plane containers to apply the change; data planes continue serving traffic.

dp_manager_conf/conf.yaml needs no change when using direct scraping. With telemetry off, DP Manager stops contacting Prometheus, so its addr value is no longer used.

Configure Authentication

If the external Prometheus does not require authentication, continue to Remove the Bundled Prometheus. Otherwise, configure the authentication method used by that instance.

Basic Authentication

Add a nested basic_auth block. The control plane ignores username and password set directly under prometheus.

Send Basic credentials over HTTPS or a trusted network because plain http:// sends them in cleartext. Point addr at an https:// endpoint, or terminate TLS on a co-located reverse proxy.

dashboard_conf/conf.yaml

prometheus:
  addr: "https://<external-prometheus>:9090"   # use https so credentials are not sent in cleartext
  basic_auth:
    username: "api7-readonly"
    password: "<password>"

When using remote write, add the same basic_auth block to dp_manager_conf/conf.yaml. The DP Manager credential needs write access to /api/v1/write.

Mutual TLS

The control plane supports client certificates through a tls block without a proxy. Point addr at the https:// endpoint and reference the client key pair and CA:

dashboard_conf/conf.yaml

prometheus:
  addr: "https://<external-prometheus>:9090"
  tls:
    enable_client_cert: true
    cert_file: /path/to/client.crt
    key_file: /path/to/client.key
    ca_file: /path/to/ca.crt

When using remote write, add the same tls block to dp_manager_conf/conf.yaml and mount the certificate files into the control-plane containers.

Other Authentication Schemes

For Bearer tokens or any scheme the control plane does not support natively, put a reverse proxy in front of Prometheus to inject the credential. Point addr at the proxy, and set query_path_prefix if the proxy serves the API under a path prefix.

With remote write, the same prefix applies to writes. Set remote_write_path in dp_manager_conf/conf.yaml to include it, for example /prom-proxy/api/v1/write, or have the proxy preserve /api/v1/write.

Separate accounts do not enforce least privilege

With remote write, the Dashboard and DP Manager can use separate accounts because they read credentials from different files. Native Prometheus authorization gives every valid account the same read and write access, so separate accounts improve credential hygiene but do not make the Dashboard account read-only.

To enforce endpoint-specific permissions, put a reverse proxy in front of Prometheus. Allow the Dashboard account only /api/v1/query and /api/v1/query_range, and allow the DP Manager account /api/v1/write. Blocking the write endpoint causes remote write to fail.

Remove the Bundled Prometheus

After configuring one metric path and any required authentication, remove the bundled instance from Docker Compose.

Delete the prometheus service block, the depends_on entry that references it, and its data volume from docker-compose.yaml:

docker-compose.yaml

services:
  # remove the whole prometheus service block
  # prometheus:
  #   image: api7/prometheus:...

  dashboard:
    depends_on:
      postgresql: {condition: service_healthy}
      # remove the line below
      # prometheus: {condition: service_healthy}

volumes:
  # remove the line below
  # prometheus_data:

If the depends_on entry is left in, the stack fails to start with service "dashboard" depends on undefined service "prometheus": invalid compose project. Run docker compose config >/dev/null to catch it before restarting.

Restart the Control Plane

The docker compose commands in this section, Verify, and Roll Back use the service names dashboard and dp-manager. If your Compose file names them api7-ee-dashboard and api7-ee-dp-manager, substitute those names everywhere.

docker compose config >/dev/null && echo "compose OK"

# recreate the control plane components
docker compose up -d --force-recreate dashboard dp-manager

# remove the now-orphaned bundled Prometheus container by name.
# Its service is no longer in the Compose file, so `docker compose rm prometheus`
# reports "no such service". Delete the container directly instead:
docker rm -f <compose-project>-prometheus-1

docker compose logs --tail=100 dp-manager | grep -i prometheus

The DP Manager logs should not contain failed to write prometheus metrics. Data planes continue serving traffic because this procedure does not restart them.

Why not --remove-orphans

docker compose up -d --remove-orphans removes every undefined container in the Compose project. If a data plane or another container shares the project without a service entry, it is removed too. Delete Prometheus by name to limit the operation to the intended container.

Remove the Existing Scrape Job

If you kept direct scraping on a verified fixed build, skip this section. Otherwise, wait until the restarted control plane begins remote-writing metrics, then remove any existing data-plane scrape job. Reload Prometheus by sending SIGHUP to the process or posting to /-/reload when Prometheus runs with --web.enable-lifecycle.

Remote-written series do not carry the scrape-added job and instance labels. Update any Grafana panels or alert rules that filter on those labels.

Verify

Send some traffic, then wait five minutes before checking. Prometheus keeps returning a series for up to five minutes after its last sample, so stale series from the path you disabled can still appear until then.

• No duplicate series: One apisix_http_status{code="200"} series per route, service, and instance is expected. When both paths are active, the same label set arrives twice and differs only by the scrape-added job and instance labels. The following query groups those labels away and returns no results when only one path is active:

  curl -sG --data-urlencode \
    'query=count without (job, instance) (apisix_http_status{code="200"}) > 1' \
    "http://<external-prometheus>:9090/api/v1/query"

• Values match the data plane: compare sum(apisix_http_status{code="200"}) from the external Prometheus with the same metric summed from the data plane endpoint (curl "http://127.0.0.1:9091/apisix/prometheus/metrics"). The two values should match.

• Dashboard Monitoring page: Open a 30-minute range and confirm that every panel contains data.

• DP Manager with remote write logs no write errors: docker compose logs --tail=200 dp-manager | grep -c 'failed to write prometheus' returns 0.

Roll Back

Restore the backup and bring the bundled Prometheus back. The data plane does not need to restart.

cd <control-plane-directory>
cp -a /var/tmp/api7-prom-backup-<date>/* .
docker compose up -d prometheus
docker compose up -d --force-recreate dashboard dp-manager

Metrics produced while the bundled Prometheus was offline are not copied to it later.

If you used direct scraping, the external Prometheus needs no changes. If you used remote write:

1. Re-add any scrape job you removed.
2. Remove --web.enable-remote-write-receiver once no sender needs it. Apply the change the same way you enabled the flag in Enable the Remote-Write Receiver.

Under Docker Compose, recreate the Prometheus container because a plain restart reuses the old command. Under systemd, edit ExecStart and restart the service.

The receiver flag does not add authentication. If you leave it enabled, protect /api/v1/write with authentication and network controls.

Caveats

• History does not migrate. Data in the bundled Prometheus stays in its volume. Export it first (promtool tsdb dump) if you need it.
• New data planes need a scrape target with direct scraping. Add every new data plane to Prometheus scrape_configs; otherwise, that instance is absent from Prometheus. Remote write uses the existing data plane-to-DP Manager channel and does not require a scrape target for each instance.
• adc sync can stop most traffic metrics. HTTP traffic metrics and most AI metric families depend on the default prometheus global rule. If the ADC configuration omits global_rules.prometheus, synchronization deletes that rule. Those families stop updating on routes that do not enable the plugin directly. Node metrics and apisix_llm_active_connections can remain present. See Monitor Metrics for enabling the plugin.
• Control plane self-metrics are optional. The bundled Prometheus also scraped the control plane's own metrics (api7_dashboard_*, etcd_*). The Dashboard does not use them; add a scrape target for the control plane port 7081 only if you want them for troubleshooting.

Additional Resources

• Monitor Metrics: data plane scrape endpoint and key metric names.
• Data Plane Metrics Reference: metric types and built-in labels.
• Configure Alerts: alerting on metric thresholds.