Diagnose CPU and Memory Usage

Use Diagnostic Agents when a gateway worker consumes excessive CPU or its memory keeps growing. The Dashboard generates the agent's deployment command and credentials, lists connected agents, and opens a console for collecting and downloading diagnostic reports.

The agent runs on the gateway's host and connects to the DP Manager over mTLS. The Dashboard forwards diagnostic requests through that connection, so the agent needs no published port or inbound connection. Each agent sees processes on its own host, including processes outside the selected gateway group.

note

Diagnostic Agents were introduced in API7 Enterprise 3.10.7. Upgrade both the Dashboard and DP Manager to a version that supports this feature. The ngxdig image is released independently; the available capture options depend on the installed image. This guide uses the console with cpu-on and memsnap commands.

For individual request traces and plugin execution timings, use Debug Sessions.

Prerequisites

• A running, licensed API7 Gateway deployment and a running gateway instance. See Deploy with Docker Compose or Deploy on Kubernetes. Choose control-plane image tags for 3.10.7 or later when following those guides; earlier example tags do not include Diagnostic Agents.
• An administrator account, or a role with gateway:ListDiagnosticAgents, gateway:GetDiagnosticAgent, gateway:DeployDiagnosticAgent, and gateway:UseDiagnosticAgentConsole permissions.
• Access to run Docker commands on the gateway host, or to create the generated Kubernetes Secret and Pod on the gateway's node.
• A Linux host supported by the ngxdig image. CPU profiling requires a kernel of at least 5.10, kernel BTF data at /sys/kernel/btf, and permission to use eBPF and performance events. The workflow below was verified on Linux amd64.
• A DP Manager TLS address reachable from the agent, normally on port 7943. Use the address configured for your deployment, with a hostname covered by its certificate. The Dashboard's browser address is not necessarily the DP Manager address.
• In an HA deployment, each DP Manager must be reachable by the other DP Manager instances at its advertised address. Configure server.advertise unless peers can reach the container address derived from the accepted connection. See Control Plane HA.

Memory snapshots read process memory and /proc and do not require eBPF. They still require access to the target process and its PID namespace.

caution

The generated deployment shares the host PID namespace and grants diagnostic capabilities, including BPF, PERFMON, SYS_PTRACE, and SYS_ADMIN. Deploy it only on hosts you intend to diagnose. The generated command or manifest contains a private key; keep it private and do not attach it to a support ticket.

Deploy a Diagnostic Agent

Open Diagnostic Agents in the Dashboard and select Add Diagnostic Agent. Choose the deployment method that matches the gateway host.

Docker

1. Select Docker.

2. Set Control Plane Address to the DP Manager TLS URL reachable from the agent container, including the https:// scheme and port.

3. Set Name to a recognizable host-specific name. Replace <agent-container> in every Docker command below with this name. Optionally set Image to an approved ngxdig image and choose the Certificate Validity Period.

4. Select Generate and save the complete command in a private file, such as diagnostic-agent.sh.

5. Run the generated command on the same Docker host as the gateway:

   chmod 700 diagnostic-agent.sh
   bash diagnostic-agent.sh

The command includes --pid host, capabilities, and mTLS credentials. Keep these settings. It does not publish an agent port.

If the DP Manager address uses a Docker Compose service name, add --network <control-plane-network> to the generated docker run options, before the image name. For example, an agent using https://dp-manager:7943 must join the network where dp-manager resolves. localhost inside the agent container refers to that container, not the Docker host.

Check the agent logs:

docker logs <agent-container> --tail=50

A successful connection reports session established. Return to Diagnostic Agents, wait for Online, and select Open Console.

Record the installed tool version and image identity with the diagnostic results:

docker exec <agent-container> ngxdig version
docker inspect <agent-container> --format '{{.Image}}'

Kubernetes

First find the node hosting the gateway Pod:

kubectl -n <gateway-namespace> get pod <gateway-pod> -o wide

On Add Diagnostic Agent, select Kubernetes and set:

• Control Plane Address: the DP Manager TLS URL reachable from the agent Pod.
• Node Name: the gateway Pod's node from the previous command.
• Namespace: an existing namespace whose admission policy permits the generated host PID access and capabilities.
• Name, Image, and Certificate Validity Period as needed.

Select Generate and save the complete manifest as diagnostic-agent.yaml. It contains a Secret and one Pod pinned to the selected node; it does not deploy an agent to every node.

Apply the manifest and check the resulting Pod, using the namespace and name you entered:

chmod 600 diagnostic-agent.yaml
kubectl apply -f diagnostic-agent.yaml
kubectl -n <agent-namespace> get pod <agent-pod> -o wide
kubectl -n <agent-namespace> logs <agent-pod> --tail=50

Confirm that the gateway and agent are on the same node. Return to Diagnostic Agents and select Open Console once the agent is Online. Deploy a separate agent for each additional node you need to inspect. If the gateway moves to another node, select or deploy an agent there.

note

For CPU profiling, use a host or virtual machine where the agent sees the host PID namespace. Nested container nodes, such as kind or k3d, have a different PID namespace from the kernel's initial namespace and are unsuitable for this CPU profiling workflow. An agent appearing Online only confirms its control-plane connection, not its ability to collect CPU samples.

Every Generate action issues a new certificate and creates a distinct agent identity. Generating again does not replace or stop the previous deployment. Keep the deployment file so you can manage the container or Pod you actually installed.

Select the Gateway Worker

The diagnostic console lists visible processes with their PID, role, master PID, CPU usage, RSS, and uptime. Select a worker from the gateway you are investigating.

For Docker, identify the gateway's host process IDs with:

docker top <gateway-container> -eo pid,ppid,args

Match those process IDs to the console. A PID from docker exec <gateway-container> ps is in the container's namespace and can differ from the host PID shown by the agent. Workers sharing the same master PID belong to the same NGINX instance. Do not select another gateway merely because it is the busiest process on the host.

For CPU issues, choose the affected worker with high CPU usage. For memory issues, start with a worker showing high or growing RSS. Refresh the process list after a gateway restart or reload because worker process IDs can change.

Before CPU profiling, check the collection environment:

# Docker
docker exec <agent-container> ngxdig doctor

# Kubernetes
kubectl -n <agent-namespace> exec <agent-pod> -- ngxdig doctor

Resolve failed CPU collection checks before proceeding. BTF or eBPF failures do not by themselves prevent a memory snapshot.

Capture a CPU Profile

Collect while the CPU problem is occurring. For a test deployment, send representative traffic through an existing route throughout the capture. A nearly idle worker can fail the CPU preflight check or produce too few samples.

1. Select the target worker in the diagnostic console.
2. Under Run Diagnostic, choose cpu-on for Command.
3. Start with Duration seconds set to 30, Hz set to 100, and CPU profile level set to C + Kernel.
4. Set Format to html and select Run.
5. Wait for the run to show succeeded, then open its HTML artifact or select Download all.

The report includes a CPU flame graph and collection quality information. Wider frames account for more samples in that capture; the horizontal axis is not a timeline. Follow wide stacks to identify where CPU time was spent, and check unresolved frames and quality warnings before drawing a conclusion.

A run can succeed while its report shows Degraded status or Low quality. For example, restricted kernel symbols or incomplete native stack unwinding can leave parts of the profile unresolved. Keep those warnings with the report and avoid attributing unresolved samples to a specific function.

C + Kernel + Lua and Lua only are marked experimental in the console. Use them when investigating Lua execution, and retain the report's coverage and quality information. Missing Lua frames are not evidence that Lua consumed no CPU.

Choose evidence instead of html when you need the structured CPU evidence JSON. This produces a JSON artifact rather than an HTML report. Download the JSON for offline analysis or support.

An agent runs one capture at a time. Wait for the current run to finish before starting another.

Capture Memory Snapshots

A memory snapshot describes one worker at one point in time, including an idle worker. To investigate growth, collect a baseline and later snapshots from the same worker under comparable workload conditions.

1. Select the target worker.
2. Choose memsnap for Command.
3. Keep Memory deep mode set to best-effort and select Run.
4. After the run succeeds, open bundle/report.html and download the complete bundle.
5. Repeat when memory has increased. Record the collection times, traffic conditions, and any configuration changes.

The bundle contains report.html, memsnap.json, maps.csv, and meta.json. Keep all four files: the HTML is convenient to read, while the other files preserve the measurements and collection context.

Compare at least three snapshots when possible: two show a difference, while later snapshots help distinguish a one-time increase from continuing growth. A worker restart or reload ends the comparison series; establish a new baseline for the new process.

Interpret the Memory Report

Read the report's confidence and warning fields along with its measurements:

Observation across snapshots	What to investigate
LuaJIT allocated bytes (gc.total) keep increasing	LuaJIT allocation pressure and the approximate object mix. This counter does not prove object reachability or a memory leak.
LuaJIT allocator footprint increases while gc.total stays stable	Allocator retention or fragmentation. This alone does not prove a leak.
A shared zone's used bytes increase	The named module's shared-memory usage and capacity.
Private anonymous memory increases without a corresponding LuaJIT or shared-zone increase	Native allocations or other memory not attributed by those decoders; retain the complete evidence for further analysis.

RSS, PSS, LuaJIT object sizes, allocator footprint, and shared-zone sizes measure different, overlapping aspects of memory. Do not add them together or sum workers' RSS to estimate unique gateway memory without accounting for shared pages.

best-effort reads a running worker. Concurrent allocation can produce a partial or degraded decoder result even when the run succeeds. Use the available measurements with their stated confidence; missing or unsupported measurements are not zero.

caution

The consistent memory mode pauses the target worker while reading its memory and can delay requests handled by that worker. Use it only during an acceptable interruption window when a consistent snapshot is necessary. Set Max pause seconds to a finite limit; 0 means unlimited. The normal workflow uses best-effort, which does not deliberately pause the worker.

Download and Preserve Results

In Runs, use Download all for one capture, or select multiple runs and use Download selected to download them together. Open downloaded HTML reports in a browser; keep the original evidence files when sharing results with support.

Include the gateway version, ngxdig version and image identity, host or node, worker PID, symptom time window, and traffic conditions. Review reports before sharing because process details, paths, and operational data can be sensitive. Do not include the deployment command, manifest, or mTLS private key.

Download results before removing the agent. The generated Docker deployment stores results in the container, and the Kubernetes deployment uses an emptyDir volume. Removing the container or Pod can remove those results; the Dashboard is not a durable report archive.

Troubleshooting

Symptom	Check and action
Agent remains Offline or never appears	Inspect agent logs. Check DNS, network access to the DP Manager TLS port, certificate validity, and the configured address. An agent does not appear until it has connected.
TLS hostname verification fails	Use the DP Manager address covered by its certificate and confirm the control plane's dp_manager_address configuration. Regenerate credentials after correcting the configuration; do not disable TLS verification.
Console does not list the intended worker	Confirm the agent is on the gateway's host or node and retains --pid host or hostPID: true. Compare host process IDs and refresh after a restart.
Kubernetes rejects the agent Pod	Inspect Pod events and admission errors. Use a namespace approved for host PID access and the generated capabilities.
CPU collection fails on BTF, permissions, or performance events	Run ngxdig doctor. Check the host kernel, /sys/kernel/btf visibility, generated capabilities, and runtime security policy.
CPU preflight reports an idle target or insufficient samples	Capture the affected worker while the issue and representative traffic are present. Confirm that the selected PID belongs to that gateway.
A run is already in progress	Wait for it to finish before starting the next capture on that agent.
Memory capture cannot read /proc/<pid>/mem or smaps_rollup	Confirm the PID still exists, host PID access is enabled, and SYS_PTRACE and the runtime security policy permit reading the worker.
A memory decoder reports partial, degraded, or unsupported data	Keep the bundle and inspect its warnings. Do not treat unavailable data as zero or assume every decoder succeeded because the run succeeded.

Remove the Agent

After downloading the reports, remove the Docker agent by its deployment name:

docker rm -f <agent-container>

For Kubernetes, delete the resources from the saved generated manifest:

kubectl delete -f diagnostic-agent.yaml

The agent stops collecting and eventually appears Offline in the Dashboard. Removing its container or Pod does not remove the gateway.