Skip to main content
API7.ai Logo
Version: 3.10.x

Capture Request Traces with Debug Sessions

Metrics tell you an error rate went up. Access logs tell you which route and which upstream. Neither tells you what the gateway actually did to the request that failed.

A debug session answers that. You pick the instances, a window of time, and the traffic you care about. For each matching request the gateway records every plugin that ran, in the order it ran, how long each took, and every log line it produced.

What You Can Find Out​

Which plugin made the request slow. A checkout endpoint takes 400 ms and you cannot tell where the time goes. The trace breaks the request into phases and then into individual plugin calls, each with its own duration, so the slow one is visible at a glance.

Why a plugin did not take effect. Plugin execution order is computed from priorities and merged across route, service, and global-rule configuration. Working that out by reading configuration is error-prone. The trace shows the plugins that actually ran on one real request, in the order the gateway ran them — no reasoning required.

What the gateway logged for one specific request. Log lines from that request are attached to its trace. You no longer have to raise the log level in production, then sift the output for the lines that belong together.

What happened at the moment an alert fired. An alert policy can start a session by itself, so the traffic that set off the alert is captured while it is happening. You are not left trying to reproduce an intermittent failure after the fact.

How It Works​

You create a session against a gateway group. The control plane pushes it to the instances you selected with a lease equal to the session duration, so capture stops on its own when the window closes.

Each matching request is recorded as an OpenTelemetry trace. The data plane sends it to the control plane over the same authenticated channel it already uses for heartbeats. The control plane stores it in your own Jaeger deployment, so trace data never leaves your network.

note

Debug sessions trace HTTP traffic only. Stream (L4) proxying and the TLS handshake are not traced.

Prerequisites​

  • A gateway group with at least one online instance running API7 Gateway 3.9.13 or later.
  • Jaeger configured on the control plane. The Helm chart and Docker Compose deployments include Jaeger by default. For RPM installations, deploy Jaeger yourself and set jaeger.addr and jaeger.collector_addr in the Dashboard and DP Manager configuration.
  • The gateway:CreateDebugSession permission on the gateway group.

You do not need to enable the opentelemetry plugin or the static apisix.tracing option. Debug sessions use an independent runtime switch.

Step 1: Start a Session​

  1. Go to Gateway Groups, select your gateway group, and click Debug Sessions.

  2. Click Create Debug Session.

  3. Fill in the form:

    FieldDescription
    NameAn identifier for the session, up to 100 characters.
    DescriptionOptional notes about why you started the session.
    Target InstancesThe instances to trace. At least one is required.
    Duration (seconds)How long instances keep tracing, from 10 to 600. Default 60.
    Max SamplesHow many traces the session lists, from 1 to 200. Default 5.
    Sampling RuleWhich requests to capture. Leave empty to capture every request.

  4. Click Submit. The session opens with a countdown of the remaining capture time, and traces appear in the list as they arrive.

Step 2: Capture Only the Traffic You Care About​

On a busy gateway, capturing everything buries the request you are looking for. A sampling rule narrows the capture to the traffic that matters.

Rules are evaluated after the response completes, so you can filter on the outcome — status code, upstream latency — not just on the incoming request.

Only server errors:

[["status", ">=", 500]]

Slow requests to one endpoint:

[["uri", "==", "/checkout"], ["upstream_response_time", ">", 1]]

One tenant, matched either by address range or by header:

["OR", ["remote_addr", "ipmatch", "10.1.0.0/16"], ["http_x_tenant_id", "==", "acme"]]

A rule is an array of [variable, operator, value] conditions, combined with AND unless the first element is OR, AND, !OR, or !AND.

Common variables
uri, request_uri, method, statusPath, path with query string, method, response status
host, remote_addrRequest host and client address
route_id, route_name, service_id, service_nameMatched resource
consumer_name, consumer_group_idAuthenticated consumer
upstream_status, upstream_response_timeUpstream result and timing
http_<header>, arg_<name>, cookie_<name>A request header, query argument, or cookie

Operators are ==, ~=, >, >=, <, <=, ~~ and ~* for regular expressions, in, has, and ipmatch.

caution

uri, method, and status are the request path, method, and response status. Do not write them as http_uri, http_method, or http_status — the http_ prefix reads a request header of that name, so those expressions never match and the session silently captures nothing.

The Dashboard's Match URI, Match Method, and 5xx Errors preset buttons currently insert the http_-prefixed form. After clicking a preset, edit the variable name.

Step 3: Read the Trace​

Open a trace to see the request as a waterfall. Reading it top to bottom gives you the execution order directly:

GET /checkout                                       412ms
├─ apisix.phase.access                              380ms
│  ├─ http_router_match                             0.16ms
│  ├─ apisix.phase.rewrite.plugins.key-auth         3.1ms
│  ├─ apisix.phase.rewrite.plugins.proxy-rewrite    0.4ms
│  └─ apisix.phase.access.plugins.limit-count       371ms   ←
├─ apisix.phase.header_filter                       0.25ms
└─ apisix.phase.body_filter                         1.4ms

Each plugin call is its own span named apisix.phase.<phase>.plugins.<plugin>, nested under the phase span that was open when it ran. Within a phase, plugins run in descending priority order, and that is the order you see. Here limit-count accounts for almost the whole request.

The gateway opens phase spans for access, header_filter, and body_filter. Request-side plugins — including rewrite-phase ones — therefore all appear under apisix.phase.access. You will also see http_router_match for route matching, run_global_rules.<phase> for global rules, and resolve_dns or fetch_secret when they occur.

The Request's Own Logs​

Select the root span and open the Events tab to see the log lines that request produced, each with its level, source location, and message.

Capture is independent of the gateway's configured error_log level, so a debug-level message appears here even when it is never written to the error log. Up to 200 entries are kept per request, each truncated at 2048 bytes. Lines written by nginx itself, such as upstream connection errors, are not captured, and structured arguments logged as Lua tables appear as {...}.

note

Log capture as span events requires API7 Gateway 3.9.15 or 3.10.2 and later.

Take the Trace With You​

Click Download OTLP JSON to export a trace as an OpenTelemetry document — useful for attaching to a support case or analyzing offline.

Capture Automatically When an Alert Fires​

Intermittent failures are hard to catch by hand: by the time someone reads the alert and starts a session, the request that triggered it is long gone. Let the alert start the session instead.

  1. Go to Alerts, open an alert policy, and find the Debug Sessions section.
  2. Add a configuration with a duration, a maximum number of samples, and an optional sampling rule.

When the policy fires, the control plane creates one session per configuration for each gateway group whose condition was met, targeting every instance in that group. The alert history records the session, and you can open it straight from the alert detail.

If a session created by the same policy is still active on that gateway group, no new one is created. A flapping alert therefore produces one capture window rather than hundreds, and the active session's duration acts as the cooldown.

Before You Run a Session in Production​

A trace records real traffic. Know what lands in it.

Captured: the full request URL including its query string. The request host, method, and response status. The User-Agent header. Route, service, and instance identifiers. Plugin and phase names with durations, and the request's own log lines.

Not captured: request bodies, response bodies, and any request or response header other than Host and User-Agent.

warning

Trace content is stored as captured, with no redaction. If your URLs carry sensitive values in the query string, or your plugins log sensitive values, that data appears in the trace and in exported files. Prefer a narrow sampling rule over capturing all traffic, and restrict the gateway:GetDebugSession and gateway:ExportDebugSession permissions.

Limits and Behavior​

PropertyValue
Session duration10 to 600 seconds
Max samples1 to 200 — caps how many traces the session lists, not how many are captured. Every request matching the rule is captured for the full duration.
Traffic scopeHTTP only
Concurrent sessionsNot limited
Trace retentionGoverned by your Jaeger deployment's retention settings

Each request is captured by at most one session. Sessions with a sampling rule are considered before sessions without one, and among equals the tie-break is arbitrary. When several people debug the same gateway group at once, give every session a sampling rule.

Stopping a session keeps the traces it collected. Deleting a session removes it and its trace list from the control plane, but the span data stays in Jaeger until your retention policy expires it. To remove captured content, use your Jaeger deployment's own controls.