Use Debug Mode

APISIX provides a debug mode to help developers better understand and troubleshoot the runtime behavior of the gateway. The complete configuration options can be found in debug.yaml.

This guide will show you how to enable the debug mode to inspect what plugins are enabled on the requested route and how to add hooks to log input arguments and returned values of APISIX module's functions.

Prerequisite(s)

• Install Docker.
• Install cURL to send requests to APISIX for validation.
• Follow the Getting Started tutorial to start a new APISIX instance in Docker.

Enable Basic Debug Mode

Enabling the basic debug mode will allow you to inspect what plugins are enabled on the requested route in the Apisix-Plugins header.

By default, the debug mode is turned off. To enable the debug mode, you can set basic.enable to true in your debug.yaml file.

Docker

docker exec apisix-quickstart /bin/sh -c "echo '
basic:
  enable: true
#END
' > /usr/local/apisix/conf/debug.yaml"

The debug.yaml file is loaded into memory at startup and monitored for changes at a regular interval, so that you do not need to manually reload APISIX after updating the file.

Kubernetes

Create a ConfigMap:

apisix-debug-cm.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: apisix-debug
  namespace: ingress-apisix
data:
  debug.yaml: |
    basic:
      enable: true
    #END

Apply the ConfigMap to your cluster:

kubectl apply -f apisix-debug-cm.yaml

Patch the APISIX deployment:

kubectl patch deployment apisix \
  --type='strategic' \
  -p '{
    "spec": {
      "template": {
        "spec": {
          "volumes": [
            {
              "name": "debug-config",
              "configMap": {
                "name": "apisix-debug"
              }
            }
          ],
          "containers": [
            {
              "name": "apisix",
              "volumeMounts": [
                {
                  "name": "debug-config",
                  "mountPath": "/usr/local/apisix/conf/debug.yaml",
                  "subPath": "debug.yaml"
                }
              ]
            }
          ]
        }
      }
    }
  }'

Restart the deployment:

kubectl rollout restart deployment apisix

info

The configurations should end with #END, or else APISIX will not load the configurations.

You can also use the #END flag as a breakpoint for APISIX to only load configurations up to the specified point.

To verify, create a route without any plugin:

Admin API

curl -i "http://127.0.0.1:9180/apisix/admin/routes" -X PUT -d '
{
  "id": "getting-started-anything",
  "uri": "/anything",
  "upstream": {
    "type": "roundrobin",
    "nodes": {
      "httpbin.org:80": 1
    }
  }
}'

ADC

adc.yaml

services:
  - name: httpbin Service
    routes:
      - uris:
          - /anything
        name: getting-started-anything
    upstream:
      type: roundrobin
      nodes:
        - host: httpbin.org
          port: 80
          weight: 1

Synchronize the configuration to APISIX:

adc sync -f adc.yaml

Ingress Controller

Gateway API

httpbin-route.yaml

apiVersion: v1
kind: Service
metadata:
  namespace: ingress-apisix
  name: httpbin-external-domain
spec:
  type: ExternalName
  externalName: httpbin.org
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  namespace: ingress-apisix
  name: getting-started-anything
spec:
  parentRefs:
  - name: apisix
  rules:
  - matches:
    - path:
        type: Exact
        value: /anything
    backendRefs:
    - name: httpbin-external-domain
      port: 80

APISIX CRD

httpbin-route.yaml

apiVersion: apisix.apache.org/v2
kind: ApisixUpstream
metadata:
  namespace: ingress-apisix
  name: httpbin-external-domain
spec:
  externalNodes:
  - type: Domain
    name: httpbin.org
---
apiVersion: apisix.apache.org/v2
kind: ApisixRoute
metadata:
  namespace: ingress-apisix
  name: getting-started-anything
spec:
  ingressClassName: apisix
  http:
    - name: getting-started-anything
      match:
        paths:
          - /anything
      upstreams:
      - name: httpbin-external-domain

Apply the configuration to your cluster:

kubectl apply -f httpbin-route.yaml

Send a request to the route:

curl -i "http://127.0.0.1:9080/anything"

You should receive an HTTP/1.1 200 OK response and observe the following headers:

Content-Type: application/json
Content-Length: 390
Connection: keep-alive
Apisix-Plugins: no plugin
...

Admin API

Update the route with a plugin:

curl -i "http://127.0.0.1:9180/apisix/admin/routes/getting-started-anything" -X PATCH -d '
{
  "plugins": {
    "limit-count": {
      "count": 5,
      "time_window": 10,
      "rejected_code": 429
    }
  }
}'

Additionally, add a global plugin:

curl -i "http://127.0.0.1:9180/apisix/admin/global_rules" -X PUT -d '{
  "id": "global-prometheus",
  "plugins": {
    "prometheus":{}
  }
}'

ADC

Update the route with a plugin and add another global plugin:

adc.yaml

services:
  - name: httpbin Service
    routes:
      - uris:
          - /anything
        name: getting-started-anything
        plugins:
          limit-count:
            count: 5
            time_window: 10
            rejected_code: 429
    upstream:
      type: roundrobin
      nodes:
        - host: httpbin.org
          port: 80
          weight: 1
global_rules:
  prometheus: {}

Synchronize the configuration to APISIX:

adc sync -f adc.yaml

Ingress Controller

Gateway API

Update the route manifest file with a plugin:

httpbin-route.yaml

apiVersion: v1
kind: Service
metadata:
  namespace: ingress-apisix
  name: httpbin-external-domain
spec:
  type: ExternalName
  externalName: httpbin.org
---
apiVersion: apisix.apache.org/v1alpha1
kind: PluginConfig
metadata:
  namespace: ingress-apisix
  name: limit-count-plugin-config
spec:
  plugins:
    - name: limit-count
      config:
        count: 5
        time_window: 10
        rejected_code: 429
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  namespace: ingress-apisix
  name: getting-started-anything
spec:
  parentRefs:
  - name: apisix
  rules:
  - matches:
    - path:
        type: Exact
        value: /anything
    filters:
    - type: ExtensionRef
      extensionRef:
        group: apisix.apache.org
        kind: PluginConfig
        name: limit-count-plugin-config
    backendRefs:
    - name: httpbin-external-domain
      port: 80

Additionally, create a Kubernetes manifest file for a global Prometheus plugin:

global-prometheus.yaml

apiVersion: apisix.apache.org/v1alpha1
kind: GatewayProxy
metadata:
  namespace: ingress-apisix
  name: apisix-config
spec:
  plugins:
  - name: prometheus
    enabled: true

APISIX CRD

Update the route with a plugin:

httpbin-route.yaml

apiVersion: apisix.apache.org/v2
kind: ApisixUpstream
metadata:
  namespace: ingress-apisix
  name: httpbin-external-domain
spec:
  externalNodes:
  - type: Domain
    name: httpbin.org
---
apiVersion: apisix.apache.org/v2
kind: ApisixRoute
metadata:
  namespace: ingress-apisix
  name: getting-started-anything
spec:
  ingressClassName: apisix
  http:
    - name: getting-started-anything
      match:
        paths:
          - /anything
      upstreams:
      - name: httpbin-external-domain
      plugins:
      - name: limit-count
        enable: true
        config:
          count: 5
          time_window: 10
          rejected_code: 429

Additionally, create a Kubernetes manifest file for a global Prometheus plugin:

global-prometheus.yaml

apiVersion: apisix.apache.org/v2
kind: ApisixGlobalRule
metadata:
  namespace: ingress-apisix
  name: global-prometheus
spec:
  ingressClassName: apisix
  plugins:
    - name: prometheus
      enable: true

Apply the configuration to your cluster:

kubectl apply -f httpbin-route.yaml -f global-prometheus.yaml

Send another request to the route:

curl -i "http://127.0.0.1:9080/anything"

You should receive an HTTP/1.1 200 OK response with headers similar to the following:

Content-Type: application/json
Content-Length: 388
Connection: keep-alive
X-RateLimit-Limit: 5
X-RateLimit-Remaining: 4
X-RateLimit-Reset: 10
Apisix-Plugins: limit-count, prometheus
...

info

If the plugin information cannot be included in a response header (e.g. L4 stream plugins), the debug information will be logged in the error log at the warn severity level.

Configure Advanced Debug Mode

There are a few other advanced options in debug.yaml that you can configure.

Log Function Input Arguments and Returned Values

You can add hooks to APISIX phases to log input arguments and return values with the following configuration:

debug.yaml

basic:
  enable: true
hook_conf:
  enable: true
  name: hook_phase
  log_level: warn
  is_print_input_args: true
  is_print_return_value: true
hook_phase:
  apisix:
    - http_access_phase
    - http_header_filter_phase
    - http_body_filter_phase
    - http_log_phase
#END

❶ Enable hook debug trace to log the target module function's input arguments or returned values.

❷ Name of module and function list.

❸ Severity level for input arguments and returned values in the error log.

❹ Log the input arguments.

❺ Log the return values.

❻ Name of module and function list.

To verify, send a request to the route:

curl -i "http://127.0.0.1:9080/anything"

You should receive an HTTP/1.1 200 OK response and see the following information in the error log:

call require("apisix").http_access_phase() args:{}
call require("apisix").http_access_phase() return:{}
call require("apisix").http_header_filter_phase() args:{}
call require("apisix").http_header_filter_phase() return:{}
call require("apisix").http_body_filter_phase() args:{}
call require("apisix").http_body_filter_phase() return:{}
call require("apisix").http_log_phase() args:{}
call require("apisix").http_log_phase() return:{}

Apply Advanced Debug Settings Dynamically

You can also log function input arguments and returned values dynamically when a certain header is present with the following configuration:

debug.yaml

docker exec apisix-quickstart /bin/sh -c "echo '
basic:
  enable: true
http_filter:
  enable: true
  enable_header_name: X-APISIX-Dynamic-Debug
hook_conf:
  enable: true
  name: hook_phase
  log_level: warn
  is_print_input_args: true
  is_print_return_value: true
hook_phase:
  apisix:
    - http_access_phase
    - http_header_filter_phase
    - http_body_filter_phase
    - http_log_phase
#END

❶ Enable HTTP filter to dynamically apply advanced debug settings.

❷ Log input arguments and returned values only when the request has the X-APISIX-Dynamic-Debug header.

To verify, send a request to the route without any header:

curl -i "http://127.0.0.1:9080/anything"

You should receive an HTTP/1.1 200 OK response but observe no debugging information in the error log.

Send another request with the debug header:

curl -i "http://127.0.0.1:9080/anything" -H "X-APISIX-Dynamic-Debug: 1"

You should receive an HTTP/1.1 200 OK response and see the following information in the error log:

call require("apisix").http_access_phase() args:{}
call require("apisix").http_access_phase() return:{}
call require("apisix").http_header_filter_phase() args:{}
call require("apisix").http_header_filter_phase() return:{}
call require("apisix").http_body_filter_phase() args:{}
call require("apisix").http_body_filter_phase() return:{}
call require("apisix").http_log_phase() args:{}
call require("apisix").http_log_phase() return:{}

Manage debug.yaml by Environments

You may wish to have different debug.yaml files for each environment, such as using conf/debug-dev.yaml for development and conf/debug-prod.yaml for production. This can be done by setting the APISIX_PROFILE variable.

For more details, see Manage Configuration Files by Environments.

Next Steps

You have now learned how to leverage the debug mode in APISIX for troubleshooting and the various debugging options it offers.

You can find the complete configuration options in debug.yaml.