Skip to main content
API7.ai Logo

error-log-logger

The error-log-logger plugin sends APISIX and API7 Gateway error logs (error.log) to TCP, Apache SkyWalking, Apache Kafka, or ClickHouse servers in batches. Configure a severity threshold to send entries at that level and every more severe level.

The plugin is disabled by default. Once enabled, it will automatically start pushing error logs to remote servers. You should configure remote server details in plugin metadata only, instead of on other resources, such as routes.

Examples​

The examples below demonstrate how you can configure error-log-logger plugin for different scenarios.

The APISIX and API7 Gateway runtime configurations do not load error-log-logger by default. Enable it in the gateway static configuration before configuring plugin metadata.

Keep the existing plugin list in config.yaml and add error-log-logger:

config.yaml
plugins:
  # Keep the complete plugin list used by your gateway.
  - error-log-logger

Reload the gateway for changes to take effect.

Send Logs to a TCP Server​

The following example sends gateway error logs to a TCP server.

Start a TCP server listening on port 19000:

Set the running APISIX or API7 Gateway container name:

export GATEWAY_CONTAINER=replace-with-gateway-container-name

Create a dedicated network:

docker network create gateway-error-log-net

Connect the gateway to the network:

docker network connect gateway-error-log-net "$GATEWAY_CONTAINER"

Start a pinned TCP receiver on the network:

docker run -d \
  --name error-log-tcp \
  --network gateway-error-log-net \
  alpine/socat:1.8.1.3 \
  -u TCP-LISTEN:19000,fork,reuseaddr STDOUT

Configure the TCP destination in the plugin metadata for error-log-logger:

curl "http://127.0.0.1:9180/apisix/admin/plugin_metadata/error-log-logger" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "tcp": {
      "host": "error-log-tcp",
      "port": 19000
    },
    "level": "WARN",
    "batch_max_size": 1
  }'

❶ Configure the receiver hostname reachable from the gateway.

❷ Configure the receiver's TCP listening port.

❸ Forward warning, error, critical, alert, and emergency entries.

The example sets batch_max_size to 1 so each matching error-log entry is forwarded immediately for verification.

The following deterministic verification uses Docker with the Admin API and proxy available on 127.0.0.1. If the gateway runs in Kubernetes, generate a warning or more severe error through the deployment's normal route workflow, then use the Kubernetes log command below.

For the Docker verification, create a route whose upstream points to a closed port:

curl "http://127.0.0.1:9180/apisix/admin/routes/error-log-tcp-source" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d '{
    "uri": "/error-log-tcp",
    "upstream": {
      "type": "roundrobin",
      "nodes": {
        "127.0.0.1:1": 1
      }
    }
  }'

Send a request to trigger an upstream connection error:

curl -i "http://127.0.0.1:9080/error-log-tcp"

You should receive an HTTP/1.1 502 Bad Gateway response.

View the forwarded error:

docker logs error-log-tcp

The receiver should contain an entry similar to the following:

2026/09/21 12:02:40 [error] 358#358: *5675628 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.155.1, server: _, request: "GET /error-log-tcp HTTP/1.1", upstream: "http://127.0.0.1:1/error-log-tcp", host: "127.0.0.1:9080", request_id: "0e772dc645823636907d5b3be972c42f"

Send Logs to SkyWalking​

The following example configures the error-log-logger plugin to send gateway error logs to SkyWalking OAP 11.0.0 with BanyanDB 0.11.0 and Horizon 1.0.0.

Set up SkyWalking OAP server:

Create the network used by the SkyWalking containers:

docker network create gateway-skywalking-net

If the gateway also runs in Docker, set GATEWAY_CONTAINER and connect it to the network. Skip this step for a host-installed gateway:

export GATEWAY_CONTAINER=replace-with-gateway-container-name
docker network connect gateway-skywalking-net "$GATEWAY_CONTAINER"

Create the following Docker Compose file:

skywalking-compose.yaml
services:
  banyandb:
    image: apache/skywalking-banyandb:0.11.0
    command: standalone
    networks:
      - skywalking

  oap:
    image: apache/skywalking-oap-server:11.0.0
    environment:
      SW_STORAGE: banyandb
      SW_STORAGE_BANYANDB_TARGETS: banyandb:17912
    ports:
      - "127.0.0.1:12800:12800"
    depends_on:
      - banyandb
    networks:
      skywalking:
        aliases:
          - skywalking-oap

  horizon:
    image: apache/skywalking-ui:horizon-1.0.0
    environment:
      HORIZON_OAP_QUERY_URL: http://skywalking-oap:12800
      HORIZON_OAP_ADMIN_URL: http://skywalking-oap:17128
      HORIZON_AUTH_LOCAL_USERS: '[{"username":"admin","passwordHash":"$$argon2id$$v=19$$m=65536,t=3,p=4$$eemqy1r72oSXR58y8VpRqw$$Bn/dULrmJTHEi3263KfgWDEwQmUsqNLi3xwyv/DekHM","roles":["admin"]}]'
    ports:
      - "127.0.0.1:8081:8081"
    depends_on:
      - oap
    networks:
      - skywalking

networks:
  skywalking:
    name: gateway-skywalking-net
    external: true

Start the services:

docker compose -f skywalking-compose.yaml up -d

Horizon is available at http://localhost:8081. Sign in with username admin and password admin.

caution

The local user in this example uses public demonstration credentials. Use it only for a trusted local evaluation. Configure an identity provider or generate a unique password hash before exposing Horizon outside the local environment.

Configure the SkyWalking destination in the plugin metadata for error-log-logger:

For a host-installed gateway, use http://127.0.0.1:12800/v3/logs.

curl "http://127.0.0.1:9180/apisix/admin/plugin_metadata/error-log-logger" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "skywalking": {
      "endpoint_addr": "http://skywalking-oap:12800/v3/logs"
    },
    "level": "INFO"
  }'

❶ Configure the endpoint address to the SkyWalking server.

❷ Configure the severity level to INFO so most logs would be sent, for easier verification.

To verify, you can manually generate a log at warn level by reloading APISIX.

In Horizon, navigate to General Service → Logs, select the APISIX service, and run a query. You should see the generated warning log:

2025/01/27 07:40:06 [warn] 211#211: *35552 [lua] plugin.lua:205: load(): new plugins: {"cas-auth":true,"real-ip":true,"ai":true,"client-control":true,"proxy-control":true,"request-id":true,"zipkin":true,"ext-plugin-pre-req":true,"fault-injection":true,"mocking":true,"serverless-pre-function":true,"cors":true,"ip-restriction":true,"ua-restriction":true,"referer-restriction":true,"csrf":true,"uri-blocker":true,"request-validation":true,"chaitin-waf":true,"multi-auth":true,"openid-connect":true,"authz-casbin":true,"authz-casdoor":true,"wolf-rbac":true,"ldap-auth":true,"hmac-auth":true,"basic-auth":true,"jwt-auth":true,"redirect":true,"key-auth":true,"consumer-restriction":true,"attach-consumer-label":true,"authz-keycloak":true,"proxy-cache":true,"body-transformer":true,"ai-prompt-template":true,"ai-prompt-decorator":true,"proxy-mirror":true,"proxy-rewrite":true,"workflow":true,"api-breaker":true,"ai-proxy":true,"limit-conn":true,"limit-count":true,"limit-req":true,"gzip":true,"server-info":true,"traffic-split":true,"response-rewrite":true,"degraphql":true,"kafka-proxy":true,"grpc-transcode":true,"grpc-web":true,"http-dubbo":true,"public-api":true,"prometheus":true,"datadog":true,"loki-logger":true,"elasticsearch-logger":true,"echo":true,"loggly":true,"http-logger":true,"splunk-hec-logging":true,"skywalking-logger":true,"google-cloud-logging":true,"sls-logger":true,"tcp-logger":true,"kafka-logger":true,"rocketmq-logger":true,"syslog":true,"udp-logger":true,"file-logger":true,"clickhouse-logger":true,"tencent-cloud-cls":true,"inspect":true,"example-plugin":true,"aws-lambda":true,"azure-functions":true,"openwhisk":true,"openfunction":true,"error-log-logger":true,"ext-plugin-post-req":true,"ext-plugin-post-resp":true,"serverless-post-function":true,"opa":true,"forward-auth":true,"jwe-decrypt":true}, context: init_worker_by_lua*

The timestamp, worker identifiers, and loaded plugin set vary by deployment.

You should also observe logs at other severity levels, such as error, emerg, and info, when they are generated.

Send Logs to Kafka over TLS​

The following example sends error-level gateway logs to a TLS-enabled Kafka broker. Complete the trusted-CA setup in Send Logs to a TLS-Enabled Broker, then set the broker address and topic:

export KAFKA_TLS_HOST="kafka-tls"
export KAFKA_TLS_PORT="9093"
export KAFKA_ERROR_TOPIC="apisix-error-logs"

Create the dedicated error-log topic:

docker exec kafka-tls /opt/kafka/bin/kafka-topics.sh \
  --bootstrap-server kafka-tls:9093 \
  --command-config /etc/kafka/secrets/client.properties \
  --create \
  --if-not-exists \
  --topic "${KAFKA_ERROR_TOPIC}" \
  --partitions 1 \
  --replication-factor 1

Configure the plugin metadata:

curl "http://127.0.0.1:9180/apisix/admin/plugin_metadata/error-log-logger" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  --data-binary @- <<EOF
{
  "kafka": {
    "brokers": [
      {
        "host": "${KAFKA_TLS_HOST}",
        "port": ${KAFKA_TLS_PORT}
      }
    ],
    "kafka_topic": "${KAFKA_ERROR_TOPIC}",
    "tls": {
      "verify": true
    }
  },
  "level": "ERROR",
  "inactive_timeout": 1
}
EOF

To generate an error for verification, create a route whose upstream port is closed, then send a request:

curl "http://127.0.0.1:9180/apisix/admin/routes/trigger-error-log" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "uri": "/trigger-error-log",
    "upstream": {
      "type": "roundrobin",
      "nodes": {
        "127.0.0.1:1": 1
      }
    }
  }'

curl -i "http://127.0.0.1:9080/trigger-error-log"

The request should return 502 Bad Gateway. After the one-second batch timeout, consume one record from the dedicated topic:

docker exec kafka-tls /opt/kafka/bin/kafka-console-consumer.sh \
  --bootstrap-server kafka-tls:9093 \
  --topic "${KAFKA_ERROR_TOPIC}" \
  --consumer.config /etc/kafka/secrets/client.properties \
  --from-beginning \
  --max-messages 1

The record should contain an error entry similar to the following:

connect() failed (111: Connection refused) while connecting to upstream

Kafka broker information written by the plugin to its own logs contains only host and port; SASL passwords are redacted.

Send Logs to ClickHouse​

The following example configures error-log-logger to send gateway error logs to ClickHouse.

Local evaluation credentials

The example uses an HTTP connection and a fixed local password. For production, use a secured ClickHouse endpoint and protect plugin metadata and its encryption keys according to the gateway deployment's credential-management policy.

Start ClickHouse with a dedicated database and user:

Set GATEWAY_CONTAINER to the running APISIX or API7 Gateway container. Create a dedicated network and connect the gateway to it:

export GATEWAY_CONTAINER=replace-with-gateway-container-name

docker network create gateway-error-log-clickhouse-net
docker network connect gateway-error-log-clickhouse-net "$GATEWAY_CONTAINER"

Start ClickHouse on the same network:

docker run -d \
  --name error-log-clickhouse \
  --network gateway-error-log-clickhouse-net \
  -p 127.0.0.1:8123:8123 \
  -e CLICKHOUSE_DB=apisix_logs \
  -e CLICKHOUSE_USER=apisix_logger \
  -e CLICKHOUSE_PASSWORD=apisix-logger-pass \
  -e CLICKHOUSE_DEFAULT_ACCESS_MANAGEMENT=1 \
  --ulimit nofile=262144:262144 \
  clickhouse/clickhouse-server:26.8.5.13

Wait for ClickHouse to become ready:

until curl -fsS -u "apisix_logger:apisix-logger-pass" \
  "http://127.0.0.1:8123/ping" | grep -q "Ok"; do
  sleep 2
done

Create the error_logs table. The plugin writes each error-log entry to its data column:

curl "http://127.0.0.1:8123" \
  -u "apisix_logger:apisix-logger-pass" \
  -d '
  CREATE TABLE apisix_logs.error_logs (
    data String,
    PRIMARY KEY(`data`)
  )
  ENGINE = MergeTree()
'

Configure the ClickHouse destination in the plugin metadata for error-log-logger:

curl "http://127.0.0.1:9180/apisix/admin/plugin_metadata/error-log-logger" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "clickhouse": {
      "endpoint_addr": "http://error-log-clickhouse:8123",
      "user": "apisix_logger",
      "password": "apisix-logger-pass",
      "database": "apisix_logs",
      "logtable": "error_logs"
    },
    "level": "INFO",
    "batch_max_size": 1
  }'

level: INFO captures informational messages and every more severe level, while batch_max_size: 1 sends each entry immediately for this verification.

Create a route with an unavailable upstream so that the gateway writes a predictable error entry:

curl "http://127.0.0.1:9180/apisix/admin/routes/clickhouse-error-source" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "uri": "/clickhouse-error",
    "upstream": {
      "type": "roundrobin",
      "nodes": {
        "127.0.0.1:1": 1
      }
    }
  }'

Send a request. The unavailable upstream should produce an HTTP/1.1 502 Bad Gateway response and an error-log entry:

curl -i "http://127.0.0.1:9080/clickhouse-error"

Send a request to ClickHouse to see the log entries:

echo "SELECT substring(data, 1, 220) AS data FROM apisix_logs.error_logs WHERE position(data, 'clickhouse-error') > 0 ORDER BY data DESC LIMIT 1 FORMAT JSONEachRow" | \
  curl "http://127.0.0.1:8123/?" \
  -u "apisix_logger:apisix-logger-pass" \
  --data-binary @-

You should see a log entry similar to the following:

{"data":"2026/09/16 12:26:17 [error] ... connect() failed (111: Connection refused) while connecting to upstream ... request: \"GET /clickhouse-error HTTP/1.1\" ..."}