Log API Traffic

API7 Enterprise supports collecting route access information and recording it as logs, such as host, client IP address, and request timestamp. The information helps troubleshoot related problems. A flexible plugin extension system and a number of existing plugins for logging are provided. For example:

• Push to HTTP/TCP/UDP Logging Server
• SkyWalking
• Kafka
• RocketMQ
• Clickhouse
• Syslog
• Aliyun SLS
• Google Cloud Logging Service
• Splunk HTTP Event Collector (HEC)
• Specific file on the disk.
• Elasticsearch
• TencentCloud CLS
• Grafana Loki

Prerequisites

1. Obtain a user account with Super Admin or Runtime Admin role.
2. Publish a service.
3. Get the host of your own ClickHouse database.
4. Install Docker.

Configure ClickHouse

1. Start a ClickHouse instance named quickstart-clickhouse-server with a default database quickstart_db, a default user quickstart-user and password quickstart-pass in Docker:

   docker network create apisix-quickstart-net
   
   docker run -d \
     --name quickstart-clickhouse-server \
     --network=apisix-quickstart-net \
     -e CLICKHOUSE_DB=quickstart_db \
     -e CLICKHOUSE_USER=quickstart-user \
     -e CLICKHOUSE_PASSWORD=quickstart-pass \
     -e CLICKHOUSE_DEFAULT_ACCESS_MANAGEMENT=1 \
     --ulimit nofile=262144:262144 \
     clickhouse/clickhouse-server

2. Connect to the ClickHouse instance using the command line tool clickhouse-client in Docker:

   docker exec -it quickstart-clickhouse-server clickhouse-client

3. Create a table test in database quickstart_db with fields host, client_ip, route_id, @timestamp of String type, or adjust the command accordingly based on your needs:

   CREATE TABLE quickstart_db.test (
     `host` String,
     `client_ip` String,
     `route_id` String,
     `@timestamp` String,
     PRIMARY KEY(`@timestamp`)
   ) ENGINE = MergeTree()

   If successful, you should see Ok on the output.

4. Enter exit to exit from the command line interface in Docker.

Configure Logging for All Services

For optimal monitoring and tracking, enabling the logging plugin as a global rule is strongly recommended to ensure that all services and routes are consistently tracked.

1. Select Gateway Groups from the side navigation bar and then select Test Group.

2. Select Plugin Settings from the side navigation bar.

3. Select the Plugin Global Rules tab and then search the clickhouse-logger plugin in the Plugins field.

4. Click the Plus icon (+) and a dialog box appears.

5. Apply the following configurations:

   {
     "log_format": {
       "host": "$host",
       "@timestamp": "$time_iso8601",
       "client_ip": "$remote_addr"
     },
     "user": "quickstart-user",
     "password": "quickstart-pass",
     "database": "quickstart_db",
     "logtable": "test",
     "endpoint_addrs": ["http://quickstart-clickhouse-server:8123"]
   }

6. Click Enable.

Validate

1. Send a request to the route to generate an access log entry:

   curl -i "http://127.0.0.1:9080/pet/1"  

2. Connect to the ClickHouse instance using the command line tool clickhouse-client in Docker:

   docker exec -it quickstart-clickhouse-server clickhouse-client

3. Query all records in table quickstart_db.test:

   SELECT * from quickstart_db.test

   You should see an access record similar to the following:

   ┌─host───────────┬─client_ip───┬─route_id─────────────────────────────┬─@timestamp────────────────┐
   │ 127.0.0.1      │ 172.19.0.15 │ 6433300c-311b-4047-800e-579244d42aa7 │ 2023-09-01T09:24:16+00:00 │
   └────────────────┴─────────────┴──────────────────────────────────────┴───────────────────────────┘