Skip to main content

Version: 3.2.16.3

Log API Traffic

API7 Enterprise supports recording route access logs with information such as the host, client IP address, and request timestamp. These logs can help identify and troubleshoot issues.

API Enterprise integrates with many logging platforms through its plugin system. The following integrations are available out of the box:

  • HTTP/TCP/UDP Logging Server
  • SkyWalking
  • Kafka
  • RocketMQ
  • ClickHouse
  • Syslog
  • Aliyun SLS
  • Google Cloud Logging Service
  • Splunk HTTP Event Collector (HEC)
  • File logging
  • Elasticsearch
  • TencentCloud CLS
  • Grafana Loki

This tutorial guides you through configuring the clickhouse-logger plugin to log API traffic to a ClickHouse database.

Prerequisites

  1. Install API7 Enterprise.
  2. Have a running API on the gateway group.

Start a ClickHouse Server

Start a ClickHouse instance named quickstart-clickhouse-server with a default database quickstart_db, a default user quickstart-user, and password quickstart-pass in Docker:

docker run -d \
--name quickstart-clickhouse-server \
--network=api7-ee_api7 \
-e CLICKHOUSE_DB=quickstart_db \
-e CLICKHOUSE_USER=quickstart-user \
-e CLICKHOUSE_PASSWORD=quickstart-pass \
-e CLICKHOUSE_DEFAULT_ACCESS_MANAGEMENT=1 \
--ulimit nofile=262144:262144 \
clickhouse/clickhouse-server

Connect to the ClickHouse instance using the command line tool clickhouse-client locally or in Docker:

docker exec -it quickstart-clickhouse-server clickhouse-client

Create a table test in database quickstart_db with fields host, client_ip, route_id, and @timestamp of String type, or according to your needs:

CREATE TABLE quickstart_db.test (
`host` String,
`client_ip` String,
`route_id` String,
`@timestamp` String,
PRIMARY KEY(`@timestamp`)
) ENGINE = MergeTree()

If successful, you should see Ok on the output.

Enter exit to exit from the Docker container.

Configure Logging for All Services

Enabling the logging plugins in a global rule is strongly recommended to ensure all traffic to all services and routes is consistently tracked.

  1. Select Plugin Settings of your gateway group from the side navigation bar.
  2. Select the Plugin Global Rules tab, then click Enable Plugin.
  3. Search for the clickhouse-logger plugin in the Plugins field, then click Enable.
  4. In the dialog box do the following:
  • Apply the following configuration to the JSON Editor:

    {
    "log_format": {
    "host": "$host",
    "@timestamp": "$time_iso8601",
    "client_ip": "$remote_addr"
    },
    "user": "quickstart-user",
    "password": "quickstart-pass",
    "database": "quickstart_db",
    "logtable": "test",
    "endpoint_addrs": ["http://quickstart-clickhouse-server:8123"]
    }
  • Click Enable.

Validate

Send a request to the route to generate an access log entry:

curl -i "http://127.0.0.1:9080/ip"

Connect to the ClickHouse database using the command line tool clickhouse-client in Docker:

docker exec -it quickstart-clickhouse-server clickhouse-client

Query all records in table quickstart_db.test:

SELECT * from quickstart_db.test

You should see an access log entry similar to the following:

┌─host───────────┬─client_ip────┬─route_id─────────────────────────────┬─@timestamp────────────────┐
│ 127.0.0.1 │ 192.168.2.10 │ f4a6ec82-5b9d-4c2d-9d36-2d3a7cf1c3b4 │ 2024-07-18T19:24:16+00:00 │
└────────────────┴──────────────┴──────────────────────────────────────┴───────────────────────────┘

API7.ai Logo

API Management for Modern Architectures with Edge, API Gateway, Kubernetes, and Service Mesh.

Product

API7 Cloud

SOC2 Type IIISO 27001HIPAAGDPRRed Herring

Copyright © APISEVEN Ltd. 2019 – 2024. Apache, Apache APISIX, APISIX, and associated open source project names are trademarks of the

Apache Software Foundation