Log API Traffic

API7 Enterprise supports recording route access logs with information such as the host, client IP address, and request timestamp. These logs can help identify and troubleshoot issues.

API Enterprise integrates with many logging platforms through its plugin system. The following integrations are available out of the box:

• HTTP/TCP/UDP Logging Server
• SkyWalking
• Kafka
• RocketMQ
• ClickHouse
• Syslog
• Aliyun SLS
• Google Cloud Logging Service
• Splunk HTTP Event Collector (HEC)
• File logging
• Elasticsearch
• TencentCloud CLS
• Grafana Loki

This tutorial guides you through configuring the clickhouse-logger plugin to log API traffic to a ClickHouse database.

Prerequisites

1. Install API7 Enterprise.
2. Have a running API on the gateway group.

Start a ClickHouse Server

Docker

Start a ClickHouse instance named quickstart-clickhouse-server with a default database quickstart_db, a default user quickstart-user, and password quickstart-pass in Docker:

docker run -d \
  --name quickstart-clickhouse-server \
  --network=api7-ee_api7 \
  -e CLICKHOUSE_DB=quickstart_db \
  -e CLICKHOUSE_USER=quickstart-user \
  -e CLICKHOUSE_PASSWORD=quickstart-pass \
  -e CLICKHOUSE_DEFAULT_ACCESS_MANAGEMENT=1 \
  --ulimit nofile=262144:262144 \
  clickhouse/clickhouse-server

Connect to the ClickHouse instance using the command line tool clickhouse-client locally or in Docker:

docker exec -it quickstart-clickhouse-server clickhouse-client

Create a table test in database quickstart_db with fields host, client_ip, route_id, and @timestamp of String type, or according to your needs:

CREATE TABLE quickstart_db.test (
  `host` String,
  `client_ip` String,
  `route_id` String,
  `@timestamp` String,
  PRIMARY KEY(`@timestamp`)
) ENGINE = MergeTree()

If successful, you should see Ok on the output.

Enter exit to exit from the Docker container.

Kubernetes

Install ClickHouse Operator

If you have installed the gateway instance on Kubernetes and use Ingress Controller for configurations, install the ClickHouse Operator:

curl -s "https://raw.githubusercontent.com/Altinity/clickhouse-operator/master/deploy/operator-web-installer/clickhouse-operator-install.sh" | OPERATOR_NAMESPACE=<YOUR_NAMESPACE> bash 

Check all resources:

kubectl get all

You should see the following ClickHouse operator resources:

NAME                                             READY   STATUS    RESTARTS   AGE
pod/clickhouse-operator-56c57f989-kjqtn          2/2     Running   0          32m

NAME                                             TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
service/clickhouse-operator-metrics              ClusterIP   10.96.253.156   <none>        8888/TCP,9999/TCP            32m

NAME                                             READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/clickhouse-operator              1/1     1            1           32m

NAME                                             DESIRED   CURRENT   READY   AGE
replicaset.apps/clickhouse-operator-56c57f989    1         1         1       32m

Install ClickHouse Server

Create a manifest file for a simple 1 shard 1 replica installation:

clickhouse-installation.yaml

apiVersion: clickhouse.altinity.com/v1
kind: ClickHouseInstallation
metadata:
  name: clickhouse-installation
  # namespace: api7    # replace with your namespace
spec:
  configuration:
    users:
      test_user/password_sha256_hex: c019743ba8bc32d17800bca6d36de903feff9854897c61b6e63ce3e5e93c37ca
      test_user/password: quickstart-pass
      test_user/networks/ip:
        - 0.0.0.0/0
    clusters:
      - name: simple

Apply the configuration to your cluster:

kubectl apply -f clickhouse-installation.yaml

Check running pods:

kubectl get pod

You should see the ClickHouse server pod running:

NAME                                                    READY   STATUS    RESTARTS   AGE
chi-clickhouse-installation-simple-0-0-0                1/1     Running   0          11m

Configure ClickHouse Server

Connect to the ClickHouse instance:

kubectl exec -it chi-clickhouse-installation-simple-0-0-0 -- clickhouse-client

Create a database:

CREATE DATABASE quickstart_db

Create a table test in database quickstart_db with fields host, client_ip, route_id, and @timestamp of String type, or according to your needs:

CREATE TABLE quickstart_db.test (
  `host` String,
  `client_ip` String,
  `route_id` String,
  `@timestamp` String,
  PRIMARY KEY(`@timestamp`)
) ENGINE = MergeTree()

If successful, you should see Ok on the output.

Enter exit to exit from the ClickHouse server.

Configure Logging for All Services

Enabling the logging plugins in a global rule is strongly recommended to ensure all traffic to all services and routes is consistently tracked.

Dashboard

1. Select Plugin Settings of your gateway group from the side navigation bar.
2. Select the Plugin Global Rules tab, then click Enable Plugin.
3. Search for the clickhouse-logger plugin in the Plugins field, then click Enable.
4. In the dialog box do the following:

• Apply the following configuration to the JSON Editor:

  {
    "log_format": {
      "host": "$host",
      "@timestamp": "$time_iso8601",
      "client_ip": "$remote_addr"
    },
    "user": "quickstart-user",
    "password": "quickstart-pass",
    "database": "quickstart_db",
    "logtable": "test",
    "endpoint_addrs": ["http://quickstart-clickhouse-server:8123"]
  }

• Click Enable.

ADC

To use ADC to enable logging, create the following configuration:

adc.yaml

global_rules:
  clickhouse-logger:
    _meta:
      disable: false
    database: quickstart_db
    endpoint_addrs:
      - http://quickstart-clickhouse-server:8123
    log_format:
      "@timestamp": $time_iso8601
      client_ip: $remote_addr
      host: $host
    logtable: test
    password: quickstart-pass
    user: quickstart-user

Synchronize the configuration to API7 Enterprise:

adc sync -f adc.yaml

note

ADC uses the configuration files as the single source of truth. Make sure to pass all configuration files to the adc sync command using the -f flag.

Ingress Controller

Create a manifest file to enable the Prometheus plugin globally for all routes:

global-observability.yaml

apiVersion: apisix.apache.org/v2
kind: ApisixGlobalRule
metadata:
  name: global-observability
  # namespace: api7    # replace with your namespace
spec:
  plugins:
  - name: clickhouse-logger
    enable: true
    config:
      endpoint_addr: http://clickhouse-clickhouse-installation.api7.svc.cluster.local:8123
      user: quickstart-user
      password: quickstart-pass
      logtable: test
      database: quickstart_db
      log_format:
        "@timestamp": $time_iso8601
        client_ip: $remote_addr
        host: $host

Apply the configuration to your cluster:

kubectl apply -f global-observability.yaml

Validate

Send a request to the route to generate an access log entry:

curl -i "http://127.0.0.1:9080/ip"

Docker

Connect to the ClickHouse database using the command line tool clickhouse-client in Docker:

docker exec -it quickstart-clickhouse-server clickhouse-client

Kubernetes

Connect to the ClickHouse database:

kubectl exec -it chi-clickhouse-installation-simple-0-0-0 -- clickhouse-client

Query all records in table quickstart_db.test:

SELECT * from quickstart_db.test

You should see an access log entry similar to the following:

┌─host───────────┬─client_ip────┬─route_id─────────────────────────────┬─@timestamp────────────────┐
│ 127.0.0.1      │ 192.168.2.10 │ f4a6ec82-5b9d-4c2d-9d36-2d3a7cf1c3b4 │ 2024-07-18T19:24:16+00:00 │
└────────────────┴──────────────┴──────────────────────────────────────┴───────────────────────────┘