Skip to main content

Make Log Collection with Kafka Logger

Learning the running status of your API Gateway cluster is necessary in the real world. It's helpful to know if the cluster is healthy or not so that you can take action in time and avoid fatal faults. Logs are one of the most valuable ways to monitor the API Gateway. It also can analyze your API requests (calculate the top N routes and status codes distribution).

API7 Cloud provides the Log Collection feature to configure "loggers" to collect log messages. With the help of Log Collection, you can transmit logs from Apache APISIX to different log servers ( Apache Kafka, ClickHouse and so on). This guide will show you how to use the Log Collection feature to send Apache APISIX logs to an Apache Kafka.

Prepare the Environment

Deploy Apache Kafka Cluster

We'll deploy a Kafka cluster with only one broker. You can skip this step if you already have an existing cluster.

docker run \
--name kafka \
--hostname kafka \
--env KAFKA_ENABLE_KRAFT=yes \
--env KAFKA_CFG_AUTO_CREATE_TOPICS_ENABLE=true \
--env KAFKA_CFG_PROCESS_ROLES=broker,controller \
--env KAFKA_CFG_LISTENERS=PLAINTEXT://:9092,CONTROLLER://:9093 \
--env KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT \
--env KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://kafka:9092 \
--env KAFKA_BROKER_ID=1 \
--env ALLOW_PLAINTEXT_LISTENER=yes \
--env KAFKA_CFG_CONTROLLER_QUORUM_VOTERS=1@127.0.0.1:9093 \
--env KAFKA_CFG_CONTROLLER_LISTENER_NAMES=CONTROLLER \
--network f20f3d77c6d7 \
bitnami/kafka:latest

Deploy Apache APISIX

Please refer to How to Deploy Apache APISIX to learn how to deploy Apache APISIX and connect it to API7 Cloud. In this guide, we'll deploy an Apache APISIX instance on Docker.

tip

When you deploy Apache APISIX with Docker, add a host mapping for the Kafka container.

--docker-run-arg --add-host=<Kafka Container IP>

You can run the command below to fetch the Kafka container address.

docker inspect kafka --format '{{ .NetworkSettings.Networks.bridge.IPAddress }}'

Configure Log Collection Plan

Let's enter the Log Collection page and click on the Create Log Collection button to create a new Log. Collection Plan.

Log Collection Page Create Log Collection

In this case, we create a Kafka logger, which sends log entries to the specified Kafka cluster. We use the default JSON format.

tip

Refer to Log Format in Log Collection to learn more about log format. Here we use the Default JSON Log Format.

Create Service and Route

We'll create a service with the following details in this guide.

  • The service name is kafka-httpbin.
  • The path prefix is /v1.
  • The protocol is HTTP.
  • The HTTP Host is kafka.httpbin.org.
  • The upstream URL is https://httpbin.org.

Besides, we'll create a route inside the kafka-httpbin Service.

  • The route name is json.
  • The path is /json (exact match).
  • Accepted HTTP method is GET.
tip

If you don't know how to configure a service and route, please refer to the Getting Started guides first

Add Logging Plugin

Now let's add the Logging plugin to the kafka-httpbin Service.

Add Logging Plugin

The Logging plugin binds some existing Log Collection Plans to the service or route. Here we bind the kafka-logger.

Send Some Requests

Now let's send some requests.

for ((i=0; i<5; i++)); do
curl http://127.0.0.1:9080/v1/json -H 'Host: kafka.httpbin.org' -v
done

Please wait a while, and let's use the kafka-console-consumer.sh to fetch log messages.

docker exec kafka  /opt/bitnami/kafka/bin/kafka-console-consumer.sh --topic apisix --from-beginning --bootstrap-server kafka:9092
[{"latency":4749.0000724792,"response":{"headers":{"access-control-allow-credentials":"true","server":"APISIX\/2.15.0","access-control-allow-origin":"*","content-type":"application\/json","x-apisix-upstream-status":"200","content-length":"429","connection":"close","date":"Mon, 15 Aug 2022 07:04:53 GMT"},"status":200,"size":688},"apisix_latency":3461.0000724792,"upstream_latency":1288,"upstream":"3.94.154.124:443","start_time":1660547089245,"client_ip":"172.17.0.1","route_id":"51ef3022","server":{"hostname":"apisix-fedora34","version":"2.15.0"},"request":{"size":88,"uri":"\/v1\/json","method":"GET","headers":{"accept":"*\/*","host":"kafka.httpbin.org","user-agent":"curl\/7.76.1"},"url":"http:\/\/kafka.httpbin.org:9080\/v1\/json","querystring":{}},"service_id":"421106273338000218"}]
[{"latency":1244.9998855591,"response":{"headers":{"connection":"close","access-control-allow-credentials":"true","server":"APISIX\/2.15.0","content-type":"application\/json","access-control-allow-origin":"*","content-length":"429","x-apisix-upstream-status":"200","date":"Mon, 15 Aug 2022 07:04:55 GMT"},"status":200,"size":688},"apisix_latency":0.99988555908203,"upstream_latency":1244,"upstream":"3.94.154.124:443","start_time":1660547093999,"client_ip":"172.17.0.1","route_id":"51ef3022","server":{"hostname":"apisix-fedora34","version":"2.15.0"},"request":{"size":88,"uri":"\/v1\/json","method":"GET","headers":{"accept":"*\/*","host":"kafka.httpbin.org","user-agent":"curl\/7.76.1"},"url":"http:\/\/kafka.httpbin.org:9080\/v1\/json","querystring":{}},"service_id":"421106273338000218"}]
[{"latency":1397.0000743866,"response":{"headers":{"x-apisix-upstream-status":"200","access-control-allow-origin":"*","server":"APISIX\/2.15.0","content-type":"application\/json","access-control-allow-credentials":"true","content-length":"429","connection":"close","date":"Mon, 15 Aug 2022 07:04:56 GMT"},"status":200,"size":688},"apisix_latency":1.0000743865967,"upstream_latency":1396,"upstream":"3.94.154.124:443","start_time":1660547095553,"client_ip":"172.17.0.1","route_id":"51ef3022","server":{"hostname":"apisix-fedora34","version":"2.15.0"},"request":{"size":88,"uri":"\/v1\/json","method":"GET","headers":{"accept":"*\/*","host":"kafka.httpbin.org","user-agent":"curl\/7.76.1"},"url":"http:\/\/kafka.httpbin.org:9080\/v1\/json","querystring":{}},"service_id":"421106273338000218"}]
[{"latency":292.99998283386,"response":{"headers":{"x-apisix-upstream-status":"200","access-control-allow-origin":"*","access-control-allow-credentials":"true","content-type":"application\/json","server":"APISIX\/2.15.0","content-length":"429","connection":"close","date":"Mon, 15 Aug 2022 07:04:55 GMT"},"status":200,"size":688},"apisix_latency":0,"upstream_latency":293,"upstream":"34.227.213.82:443","start_time":1660547095253,"client_ip":"172.17.0.1","route_id":"51ef3022","server":{"hostname":"apisix-fedora34","version":"2.15.0"},"request":{"size":88,"uri":"\/v1\/json","method":"GET","headers":{"accept":"*\/*","host":"kafka.httpbin.org","user-agent":"curl\/7.76.1"},"url":"http:\/\/kafka.httpbin.org:9080\/v1\/json","querystring":{}},"service_id":"421106273338000218"},{"latency":1385.999917984,"response":{"headers":{"x-apisix-upstream-status":"200","access-control-allow-origin":"*","access-control-allow-credentials":"true","content-type":"application\/json","server":"APISIX\/2.15.0","content-length":"429","connection":"close","date":"Mon, 15 Aug 2022 07:04:58 GMT"},"status":200,"size":688},"apisix_latency":0,"upstream_latency":1387,"upstream":"3.94.154.124:443","start_time":1660547096956,"client_ip":"172.17.0.1","route_id":"51ef3022","server":{"hostname":"apisix-fedora34","version":"2.15.0"},"request":{"size":88,"uri":"\/v1\/json","method":"GET","headers":{"accept":"*\/*","host":"kafka.httpbin.org","user-agent":"curl\/7.76.1"},"url":"http:\/\/kafka.httpbin.org:9080\/v1\/json","querystring":{}},"service_id":"421106273338000218"}]

A single and well-formatted log entry will be like this:

{
"latency": 4749.0000724792,
"response": {
"headers": {
"access-control-allow-credentials": "true",
"server": "APISIX/2.15.0",
"access-control-allow-origin": "*",
"content-type": "application/json",
"x-apisix-upstream-status": "200",
"content-length": "429",
"connection": "close",
"date": "Mon, 15 Aug 2022 07:04:53 GMT"
},
"status": 200,
"size": 688
},
"apisix_latency": 3461.0000724792,
"upstream_latency": 1288,
"upstream": "3.94.154.124:443",
"start_time": 1660547089245,
"client_ip": "172.17.0.1",
"route_id": "51ef3022",
"server": {
"hostname": "apisix-fedora34",
"version": "2.15.0"
},
"request": {
"size": 88,
"uri": "/v1/json",
"method": "GET",
"headers": {
"accept": "*/*",
"host": "kafka.httpbin.org",
"user-agent": "curl/7.76.1"
},
"url": "http://kafka.httpbin.org:9080/v1/json",
"querystring": {}
},
"service_id": "421106273338000218"
}

See Also


API7.ai Logo

API Management for Modern Architectures with Edge, API Gateway, Kubernetes, and Service Mesh.

Product

API7 Cloud

SOC2 Type IRed Herring

Copyright © APISEVEN Ltd. 2019 – 2023. Apache, Apache APISIX, APISIX, and associated open source project names are trademarks of the

Apache Software Foundation