Log with Elasticsearch

Elasticsearch is a popular JSON-based datastore for storing and indexing large volumes of data. It is often used to store logs from various sources and works with tools like Logstash and Kibana to form an entire observability stack known as the Elastic (ELK) Stack.

APISIX supports forwarding its logs directly to Elasticsearch through the elasticsearch-logger plugin. These logs can then be searched, filtered, and visualized through Kibana to gather insights to manage applications.

This guide will show you how to enable the elasticsearch-logger plugin to integrate APISIX with the ELK stack for observability.

Prerequisite(s)

• Install Docker.
• Install cURL to send requests to the services for validation.
• Follow the Getting Started tutorial to start a new APISIX instance in Docker.

Start Elasticsearch and Kibana

info

APISIX currently supports Elasticsearch versions up to and including 7.x. This guide uses version 7.17.1 for both Elasticsearch and Kibana.

Start an Elasticsearch instance in Docker:

docker run -d \
  --name elasticsearch \
  --network apisix-quickstart-net \
  -v elasticsearch_vol:/usr/share/elasticsearch/data/ \
  -p 9200:9200 \
  -p 9300:9300 \
  -e ES_JAVA_OPTS="-Xms512m -Xmx512m" \
  -e discovery.type=single-node \
  -e xpack.security.enabled=false \
  docker.elastic.co/elasticsearch/elasticsearch:7.17.1

Start a Kibana instance in Docker to visualize the indexed data in Elasticsearch:

docker run -d \
  --name kibana \
  --network apisix-quickstart-net \
  -p 5601:5601 \
  -e ELASTICSEARCH_HOSTS="http://elasticsearch:9200" \
  docker.elastic.co/kibana/kibana:7.17.1

If successful, you should see the Kibana web dashboard on localhost:5601.

Enable elasticsearch-logger Plugin

Create a route to forward all requests to /ip to httpbin.org:

curl -i "http://127.0.0.1:9180/apisix/admin/routes" -X PUT -d '
{
  "id": "quickstart-client-ip",
  "uri": "/ip",
  "upstream": {
    "nodes": {
      "httpbin.org:80":1
    },
    "type": "roundrobin"
  }
}'

An HTTP/1.1 200 OK response verifies that the route is created successfully.

Enable the elasticsearch-logger plugin as a global rule for all routes, or on the route created above:

Global

Enable the elasticsearch-logger plugin on all routes:

curl "http://127.0.0.1:9180/apisix/admin/global_rules/" -X PUT -d '
{
  "id": "elasticsearch",
  "plugins": {
    "elasticsearch-logger": {
      "endpoint_addr": "http://elasticsearch:9200",
      "field": {
        "index": "gateway",
        "type": "logs"
      },
      "ssl_verify": false,
      "timeout": 60,
      "retry_delay": 1,
      "buffer_duration": 60,
      "max_retry_count": 0,
      "batch_max_size": 5,
      "inactive_timeout": 5
    }
  }
}'

Route

Enable the elasticsearch-logger plugin on a specific route:

curl "http://127.0.0.1:9180/apisix/admin/routes/quickstart-client-ip" -X PATCH -d '
{
  "plugins": {
    "elasticsearch-logger": {
      "endpoint_addr": "http://elasticsearch:9200",
      "field": {
        "index": "gateway",
        "type": "logs"
      },
      "ssl_verify": false,
      "timeout": 60,
      "retry_delay": 1,
      "buffer_duration": 60,
      "max_retry_count": 0,
      "batch_max_size": 5,
      "inactive_timeout": 5
    }
  }
}'

Customize Log Format

As an optional step, you can customize the log format for elasticsearch-logger. The log format of most APISIX logging plugins could be customized locally on the plugin (e.g. bound to a route) and/or globally with plugin metadata.

Add host address, timestamp, and client IP address to the logs with built-in variables:

curl "http://127.0.0.1:9180/apisix/admin/plugin_metadata/elasticsearch-logger" -X PUT -d '
{
  "plugins":{
    "elasticsearch-logger":{
      "log_format":{
        "host":"$host",
        "timestamp":"$time_iso8601",
        "client_ip":"$remote_addr"
      }
    }
  }
}'

Configure Kibana

Send some requests to the route to generate an access log entry:

for i in {1..10}; do
  curl -i "http://127.0.0.1:9080/ip"
done

Open Kibana dashboard on localhost:5601 and click the Discover tab from the menu. Create a new index pattern to fetch the data from Elasticsearch:

Create a pattern gateway to match the indexed data in Elasticsearch:

If your configuration is correct, you can go back to the discover page and view the logs from APISIX:

Next Steps

See elasticsearch-logger plugin reference to learn more about the plugin configuration options (coming soon).