Log with Elasticsearch

Elasticsearch is a popular JSON-based datastore for storing and indexing large volumes of data. It is often used to store logs from various sources and works with tools like Logstash and Kibana to form an entire observability stack known as the Elastic (ELK) Stack.

APISIX supports forwarding its logs directly to Elasticsearch through the elasticsearch-logger plugin. These logs can then be searched, filtered, and visualized through Kibana to gather insights to manage applications.

This guide will show you how to enable the elasticsearch-logger plugin to integrate APISIX with the ELK stack for observability.

Prerequisite(s)

• Install Docker.
• Install cURL to send requests to the services for validation.
• Follow the Getting Started tutorial to start a new APISIX instance in Docker.

Start Elasticsearch and Kibana

info

APISIX currently supports Elasticsearch versions up to and including 7.x. This guide uses version 7.17.1 for both Elasticsearch and Kibana.

Start an Elasticsearch instance in Docker:

docker run -d \
  --name elasticsearch \
  --network apisix-quickstart-net \
  -v elasticsearch_vol:/usr/share/elasticsearch/data/ \
  -p 9200:9200 \
  -p 9300:9300 \
  -e ES_JAVA_OPTS="-Xms512m -Xmx512m" \
  -e discovery.type=single-node \
  -e xpack.security.enabled=false \
  docker.elastic.co/elasticsearch/elasticsearch:7.17.1

Start a Kibana instance in Docker to visualize the indexed data in Elasticsearch:

docker run -d \
  --name kibana \
  --network apisix-quickstart-net \
  -p 5601:5601 \
  -e ELASTICSEARCH_HOSTS="http://elasticsearch:9200" \
  docker.elastic.co/kibana/kibana:7.17.1

If successful, you should see the Kibana web dashboard on localhost:5601.

Enable elasticsearch-logger Plugin

Create a route to forward all requests to /ip to httpbin.org:

Admin API

curl -i "http://127.0.0.1:9180/apisix/admin/routes" -X PUT -d '
{
  "id": "quickstart-client-ip",
  "uri": "/ip",
  "upstream": {
    "nodes": {
      "httpbin.org:80":1
    },
    "type": "roundrobin"
  }
}'

An HTTP/1.1 200 OK response verifies that the route is created successfully.

ADC

adc-route.yaml

services:
  - name: httpbin Service
    routes:
      - uris:
          - /ip
        name: quickstart-client-ip
    upstream:
      type: roundrobin
      nodes:
        - host: httpbin.org
          port: 80
          weight: 1

Synchronize the configuration to APISIX:

adc sync -f adc-route.yaml

Enable the elasticsearch-logger plugin as a global rule for all routes, or on the route created above:

Global

You can configure APISIX through the Admin API or ADC:

Admin API

Enable the elasticsearch-logger plugin on all routes:

curl "http://127.0.0.1:9180/apisix/admin/global_rules/" -X PUT -d '
{
  "id": "elasticsearch",
  "plugins": {
    "elasticsearch-logger": {
      "endpoint_addr": "http://elasticsearch:9200",
      "field": {
        "index": "gateway",
        "type": "logs"
      },
      "ssl_verify": false,
      "timeout": 60,
      "retry_delay": 1,
      "buffer_duration": 60,
      "max_retry_count": 0,
      "batch_max_size": 5,
      "inactive_timeout": 5
    }
  }
}'

ADC

Enable the elasticsearch-logger plugin on all routes:

adc-global-rule.yaml

global_rules:
  elasticsearch-logger:
    endpoint_addr: "http://elasticsearch:9200"
    field:
      index: "gateway"
      type: "logs"
    ssl_verify: false
    timeout: 60
    retry_delay: 1
    buffer_duration: 60
    max_retry_count: 0
    batch_max_size: 5
    inactive_timeout: 5

Synchronize the configurations to APISIX:

adc sync -f adc-global-rule.yaml -f adc-route.yaml

Route

You can configure APISIX through the Admin API or ADC:

Admin API

Enable the elasticsearch-logger plugin on a specific route:

curl "http://127.0.0.1:9180/apisix/admin/routes/quickstart-client-ip" -X PATCH -d '
{
  "plugins": {
    "elasticsearch-logger": {
      "endpoint_addr": "http://elasticsearch:9200",
      "field": {
        "index": "gateway",
        "type": "logs"
      },
      "ssl_verify": false,
      "timeout": 60,
      "retry_delay": 1,
      "buffer_duration": 60,
      "max_retry_count": 0,
      "batch_max_size": 5,
      "inactive_timeout": 5
    }
  }
}'

ADC

Update the route configuration to enable the elasticsearch-logger plugin:

adc-route.yaml

services:
  - name: httpbin Service
    routes:
      - uris:
          - /ip
        name: quickstart-client-ip
        plugins:
          elasticsearch-logger:
            endpoint_addr: "http://elasticsearch:9200"
            field:
              index: "gateway"
              type: "logs"
            ssl_verify: false
            timeout: 60
            retry_delay: 1
            buffer_duration: 60
            max_retry_count: 0
            batch_max_size: 5
            inactive_timeout: 5
    upstream:
        type: roundrobin
        nodes:
          - host: httpbin.org
            port: 80
            weight: 1

Synchronize the configuration to APISIX:

adc sync -f adc-route.yaml

Customize Log Format

As an optional step, you can customize the log format for elasticsearch-logger. The log format of most APISIX logging plugins could be customized locally on the plugin (e.g. bound to a route) and/or globally with plugin metadata.

Add host address, timestamp, and client IP address to the logs with built-in variables:

Admin API

curl "http://127.0.0.1:9180/apisix/admin/plugin_metadata/elasticsearch-logger" -X PUT -d '
{
  "log_format":{
    "host":"$host",
    "timestamp":"$time_iso8601",
    "client_ip":"$remote_addr"
  }
}'

ADC

adc-plugin-metadata.yaml

plugin_metadata:
  elasticsearch-logger:
    log_format:
        host: $host
        client_ip: $remote_addr
        timestamp: $time_iso8601

Synchronize the configurations to APISIX:

adc sync -f adc-plugin-metadata -f adc-global-rule -f adc-route.yaml

Configure Kibana

Send some requests to the route to generate an access log entry:

for i in {1..10}; do
  curl -i "http://127.0.0.1:9080/ip"
done

Open Kibana dashboard on localhost:5601 and click the Discover tab from the menu. Create a new index pattern to fetch the data from Elasticsearch:

Create a pattern gateway to match the indexed data in Elasticsearch:

If your configuration is correct, you can go back to the discover page and view the logs from APISIX:

Next Steps

See elasticsearch-logger plugin reference to learn more about the plugin configuration options (coming soon).