Skip to main content

elasticsearch-logger

The elasticsearch-logger plugin sends request and response logs to Elasticsearch in batches. The plugin serializes each log entry in the Elasticsearch Bulk API format and supports customized log fields and index names. This integration centralizes gateway logs for searching, analysis, and visualization in Kibana.

Examples

The examples below configure the elasticsearch-logger plugin for common logging scenarios.

To follow the examples, start Elasticsearch and Kibana. Both products use the same Elastic Stack version. In Docker, the gateway and Elasticsearch share a dedicated network so that the gateway can reach Elasticsearch by container name.

Local evaluation setup

Elastic Stack enables authentication and TLS by default. The following setup keeps authentication enabled but disables TLS on the Elasticsearch HTTP and transport interfaces to simplify local evaluation. The Docker ports are bound to the loopback interface, and the Kubernetes services are available only inside the cluster.

For a production deployment, use HTTPS with certificates issued by a trusted certificate authority. If the certificate uses a private CA, add the CA bundle to apisix.ssl.ssl_trusted_certificate, and keep the plugin's ssl_verify option enabled. Store credentials in a secret manager instead of configuration files.

On Linux, Elasticsearch requires vm.max_map_count to be at least 1048576 on the host or virtual machine that runs the containers. Check the current value before starting the Docker or Kubernetes setup:

sysctl vm.max_map_count

If the value is lower, increase it on the Docker host or on every Kubernetes node. This command requires administrative privileges:

sudo sysctl -w vm.max_map_count=1048576

For Docker Desktop and managed Kubernetes environments, follow Elastic's platform-specific virtual memory instructions to apply the setting to the underlying Linux environment.

Set GATEWAY_CONTAINER to the name of the running APISIX or API7 Gateway container. Create a dedicated Docker network and connect the gateway to it:

export GATEWAY_CONTAINER=replace-with-gateway-container-name

docker network create gateway-elasticsearch-net
docker network connect gateway-elasticsearch-net "$GATEWAY_CONTAINER"

Start Elasticsearch with authentication enabled:

docker run -d \
--name elasticsearch \
--network gateway-elasticsearch-net \
-v elasticsearch_logger_vol:/usr/share/elasticsearch/data/ \
-p 127.0.0.1:9200:9200 \
-e ELASTIC_PASSWORD=gateway-elastic-password \
-e ES_JAVA_OPTS="-Xms768m -Xmx768m" \
-e discovery.type=single-node \
-e xpack.security.enabled=true \
-e xpack.security.autoconfiguration.enabled=false \
-e xpack.security.http.ssl.enabled=false \
-e xpack.security.transport.ssl.enabled=false \
docker.elastic.co/elasticsearch/elasticsearch:9.5.3

Wait for Elasticsearch to become available:

until curl -fsS -u "elastic:gateway-elastic-password" \
"http://127.0.0.1:9200/_cluster/health?wait_for_status=yellow" > /dev/null; do
sleep 2
done

Set the password for the internal Kibana user:

curl "http://127.0.0.1:9200/_security/user/kibana_system/_password" \
-u "elastic:gateway-elastic-password" \
-H "Content-Type: application/json" \
-X POST \
-d '{"password":"gateway-kibana-password"}'

Create a role that can monitor the cluster and write only to the indices used in these examples:

curl "http://127.0.0.1:9200/_security/role/gateway_logger" \
-u "elastic:gateway-elastic-password" \
-H "Content-Type: application/json" \
-X PUT \
-d '{
"cluster": ["monitor"],
"indices": [
{
"names": ["gateway", "gateway-*"],
"privileges": ["auto_configure", "create_index", "write"]
}
]
}'

Create a user and assign the logger role:

curl "http://127.0.0.1:9200/_security/user/gateway_logger" \
-u "elastic:gateway-elastic-password" \
-H "Content-Type: application/json" \
-X PUT \
-d '{
"password": "gateway-logger-password",
"roles": ["gateway_logger"]
}'

Start Kibana to visualize the indexed data:

docker run -d \
--name kibana \
--network gateway-elasticsearch-net \
-p 127.0.0.1:5601:5601 \
-e ELASTICSEARCH_HOSTS="http://elasticsearch:9200" \
-e ELASTICSEARCH_USERNAME=kibana_system \
-e ELASTICSEARCH_PASSWORD=gateway-kibana-password \
docker.elastic.co/kibana/kibana:9.5.3

When Kibana becomes available, open localhost:5601 and log in as elastic with password gateway-elastic-password.

Log Requests in the Default Format

The following example enables the plugin on a route to send request and response information to the gateway index.

Create the route:

curl "http://127.0.0.1:9180/apisix/admin/routes/elasticsearch-logger-route" -X PUT \
-H "X-API-KEY: ${ADMIN_API_KEY}" \
-d '{
"uri": "/anything",
"plugins": {
"elasticsearch-logger": {
"endpoint_addrs": ["http://elasticsearch:9200"],
"field": {
"index": "gateway"
},
"auth": {
"username": "gateway_logger",
"password": "gateway-logger-password"
}
}
},
"upstream": {
"nodes": {
"httpbin.org:80": 1
},
"type": "roundrobin"
}
}'

❶ Configure the Elasticsearch endpoint. The endpoint does not end with a trailing slash.

❷ Configure the index field as gateway.

The credentials in these examples belong to the local evaluation user created earlier. Replace them with credentials managed according to your organization's security policies.

Send a request to the route to generate a log entry:

curl -i "http://127.0.0.1:9080/anything"

You should receive an HTTP/1.1 200 OK response. The batch processor can take several seconds to send the log entry.

In Kibana, open Discover and create a data view with the index pattern gateway. The new log entry should contain fields similar to the following:

{
"_index": "gateway",
"_id": "CE-JL5QBOkdYRG7kEjTJ",
"_version": 1,
"_score": 1,
"_source": {
"request": {
"headers": {
"host": "127.0.0.1:9080",
"accept": "*/*",
"user-agent": "curl/8.6.0"
},
"size": 85,
"querystring": {},
"method": "GET",
"url": "http://127.0.0.1:9080/anything",
"uri": "/anything"
},
"response": {
"headers": {
"content-type": "application/json",
"access-control-allow-credentials": "true",
"content-length": "390",
"access-control-allow-origin": "*",
"connection": "close",
"date": "Mon, 13 Jan 2025 10:18:14 GMT"
},
"status": 200,
"size": 618
},
"route_id": "elasticsearch-logger-route",
"latency": 585.00003814697,
"apisix_latency": 18.000038146973,
"upstream_latency": 567,
"upstream": "50.19.58.113:80",
"service_id": "",
"client_ip": "192.168.65.1"
},
"fields": {
...
}
}

Log Request and Response Headers with Plugin Metadata

The following example uses plugin metadata and built-in variables to record selected request and response headers.

Plugin metadata configures common metadata fields for all instances of the same plugin. It is useful when a plugin is enabled across multiple resources and requires a universal update to its metadata fields.

First, create a route with the plugin enabled:

curl "http://127.0.0.1:9180/apisix/admin/routes/elasticsearch-logger-route" -X PUT \
-H "X-API-KEY: ${ADMIN_API_KEY}" \
-d '{
"uri": "/anything",
"plugins": {
"elasticsearch-logger": {
"endpoint_addrs": ["http://elasticsearch:9200"],
"field": {
"index": "gateway"
},
"auth": {
"username": "gateway_logger",
"password": "gateway-logger-password"
}
}
},
"upstream": {
"nodes": {
"httpbin.org:80": 1
},
"type": "roundrobin"
}
}'

Next, configure the plugin metadata for elasticsearch-logger:

curl "http://127.0.0.1:9180/apisix/admin/plugin_metadata/elasticsearch-logger" -X PUT \
-H "X-API-KEY: ${ADMIN_API_KEY}" \
-d '{
"log_format": {
"host": "$host",
"@timestamp": "$time_iso8601",
"client_ip": "$remote_addr",
"env": "$http_env",
"resp_content_type": "$sent_http_Content_Type"
}
}'

❶ Log the custom request header env.

❷ Log the response header Content-Type.

Send a request to the route with the env header:

curl -i "http://127.0.0.1:9080/anything" -H "env: dev"

You should receive an HTTP/1.1 200 OK response.

In Kibana Discover, the log entry should contain the custom fields:

{
"_index": "gateway",
"_id": "Ck-WL5QBOkdYRG7kODS0",
"_version": 1,
"_score": 1,
"_source": {
"client_ip": "192.168.65.1",
"route_id": "elasticsearch-logger-route",
"@timestamp": "2025-01-06T10:32:36+00:00",
"host": "127.0.0.1",
"env": "dev",
"resp_content_type": "application/json"
},
"fields": {
...
}
}

Log Request Bodies Conditionally

The following example records a request body only when the request satisfies an APISIX expression.

If you completed the preceding example with the Admin API, remove the custom plugin metadata before continuing:

curl "http://127.0.0.1:9180/apisix/admin/plugin_metadata/elasticsearch-logger" -X DELETE \
-H "X-API-KEY: ${ADMIN_API_KEY}"

When using ADC, synchronize an empty plugin_metadata mapping with --include-resource-type plugin_metadata. When using the Ingress Controller, remove the plugin metadata from the GatewayProxy resource and reapply it.

Create the route:

curl "http://127.0.0.1:9180/apisix/admin/routes/elasticsearch-logger-route" -X PUT \
-H "X-API-KEY: ${ADMIN_API_KEY}" \
-d '{
"plugins": {
"elasticsearch-logger": {
"endpoint_addrs": ["http://elasticsearch:9200"],
"field": {
"index": "gateway"
},
"auth": {
"username": "gateway_logger",
"password": "gateway-logger-password"
},
"include_req_body": true,
"include_req_body_expr": [["arg_log_body", "==", "yes"]]
}
},
"upstream": {
"nodes": {
"httpbin.org:80": 1
},
"type": "roundrobin"
},
"uri": "/anything"
}'

❶ Set include_req_body to true to include the request body.

❷ Set include_req_body_expr to include the body only when the log_body query parameter is yes.

Send a request with a query parameter that satisfies the condition:

curl -i "http://127.0.0.1:9080/anything?log_body=yes" -X POST -d '{"env": "dev"}'

You should receive an HTTP/1.1 200 OK response.

In Kibana Discover, the log entry should include the request body:

{
"_index": "gateway",
"_id": "Dk-cL5QBOkdYRG7k7DSW",
"_version": 1,
"_score": 1,
"_source": {
"request": {
"headers": {
"user-agent": "curl/8.6.0",
"accept": "*/*",
"content-length": "14",
"host": "127.0.0.1:9080",
"content-type": "application/x-www-form-urlencoded"
},
"size": 182,
"querystring": {
"log_body": "yes"
},
"body": "{\"env\": \"dev\"}",
"method": "POST",
"url": "http://127.0.0.1:9080/anything?log_body=yes",
"uri": "/anything?log_body=yes"
},
"start_time": 1735965595203,
"response": {
"headers": {
"content-type": "application/json",
"access-control-allow-credentials": "true",
"content-length": "548",
"access-control-allow-origin": "*",
"connection": "close",
"date": "Mon, 13 Jan 2025 11:02:32 GMT"
},
"status": 200,
"size": 776
},
"route_id": "elasticsearch-logger-route",
"latency": 703.9999961853,
"apisix_latency": 34.999996185303,
"upstream_latency": 669,
"upstream": "34.197.122.172:80",
"service_id": "",
"client_ip": "192.168.65.1"
},
"fields": {
...
}
}

Send a request to the route without any URL query string:

curl -i "http://127.0.0.1:9080/anything" -X POST -d '{"env": "dev"}'

In Kibana Discover, the new log entry should not include the request body:

{
"_index": "gateway",
"_id": "EU-eL5QBOkdYRG7kUDST",
"_version": 1,
"_score": 1,
"_source": {
"request": {
"headers": {
"content-type": "application/x-www-form-urlencoded",
"accept": "*/*",
"content-length": "14",
"host": "127.0.0.1:9080",
"user-agent": "curl/8.6.0"
},
"size": 169,
"querystring": {},
"method": "POST",
"url": "http://127.0.0.1:9080/anything",
"uri": "/anything"
},
"start_time": 1735965686363,
"response": {
"headers": {
"content-type": "application/json",
"access-control-allow-credentials": "true",
"content-length": "510",
"access-control-allow-origin": "*",
"connection": "close",
"date": "Mon, 13 Jan 2025 11:15:54 GMT"
},
"status": 200,
"size": 738
},
"route_id": "elasticsearch-logger-route",
"latency": 680.99999427795,
"apisix_latency": 4.9999942779541,
"upstream_latency": 676,
"upstream": "34.197.122.172:80",
"service_id": "",
"client_ip": "192.168.65.1"
},
"fields": {
...
}
}
info

Custom log formats do not add collected request or response bodies automatically. Include the corresponding variables in the format:

{
"include_req_body": true,
"include_resp_body": true,
"log_format": {
"request_body": "$request_body",
"response_body": "$resp_body"
}
}

Body size limits still apply. Use log_format_extra to add custom fields without replacing the default log entry.

Include Request Date in Elasticsearch Index

The following example uses a Lua time format in the index name to organize logs by request date.

Create the route:

curl "http://127.0.0.1:9180/apisix/admin/routes/elasticsearch-logger-route" -X PUT \
-H "X-API-KEY: ${ADMIN_API_KEY}" \
-d '{
"uri": "/anything",
"plugins": {
"elasticsearch-logger": {
"endpoint_addrs": ["http://elasticsearch:9200"],
"field": {
"index": "gateway-{%Y.%m.%d}"
},
"auth": {
"username": "gateway_logger",
"password": "gateway-logger-password"
}
}
},
"upstream": {
"nodes": {
"httpbin.org:80": 1
},
"type": "roundrobin"
}
}'

❶ Configure the endpoint address to Elasticsearch.

❷ Configure the index field to use the current year, month, and date.

Send a request to the route to generate a log entry:

curl -i "http://127.0.0.1:9080/anything"

You should receive an HTTP/1.1 200 OK response.

In Kibana, create a data view with the index pattern gateway-*. The log entry should use an index name containing the request date:

{
"_index": "gateway-2026.09.14",
"_id": "CE-KL5QB0kdYRG7dEiTJ",
"_version": 1,
"_score": 1,
"_source": {
"request": {
...
},
"response": {
"status": 200,
"size": 618,
...
}
},
...
}