Skip to main content
API7.ai Logo

splunk-hec-logging

The splunk-hec-logging plugin sends APISIX and API7 Gateway request and response logs to Splunk HTTP Event Collector (HEC) in Splunk Event Data format. The plugin batches delivery and supports customizable log formats.

Examples​

The examples show how to send gateway request logs to Splunk and customize their contents with plugin metadata.

To follow the examples, set up a Splunk HEC endpoint:

The official Splunk image is currently published for amd64. Set the gateway container name and the local evaluation credentials:

export GATEWAY_CONTAINER=replace-with-gateway-container-name
export SPLUNK_PASSWORD=Splunk-local-password-1234
export SPLUNK_HEC_TOKEN=apisix-hec-token

Create a dedicated network:

docker network create gateway-splunk-net

Connect the gateway to the network:

docker network connect gateway-splunk-net "$GATEWAY_CONTAINER"

Start Splunk Enterprise with HEC enabled. The required flags indicate acceptance of the Splunk General Terms:

docker run -d \
  --platform linux/amd64 \
  --name splunk-hec \
  --network gateway-splunk-net \
  -p 127.0.0.1:8000:8000 \
  -p 127.0.0.1:8088:8088 \
  -e SPLUNK_START_ARGS=--accept-license \
  -e SPLUNK_GENERAL_TERMS=--accept-sgt-current-at-splunk-com \
  -e SPLUNK_PASSWORD \
  -e SPLUNK_HEC_TOKEN \
  splunk/splunk:10.4.3-rhel9

After the container starts, verify that HEC is healthy:

curl -k "https://127.0.0.1:8088/services/collector/health"

You should receive {"text":"HEC is healthy","code":17}. If the command reports a connection error, wait a few seconds and run it again.

Set the HEC endpoint used by the Admin API and ADC examples:

export SPLUNK_HEC_ENDPOINT=https://splunk-hec:8088/services/collector/event

Splunk Web is available at http://localhost:8000; sign in as admin with the value of SPLUNK_PASSWORD.

Evaluation credentials

The passwords and HEC token in these examples are for a trusted evaluation environment. Use the organization's secret-management system and TLS trust configuration for production deployments.

Send Logs to Splunk​

The following example sends request logs from one route to Splunk HEC.

Create a route with the splunk-hec-logging plugin:

curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d @- <<EOF
  {
    "id": "splunk-route",
    "uri": "/anything",
    "plugins": {
      "splunk-hec-logging": {
        "endpoint": {
          "uri": "$SPLUNK_HEC_ENDPOINT",
          "token": "apisix-hec-token"
        },
        "ssl_verify": false,
        "batch_max_size": 1
      }
    },
    "upstream": {
      "nodes": {
        "httpbin.org:80": 1
      },
      "type": "roundrobin"
    }
  }
EOF

The Admin API and ADC examples set batch_max_size to 1 so each event is delivered immediately for verification. Omit this field to retain the default batching behavior. For the Docker HTTPS endpoint, ssl_verify: false accepts Splunk's self-signed evaluation certificate. Configure a trusted certificate and keep the default verification enabled in production.

Send a request to the route:

curl -i "http://127.0.0.1:9080/anything"

You should receive an HTTP/1.1 200 OK response.

In Splunk Web, open Search & Reporting. Search for source="apache-apisix-splunk-hec-logging" to find events from the gateway.

Log Request and Response Headers With Plugin Metadata​

The following example uses plugin metadata to add selected request and response headers to splunk-hec-logging instances that do not define their own log_format. The metadata values reference built-in variables, so one configuration applies the same log fields across multiple routes and services.

Create a route with the splunk-hec-logging plugin:

curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d @- <<EOF
  {
    "id": "splunk-route",
    "uri": "/anything",
    "plugins": {
      "splunk-hec-logging": {
        "endpoint": {
          "uri": "$SPLUNK_HEC_ENDPOINT",
          "token": "apisix-hec-token"
        },
        "ssl_verify": false,
        "batch_max_size": 1
      }
    },
    "upstream": {
      "nodes": {
        "httpbin.org:80": 1
      },
      "type": "roundrobin"
    }
  }
EOF

Configure the plugin metadata for splunk-hec-logging:

curl "http://127.0.0.1:9180/apisix/admin/plugin_metadata/splunk-hec-logging" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d '{
    "log_format": {
      "host": "$host",
      "@timestamp": "$time_iso8601",
      "route_id": "$route_id",
      "client_ip": "$remote_addr",
      "env": "$http_env",
      "resp_content_type": "$sent_http_Content_Type"
    }
  }'

❶ Log the custom request header env.

❷ Log the response header Content-Type.

Send a request to the route with the env header:

curl -i "http://127.0.0.1:9080/anything" -H "env: dev"

In Splunk Web, open Search & Reporting and search for source="apache-apisix-splunk-hec-logging".