splunk-hec-logging
The splunk-hec-logging plugin serializes request and response context information to Splunk Event Data format and push to your Splunk HTTP Event Collector (HEC) in batches. The plugin also supports the customization of log formats.
Examples
The examples below demonstrate how you can configure splunk-hec-logging plugin for different scenarios.
To follow along the examples, set up a Splunk HEC endpoint:
- Local Splunk
- Kubernetes
Complete the following steps to set up Splunk:
- Install Splunk. Splunk Web should be running at
localhost:8000by default. - See set up and use HTTP Event Collector in Splunk Web to create an HTTP Event Collector.
- Navigate to Settings > Data Inputs and note down the token value.
- In HTTP Event Collector > Global Settings, enable all tokens and note down the collector port, which defaults to
8088.
To verify the setup, execute the following command with your token:
curl "http://localhost:8088/services/collector/event" \
-H "Authorization: Splunk <replace-with-your-token>" \
-d '{"event": "hello world"}'
You should see a success response.
Create a Kubernetes manifest to deploy Splunk with HEC enabled:
apiVersion: v1
kind: ConfigMap
metadata:
namespace: aic
name: splunk-defaults
data:
default.yml: |
splunk:
hec:
enable: True
ssl: False
token: apisix-hec-token
---
apiVersion: apps/v1
kind: Deployment
metadata:
namespace: aic
name: splunk
spec:
replicas: 1
selector:
matchLabels:
app: splunk
template:
metadata:
labels:
app: splunk
spec:
enableServiceLinks: false
containers:
- name: splunk
image: splunk/splunk:9.4
env:
- name: SPLUNK_START_ARGS
value: "--accept-license"
# Accept Splunk General Terms: https://www.splunk.com/en_us/legal/splunk-general-terms.html
- name: SPLUNK_GENERAL_TERMS
value: "--accept-sgt-current-at-splunk-com"
- name: SPLUNK_PASSWORD
value: "Splunk@1234"
ports:
- containerPort: 8088
- containerPort: 8000
volumeMounts:
- name: defaults
mountPath: /tmp/defaults
readinessProbe:
httpGet:
path: /services/collector/health
port: 8088
initialDelaySeconds: 60
periodSeconds: 10
failureThreshold: 10
volumes:
- name: defaults
configMap:
name: splunk-defaults
---
apiVersion: v1
kind: Service
metadata:
namespace: aic
name: splunk-hec
spec:
selector:
app: splunk
ports:
- name: hec
port: 8088
targetPort: 8088
- name: web
port: 8000
targetPort: 8000
type: ClusterIP
Apply the manifest:
kubectl apply -f splunk-hec-server.yaml
Port forward the Splunk Web port to your local machine:
kubectl port-forward -n aic svc/splunk-hec 8000:8000
Then open http://localhost:8000 and log in with username admin and password Splunk@1234.
Push Log to Splunk
The following example demonstrates how you can enable the splunk-hec-logging plugin on a route, which logs client requests and pushes logs to Splunk.
Create a route as such:
- Admin API
- ADC
- Ingress Controller
curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
-H "X-API-KEY: ${ADMIN_API_KEY}" \
-d '{
"id": "splunk-route",
"uri": "/anything",
"plugins": {
"splunk-hec-logging": {
"endpoint": {
"uri": "http://127.0.0.1:8088/services/collector/event",
"token": "example-splunk-hec-token"
}
}
},
"upstream": {
"nodes": {
"httpbin.org:80": 1
},
"type": "roundrobin"
}
}'
services:
- name: httpbin
routes:
- name: splunk-route
uris:
- /anything
plugins:
splunk-hec-logging:
endpoint:
uri: http://127.0.0.1:8088/services/collector/event
token: example-splunk-hec-token
upstream:
type: roundrobin
nodes:
- host: httpbin.org
port: 80
weight: 1
Synchronize the configuration to the gateway:
adc sync -f adc.yaml
- Gateway API
- APISIX CRD
apiVersion: v1
kind: Service
metadata:
namespace: aic
name: httpbin-external-domain
spec:
type: ExternalName
externalName: httpbin.org
---
apiVersion: apisix.apache.org/v1alpha1
kind: PluginConfig
metadata:
namespace: aic
name: splunk-hec-logging-plugin-config
spec:
plugins:
- name: splunk-hec-logging
config:
endpoint:
uri: http://splunk-hec.aic.svc.cluster.local:8088/services/collector/event
token: apisix-hec-token
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
namespace: aic
name: splunk-route
spec:
parentRefs:
- name: apisix
rules:
- matches:
- path:
type: Exact
value: /anything
filters:
- type: ExtensionRef
extensionRef:
group: apisix.apache.org
kind: PluginConfig
name: splunk-hec-logging-plugin-config
backendRefs:
- name: httpbin-external-domain
port: 80
apiVersion: apisix.apache.org/v2
kind: ApisixUpstream
metadata:
namespace: aic
name: httpbin-external-domain
spec:
ingressClassName: apisix
externalNodes:
- type: Domain
name: httpbin.org
---
apiVersion: apisix.apache.org/v2
kind: ApisixRoute
metadata:
namespace: aic
name: splunk-route
spec:
ingressClassName: apisix
http:
- name: splunk-route
match:
paths:
- /anything
methods:
- GET
upstreams:
- name: httpbin-external-domain
plugins:
- name: splunk-hec-logging
enable: true
config:
endpoint:
uri: http://splunk-hec.aic.svc.cluster.local:8088/services/collector/event
token: apisix-hec-token
Apply the configuration:
kubectl apply -f splunk-hec-logging-ic.yaml
❶ Configure the Splunk HTTP collector endpoint. For Kubernetes, use the in-cluster Service address such as http://splunk-hec.aic.svc.cluster.local:8088/services/collector/event.
❷ Replace with your collector token.
Send a request to the route:
curl -i "http://127.0.0.1:9080/anything"
You should receive an HTTP/1.1 200 OK response.
Navigate to Splunk Web and select Search & Reporting in the left menu. In the search box, enter source="apache-apisix-splunk-hec-logging" and search for events from APISIX.
Log Request and Response Headers With Plugin Metadata
The following example demonstrates how you can customize log format using plugin metadata and built-in variables to log specific headers from request and response.
In APISIX, plugin metadata is used to configure the common metadata fields of all plugin instances of the same plugin. It is useful when a plugin is enabled across multiple resources and requires a universal update to their metadata fields.
Create a route as such:
- Admin API
- ADC
- Ingress Controller
curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
-H "X-API-KEY: ${ADMIN_API_KEY}" \
-d '{
"id": "splunk-route",
"uri": "/anything",
"plugins": {
"splunk-hec-logging": {
"endpoint": {
"uri": "http://127.0.0.1:8088/services/collector/event",
"token": "example-splunk-hec-token"
}
}
},
"upstream": {
"nodes": {
"httpbin.org:80": 1
},
"type": "roundrobin"
}
}'
services:
- name: httpbin
routes:
- name: splunk-route
uris:
- /anything
plugins:
splunk-hec-logging:
endpoint:
uri: http://127.0.0.1:8088/services/collector/event
token: example-splunk-hec-token
upstream:
type: roundrobin
nodes:
- host: httpbin.org
port: 80
weight: 1
Synchronize the configuration to the gateway:
adc sync -f adc.yaml
- Gateway API
- APISIX CRD
apiVersion: v1
kind: Service
metadata:
namespace: aic
name: httpbin-external-domain
spec:
type: ExternalName
externalName: httpbin.org
---
apiVersion: apisix.apache.org/v1alpha1
kind: PluginConfig
metadata:
namespace: aic
name: splunk-hec-logging-plugin-config
spec:
plugins:
- name: splunk-hec-logging
config:
endpoint:
uri: http://splunk-hec.aic.svc.cluster.local:8088/services/collector/event
token: apisix-hec-token
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
namespace: aic
name: splunk-route
spec:
parentRefs:
- name: apisix
rules:
- matches:
- path:
type: Exact
value: /anything
filters:
- type: ExtensionRef
extensionRef:
group: apisix.apache.org
kind: PluginConfig
name: splunk-hec-logging-plugin-config
backendRefs:
- name: httpbin-external-domain
port: 80
apiVersion: apisix.apache.org/v2
kind: ApisixUpstream
metadata:
namespace: aic
name: httpbin-external-domain
spec:
ingressClassName: apisix
externalNodes:
- type: Domain
name: httpbin.org
---
apiVersion: apisix.apache.org/v2
kind: ApisixRoute
metadata:
namespace: aic
name: splunk-route
spec:
ingressClassName: apisix
http:
- name: splunk-route
match:
paths:
- /anything
methods:
- GET
upstreams:
- name: httpbin-external-domain
plugins:
- name: splunk-hec-logging
enable: true
config:
endpoint:
uri: http://splunk-hec.aic.svc.cluster.local:8088/services/collector/event
token: apisix-hec-token
Apply the configuration:
kubectl apply -f splunk-hec-logging-ic.yaml
Configure the plugin metadata for splunk-hec-logging:
- Admin API
- ADC
- Ingress Controller
curl "http://127.0.0.1:9180/apisix/admin/plugin_metadata/splunk-hec-logging" -X PUT \
-H "X-API-KEY: ${ADMIN_API_KEY}" \
-d '{
"log_format": {
"host": "$host",
"@timestamp": "$time_iso8601",
"route_id": "$route_id",
"client_ip": "$remote_addr",
"env": "$http_env",
"resp_content_type": "$sent_http_Content_Type"
}
}'
plugin_metadata:
- name: splunk-hec-logging
log_format:
host: "$host"
"@timestamp": "$time_iso8601"
route_id: "$route_id"
client_ip: "$remote_addr"
env: "$http_env"
resp_content_type: "$sent_http_Content_Type"
Synchronize the configuration to the gateway:
adc sync -f adc.yaml
- Gateway API
- APISIX CRD
apiVersion: apisix.apache.org/v1alpha1
kind: GatewayProxy
metadata:
namespace: aic
name: apisix-config
spec:
provider:
type: ControlPlane
controlPlane:
# ...
# your control plane connection configuration
pluginMetadata:
splunk-hec-logging:
log_format:
host: "$host"
"@timestamp": "$time_iso8601"
route_id: "$route_id"
client_ip: "$remote_addr"
env: "$http_env"
resp_content_type: "$sent_http_Content_Type"
apiVersion: apisix.apache.org/v1alpha1
kind: GatewayProxy
metadata:
namespace: aic
name: apisix-config
spec:
provider:
type: ControlPlane
controlPlane:
# ...
# your control plane connection configuration
pluginMetadata:
splunk-hec-logging:
log_format:
host: "$host"
"@timestamp": "$time_iso8601"
route_id: "$route_id"
client_ip: "$remote_addr"
env: "$http_env"
resp_content_type: "$sent_http_Content_Type"
Apply the configuration:
kubectl apply -f gatewayproxy.yaml
❶ log the custom request header env.
❷ log the response header Content-Type.
Send a request to the route with the env header:
curl -i "http://127.0.0.1:9080/anything" -H "env: dev"
Navigate to Splunk Web and select Search & Reporting in the left menu. In the search box, enter source="apache-apisix-splunk-hec-logging" and search for events.