Send Kubernetes Error Logs to Splunk

When API7 Gateway runs on Kubernetes, its error logs are emitted by the gateway containers and collected at the platform level. This is different from route-level access-log streaming with logger plugins such as splunk-hec-logging.

Use this guide when you want to collect gateway error logs from Kubernetes pods and forward them to Splunk for centralized troubleshooting.

Prerequisites

Before you begin, ensure you have:

• API7 Gateway data plane pods are running on Kubernetes.
• You have access to a Splunk deployment, either Splunk Cloud or self-managed Splunk.
• You can install a Helm chart into the Kubernetes cluster.
• You have permission to create or use an HTTP Event Collector (HEC) token in Splunk.

Step 1: Prepare Splunk

If you already have Splunk and an HEC token, continue to Step 2.

For a local test environment, you can run Splunk on the host with Docker Compose:

docker-compose.yaml

services:
  splunk:
    image: splunk/splunk:8.2.3
    container_name: splunk
    hostname: splunk
    environment:
      - SPLUNK_START_ARGS=--accept-license
      - SPLUNK_PASSWORD=yourpassword123
      - SPLUNK_HEC_TOKEN=your-hec-token
    ports:
      - "8000:8000"
      - "8088:8088"
      - "8089:8089"
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8000"]
      interval: 30s
      timeout: 10s
      retries: 3
    restart: unless-stopped

Start Splunk:

docker compose up -d

note

The collector runs inside Kubernetes, so the HEC endpoint must be reachable from collector pods. For a host-side Docker Compose deployment, use a host name or IP that your cluster can resolve and reach. In a local kind-based setup, host.docker.internal works.

Step 2: Create a Splunk Index

Create a dedicated index for gateway error logs so searches and retention can be managed separately from access logs.

For example, create an index named gateway_error_logs.

Step 3: Create a Splunk HEC Token

In Splunk:

1. Go to Settings > Data Inputs > HTTP Event Collector.
2. Create a new token.
3. Record the HEC token value.

The HEC endpoint URL is typically:

https://<SPLUNK_HOST>:8088/services/collector/event

If you use the Docker Compose example above with a local kind cluster, set SPLUNK_HOST to host.docker.internal.

Step 4: Install the Splunk OpenTelemetry Collector

Install the Splunk OpenTelemetry Collector for Kubernetes and configure it to collect container logs.

This example enables log collection only, disables metrics and traces, and forwards entries to the gateway_error_logs index.

Create a values file such as the following:

values.yaml

clusterName: "my-cluster"

splunkPlatform:
  endpoint: "https://<SPLUNK_HOST>:8088/services/collector/event"
  token: "your-hec-token"
  index: "gateway_error_logs"
  insecureSkipVerify: true

  logsEnabled: true
  metricsEnabled: false
  tracesEnabled: false

logsCollection:
  containers:
    enabled: true
    excludeAgentLogs: true
    extraOperators:
      - type: "filter"
        expr: 'resource["k8s.container.name"] != "gateway"'
      - type: "filter"
        expr: 'not (body matches "(?i)(error|exception|fail|fatal)")'

agent:
  resources:
    limits:
      cpu: 1
      memory: 512Mi
    requests:
      cpu: 200m
      memory: 256Mi

❶ Keep only API7 Gateway container logs so the collector does not forward error-like entries from other Kubernetes workloads on the same node.

❷ Drop gateway logs that do not match common error keywords such as error, exception, fail, or fatal.

Adjust the filter expressions if your gateway container uses a different name or if your environment needs a narrower or broader definition of error logs.

Install the collector:

helm repo add splunk-otel-collector https://signalfx.github.io/splunk-otel-collector-chart
helm repo update
helm upgrade --install my-splunk-otel-collector \
  -n logging \
  --create-namespace \
  -f values.yaml \
  splunk-otel-collector-chart/splunk-otel-collector

For advanced configuration options, see the Splunk OpenTelemetry Collector Helm chart values reference.

Step 5: Verify Log Collection

Generate or wait for an error from an API7 Gateway pod, then search Splunk for matching events.

For example:

index="gateway_error_logs" sourcetype="kube:container:gateway"

You should see API7 Gateway container log entries in the configured index that match the filter keywords.

Troubleshooting

• Verify that the collector pods are running and healthy in the logging namespace.
• Confirm the HEC endpoint URL and token are correct.
• If Splunk runs outside the cluster, verify that the HEC host is reachable from collector pods, not just from your local shell.
• Check whether the filter expression is excluding too much data.
• Make sure the API7 Gateway pods are writing logs to container output that the collector can read.
• If Splunk uses a trusted certificate, replace insecureSkipVerify: true with the appropriate TLS configuration for production.

Next Steps

• Configure Centralized Logging — compare file shipping, collector-based shipping, and plugin-based logging.
• Send Access Logs to Splunk — send access logs directly from the gateway.
• Configure Distributed Tracing — correlate runtime failures with traces when request-level visibility is needed.