Collect Gateway Logs on Kubernetes
On Kubernetes, you can collect API7 Gateway access and error logs in two ways:
- From container output. The gateway writes to standard output and standard error, the container runtime persists that stream on the node, and a node-level collector reads it. This is the default and requires no gateway configuration.
- From log files inside the pod. The gateway writes real files, a volume exposes them, and a collector reads the files. This requires gateway configuration but keeps each log stream in its own file.
This guide covers both models with the OpenTelemetry Collector and with Filebeat.
For the log types the gateway produces and the configuration keys that control them, see Configure Centralized Logging.
Choose a Collection Model
| Container output | Log files in the pod | |
|---|---|---|
| Gateway configuration | None. This is the default. | Redirect the access log and error log, enable rotation, mount a volume. |
| Access and error logs | Interleaved in one stream, separated by an stdout or stderr marker. | Separate files. |
| Rotation and retention | Handled by the container runtime and kubelet. | Handled by the gateway's log-rotate plugin. |
kubectl logs | Shows gateway logs. | Shows almost nothing. |
| Long log lines | Split by the runtime at 16 KB and reassembled by the collector. | Written whole. |
| Concurrent writes | A single log line larger than the pipe buffer can interleave with another worker's line. | Each line is written to the file as one unit. |
Start with container output. It has the smallest operational surface, and both collectors handle it well.
When to Write Log Files Instead
Standard output from a container is a pipe. On Linux, a write to a pipe is atomic only up to PIPE_BUF, which is 4096 bytes. The gateway runs multiple worker processes that all write to the same pipe, so when a single log line exceeds that limit, another worker's write can be inserted in the middle of it. The result is a corrupted line that contains fragments of two different requests, and no collector can repair it because the damage happens before the runtime sees the data.
Consider writing log files when any of the following applies:
- Access log lines regularly exceed 4096 bytes, for example because the format includes large headers, request bodies, or many upstream fields.
- You have observed log entries that contain fragments of unrelated requests.
- You need the access log and the error log in separate files rather than one interleaved stream.
Writing files has costs. Rotation becomes your responsibility, kubectl logs stops being useful for the gateway container, and the collector needs a volume to reach the files. The rest of this guide covers both models so you can weigh them.
Collect Container Output
How Kubernetes Stores Container Output
The container runtime writes each container's output to the node filesystem under a path that encodes the pod identity:
/var/log/pods/<namespace>_<pod-name>_<pod-uid>/<container-name>/<restart-count>.log
The files under /var/log/containers/ are symbolic links to those files.
Each line in the file carries a runtime prefix with a timestamp, the stream name, and a partial or final flag:
2026-08-20T11:31:29.123456789Z stdout F 172.18.0.1 - - [20/Aug/2026:11:31:29 +0000] "GET /anything HTTP/1.1" 200
The runtime splits lines longer than 16 KB into several physical lines and marks all but the last with P instead of F. Both collectors described below strip the prefix and reassemble split lines, so you do not need to handle this yourself.
The log files are owned by root with mode 0640, so the collector must run as root to read them.
Collect Container Output with the OpenTelemetry Collector
Run the collector as a DaemonSet and mount /var/log/pods read-only. A single container operator handles the runtime format:
extensions:
file_storage/checkpoints:
directory: /var/lib/otelcol/storage
receivers:
filelog/gateway:
include:
- /var/log/pods/<namespace>_<gateway-release-name>-*/gateway/*.log
exclude:
- /var/log/pods/*/otel-collector/*.log
start_at: end
include_file_path: true
storage: file_storage/checkpoints
operators:
- type: container
processors:
k8sattributes:
auth_type: serviceAccount
extract:
metadata:
- k8s.node.name
- k8s.deployment.name
pod_association:
- sources:
- from: resource_attribute
name: k8s.pod.uid
batch:
timeout: 2s
exporters:
otlp:
endpoint: <your-backend>:4317
service:
extensions: [file_storage/checkpoints]
pipelines:
logs:
receivers: [filelog/gateway]
processors: [k8sattributes, batch]
exporters: [otlp]
The container operator detects the containerd, CRI-O, or Docker format automatically, removes the runtime prefix, reassembles lines that the runtime split, and derives resource attributes from the file path:
| Attribute | Source |
|---|---|
k8s.pod.name, k8s.pod.uid, k8s.namespace.name | File path |
k8s.container.name, k8s.container.restart_count | File path |
log.iostream | Runtime prefix, either stdout or stderr |
Because the pod UID comes from the path, the k8sattributes processor can associate the record with the live pod and add attributes that the path does not carry, such as the node name, workload name, and pod labels.
Mount the host paths in the DaemonSet:
volumeMounts:
- name: varlogpods
mountPath: /var/log/pods
readOnly: true
- name: checkpoints
mountPath: /var/lib/otelcol/storage
volumes:
- name: varlogpods
hostPath:
path: /var/log/pods
- name: checkpoints
hostPath:
path: /var/lib/otelcol
type: DirectoryOrCreate
Keep the checkpoint directory on a hostPath volume rather than an emptyDir. The file_storage extension records how far the collector has read in each file. A DaemonSet pod is recreated on every upgrade, and an emptyDir is deleted with it, so the collector would re-read every log file on the node after each restart.
The OpenTelemetry Collector has no equivalent of the autodiscover hints in Filebeat, which apply per-pod parsing rules from annotations. The include patterns are static. Express per-workload parsing either as one receiver per workload with a narrow glob, or as routing in the pipeline on an attribute such as k8s.container.name.
Collect Container Output with Filebeat
Filebeat reads the same files. Use a filestream input with the container parser:
filebeat.inputs:
- type: filestream
id: api7-gateway-container
paths:
- /var/log/containers/<gateway-release-name>-*_<namespace>_gateway-*.log
prospector.scanner.symlinks: true
parsers:
- container:
stream: all
format: auto
processors:
- add_kubernetes_metadata:
host: ${NODE_NAME}
matchers:
- logs_path:
logs_path: "/var/log/containers/"
prospector.scanner.symlinks: true is required. The files under /var/log/containers/ are symbolic links, and the filestream input skips symbolic links by default, so the input silently collects nothing without it. The older type: container input enabled this by default, which is why configurations migrated from it stop working.
Mount /var/log/containers and /var/log/pods read-only, because the links resolve into the second directory:
volumeMounts:
- name: varlogcontainers
mountPath: /var/log/containers
readOnly: true
- name: varlogpods
mountPath: /var/log/pods
readOnly: true
- name: data
mountPath: /usr/share/filebeat/data
volumes:
- name: varlogcontainers
hostPath:
path: /var/log/containers
- name: varlogpods
hostPath:
path: /var/log/pods
- name: data
hostPath:
path: /var/lib/filebeat-data
type: DirectoryOrCreate
Keep the registry directory (/usr/share/filebeat/data) on a hostPath volume for the same reason the OpenTelemetry Collector needs a persistent checkpoint directory.
Filebeat needs read access to pods and namespaces to resolve metadata. Grant get, watch, and list on pods, namespaces, and nodes, plus replicasets in the apps API group, and run the container as root so it can read the runtime log files.
Write Log Files Inside the Pod
Step 1: Redirect the Access Log and Error Log to Files
In the official container images, /usr/local/apisix/logs/access.log and /usr/local/apisix/logs/error.log are symbolic links to /dev/stdout and /dev/stderr. Pointing the configuration back at those paths therefore still writes to container output. Use a different directory and mount a volume over it:
logs:
enableAccessLog: true
accessLog: "/var/log/apisix/access.log"
accessLogFormatEscape: json
accessLogFormat: '{"time":"$time_iso8601","remote_addr":"$remote_addr","host":"$http_host","request":"$request","status":$status,"body_bytes_sent":$body_bytes_sent,"request_time":$request_time,"upstream_addr":"$upstream_addr","upstream_status":"$upstream_status","request_id":"$apisix_request_id"}'
errorLog: "/var/log/apisix/error.log"
errorLogLevel: "warn"
extraVolumes:
- name: api7-gateway-logs
emptyDir:
sizeLimit: 10Gi
extraVolumeMounts:
- name: api7-gateway-logs
mountPath: /var/log/apisix
Do not mount the volume over /usr/local/apisix/logs. That directory also holds the gateway's runtime files, including nginx.pid and the worker event sockets.
The gateway container runs as UID 636. An emptyDir is created with permissive mode, but setting fsGroup makes the ownership explicit:
apisix:
podSecurityContext:
fsGroup: 636
Quote fields that can be empty, such as $upstream_status, so the line stays valid JSON when there is no upstream. Numeric fields such as $status and $request_time can be left unquoted.
Layer 4 access logs are not covered by rotation. If you enable logs.stream.enableAccessLog and point it at a file, that file grows without bound. Leave it disabled unless you have separate arrangements for its size.
Step 2: Enable Log Rotation
Once the gateway writes files, the container runtime no longer rotates anything. Without rotation the volume fills until the pod is evicted. Enable the log-rotate plugin, which the control plane already includes in the plugin list it sends to the data plane:
pluginAttrs:
log-rotate:
enable: true
interval: 3600
max_kept: 24
max_size: 268435456
enable_compression: false
The plugin renames the current file and signals the gateway to reopen it. Rotated files are named with the timestamp as a prefix:
2026-08-20_19-00-00__access.log
2026-08-20_19-00-00__error.log
Exclude that pattern in the collector so rotated files are not treated as new sources. Both collectors continue reading a renamed file to its end before releasing it, so excluding the pattern does not lose the entries written just before rotation.
Leave enable_compression disabled. When compression is on, the plugin archives the rotated file and deletes the original shortly afterwards, which can race with a collector that has not finished reading it.
Step 3: Choose Where the Collector Runs
| Sidecar collector | Node-level DaemonSet | |
|---|---|---|
| Collector instances | One per gateway pod | One per node |
| Volume | emptyDir shared inside the pod | The same emptyDir, read from the node |
| Host mounts | None | /var/lib/kubelet, read-only |
| Resource cost | Higher | Lower |
| Shutdown | Drains after the gateway stops | Can lose the tail when the pod is deleted |
An emptyDir is not confined to the pod's own filesystem. The kubelet backs it with a directory on the node:
/var/lib/kubelet/pods/<pod-uid>/volumes/kubernetes.io~empty-dir/<volume-name>/
That path is what makes node-level collection of in-pod files possible, and it keeps the volume bound to the pod lifecycle, so the kubelet removes it when the pod is deleted.
A file written to the container's own filesystem, rather than to a volume, has no stable path on the node. It lives in the container's writable layer under a snapshot directory whose identifier changes whenever the container is recreated and cannot be resolved from the pod name. No collector can discover it, which is why exposing the files through a volume is required rather than optional. Filebeat has the same constraint.
Collect Log Files with a Sidecar OpenTelemetry Collector
The gateway chart does not expose an extraContainers value, but Kubernetes 1.29 and later support sidecars declared as init containers with restartPolicy: Always. Such a container starts before the gateway and stops after it, which covers both startup logs and the shutdown drain.
extraVolumes:
- name: api7-gateway-logs
emptyDir:
sizeLimit: 10Gi
- name: otelcol-config
configMap:
name: gateway-otelcol-config
- name: otelcol-storage
emptyDir: {}
extraVolumeMounts:
- name: api7-gateway-logs
mountPath: /var/log/apisix
extraInitContainers:
- name: otel-collector
image: otel/opentelemetry-collector-contrib:0.119.0
restartPolicy: Always
args: ["--config=/etc/otelcol/config.yaml"]
env:
- name: K8S_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: K8S_POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: K8S_NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
volumeMounts:
- name: api7-gateway-logs
mountPath: /var/log/apisix
readOnly: true
- name: otelcol-config
mountPath: /etc/otelcol
- name: otelcol-storage
mountPath: /var/lib/otelcol/storage
resources:
requests:
cpu: 50m
memory: 128Mi
limits:
memory: 512Mi
The collector reads the two files directly and takes pod identity from the downward API:
extensions:
file_storage/checkpoints:
directory: /var/lib/otelcol/storage
receivers:
filelog/access:
include: [/var/log/apisix/access.log]
exclude: [/var/log/apisix/*__*.log, /var/log/apisix/*.tar.gz]
start_at: beginning
include_file_path: true
storage: file_storage/checkpoints
operators:
- type: json_parser
timestamp:
parse_from: attributes.time
layout_type: gotime
layout: '2006-01-02T15:04:05Z07:00'
filelog/error:
include: [/var/log/apisix/error.log]
exclude: [/var/log/apisix/*__*.log, /var/log/apisix/*.tar.gz]
start_at: beginning
include_file_path: true
storage: file_storage/checkpoints
multiline:
line_start_pattern: '^\d{4}/\d{2}/\d{2} \d{2}:\d{2}:\d{2}'
operators:
- type: regex_parser
regex: '(?s)^(?P<ts>\d{4}/\d{2}/\d{2} \d{2}:\d{2}:\d{2}) \[(?P<sev>\w+)\] (?P<rest>.*)$'
timestamp:
parse_from: attributes.ts
layout_type: strptime
layout: '%Y/%m/%d %H:%M:%S'
location: UTC
severity:
parse_from: attributes.sev
mapping:
debug: debug
info: info
warn: [warn, notice]
error: [error, crit]
fatal: [alert, emerg]
processors:
resource:
attributes:
- key: k8s.namespace.name
value: ${env:K8S_NAMESPACE}
action: upsert
- key: k8s.pod.name
value: ${env:K8S_POD_NAME}
action: upsert
- key: k8s.node.name
value: ${env:K8S_NODE_NAME}
action: upsert
batch:
timeout: 2s
exporters:
otlp:
endpoint: <your-backend>:4317
service:
extensions: [file_storage/checkpoints]
pipelines:
logs:
receivers: [filelog/access, filelog/error]
processors: [resource, batch]
exporters: [otlp]
Collect Log Files with a Node-Level OpenTelemetry Collector
To read the same files from a DaemonSet, keep the gateway values from Step 1 without the sidecar, mount /var/lib/kubelet read-only in the collector, and change the include patterns to the kubelet path. Give the volume a distinctive name so the pattern does not match another workload's emptyDir.
receivers:
filelog/gateway-access:
include:
- /var/lib/kubelet/pods/*/volumes/kubernetes.io~empty-dir/api7-gateway-logs/access.log
exclude:
- /var/lib/kubelet/pods/*/volumes/kubernetes.io~empty-dir/api7-gateway-logs/*__*.log
start_at: beginning
include_file_path: true
storage: file_storage/checkpoints
operators:
- type: regex_parser
parse_from: 'attributes["log.file.path"]'
parse_to: attributes.k8s
regex: '^/var/lib/kubelet/pods/(?P<uid>[0-9a-f]{8}-[0-9a-f-]{27})/volumes/'
- type: move
from: attributes.k8s.uid
to: 'resource["k8s.pod.uid"]'
- type: remove
field: attributes.k8s
- type: json_parser
parse_from: body
timestamp:
parse_from: attributes.time
layout_type: gotime
layout: '2006-01-02T15:04:05Z07:00'
Pair it with the k8sattributes processor and pod_association on k8s.pod.uid, exactly as in the container output example, to resolve the pod name, namespace, node, and labels.
Quote field references such as 'attributes["log.file.path"]' and 'resource["k8s.pod.uid"]'. Unquoted square brackets inside a YAML flow mapping are a syntax error, and the collector reports it as retrieved value (type=string) cannot be used as a Conf, which does not point at the real problem.
volumeMounts:
- name: varlibkubelet
mountPath: /var/lib/kubelet
readOnly: true
mountPropagation: HostToContainer
volumes:
- name: varlibkubelet
hostPath:
path: /var/lib/kubelet
/var/lib/kubelet is the kubelet's working directory and can be relocated with its --root-dir flag. Mounting it grants broad visibility into every pod's volumes on the node, and the restricted Pod Security Standard forbids hostPath volumes altogether. Use the sidecar placement where those constraints apply.
Collect Log Files with Filebeat
Filebeat reads the kubelet path with the same volume mount. Resolve pod metadata with the pod_uid indexer and a logs_path matcher scoped to /var/lib/kubelet/pods/:
filebeat.inputs:
- type: filestream
id: api7-gateway-access
paths:
- /var/lib/kubelet/pods/*/volumes/kubernetes.io~empty-dir/api7-gateway-logs/access.log
prospector.scanner.exclude_files: ['__access\.log$', '\.tar\.gz$']
parsers:
- ndjson:
target: ""
add_error_key: true
processors:
- add_kubernetes_metadata:
host: ${NODE_NAME}
indexers:
- pod_uid: ~
matchers:
- logs_path:
logs_path: "/var/lib/kubelet/pods/"
resource_type: "pod"
- type: filestream
id: api7-gateway-error
paths:
- /var/lib/kubelet/pods/*/volumes/kubernetes.io~empty-dir/api7-gateway-logs/error.log
prospector.scanner.exclude_files: ['__error\.log$', '\.tar\.gz$']
parsers:
- multiline:
type: pattern
pattern: '^\d{4}/\d{2}/\d{2} \d{2}:\d{2}:\d{2}'
negate: true
match: after
processors:
- add_kubernetes_metadata:
host: ${NODE_NAME}
indexers:
- pod_uid: ~
matchers:
- logs_path:
logs_path: "/var/lib/kubelet/pods/"
resource_type: "pod"
The ndjson parser with an empty target promotes the JSON access log fields to the top level of the event. The multiline parser groups the continuation lines of a Lua stack trace into the entry that starts it.
To run Filebeat as a sidecar instead, keep the same inputs and replace the paths with /var/log/apisix/access.log and /var/log/apisix/error.log, then drop add_kubernetes_metadata in favor of add_fields populated from the downward API.
add_kubernetes_metadata initializes its watcher lazily. Events that Filebeat emits before the watcher has synchronized carry no kubernetes.* fields. This is most visible on a first start with a large backlog, because Filebeat reads the backlog faster than the watcher becomes ready. Steady-state events are unaffected.
Verify Log Collection
Send a request through the gateway with a distinctive path so it is easy to search for:
curl "http://<gateway-address>:9080/collection-probe"
Confirm the gateway wrote it. For container output:
kubectl logs -n <namespace> <gateway-pod> -c gateway | grep collection-probe
For file output:
kubectl exec -n <namespace> <gateway-pod> -c gateway -- grep collection-probe /var/log/apisix/access.log
Then search your logging backend for the same string. If the entry reached the gateway but not the backend, the problem is in the collector; if it never reached the gateway, the problem is in the gateway's log configuration.
Troubleshooting
The collector reports no files. Check the glob against the real path on the node. Pod directory names use underscores between namespace, pod name, and UID, and a bare pod has no generated name suffix. For Filebeat reading /var/log/containers/, confirm prospector.scanner.symlinks: true is set.
The collector starts but emits nothing. Confirm it runs as root. The runtime log files under /var/log/pods are mode 0640 and owned by root.
Timestamps are wrong or fall back to the ingestion time. The access log uses $time_iso8601, which renders the UTC offset with a colon, such as +08:00. The strptime directive %z does not accept that form, so use layout_type: gotime with 2006-01-02T15:04:05Z07:00. The error log has no offset at all, so its timestamp is interpreted in the location you configure. If you set a time zone on the gateway pod, set the same value in location, otherwise the parsed time is shifted.
Error log entries lose their severity or timestamp. A Lua stack trace spans several lines. After the lines are combined, the record body contains newlines, and in Go regular expressions . does not match a newline. Add the (?s) flag to the pattern.
Records arrive without Kubernetes metadata. For the OpenTelemetry Collector, confirm that pod_association matches an attribute the pipeline actually sets. For Filebeat, check whether the affected events were emitted during startup, before the metadata watcher synchronized.
The volume fills up. Confirm the log-rotate plugin is enabled and that max_kept and max_size bound the total to less than the volume's sizeLimit. Remember that layer 4 access logs are not rotated.
Next Steps
- Configure Centralized Logging — log types, formats, and the configuration keys that control them.
- Send Kubernetes Error Logs to Splunk — a Splunk-specific walkthrough built on container output collection.
- Include Consumer Labels in Access Logs — add per-consumer metadata to every access log line.
- Send Access Logs to Splunk — stream access logs from a route without writing them to disk.