Parameters
See plugin common configurations for configuration options available to all plugins.
broker_list
Deprecated. Use
brokersinstead. A map of Kafka broker hosts to their ports. Configure eitherbroker_listorbrokers.brokers
vaild vaule:
greater than 0
List of Kafka broker nodes. Configure either
brokersor the deprecatedbroker_list.host
The host of Kafka broker.
port
The port of Kafka broker.
sasl_config
The SASL configuration of Kafka broker
mechanism
vaild vaule:
PLAIN,SCRAM-SHA-256, orSCRAM-SHA-512The mechanism of SASL configuration.
The
SCRAM-SHA-256andSCRAM-SHA-512options are available in API7 Enterprise from version 3.8.16 and APISIX from version 3.15.0.user
The user of SASL configuration.
password
The password of SASL configuration. The value is encrypted with AES before being stored in etcd.
tls
TLS configuration for connecting to Kafka brokers. Setting this object makes the plugin connect over TLS; omit it to connect in plaintext. Available in API7 Enterprise from version 3.9.17 on the 3.9 line and from version 3.10.4 on the 3.10 line.
verify
If true, verify the Kafka broker's TLS certificate against the configured trusted CA store.
The default is
false, so enablingtlsencrypts the connection but does not authenticate the broker, which leaves it open to an active man-in-the-middle. Set it totruein production.
kafka_topic
Target topic to push the logs for organization.
producer_type
vaild vaule:
asyncorsyncKafka producer mode. In
asyncmode, messages are buffered locally before being sent to Kafka. Insyncmode, messages are sent without using the async producer buffer.required_acks
vaild vaule:
-1 or 1
Number of acknowledgements the leader needs to receive for the producer to consider the request complete. This controls the durability of the sent records. See Kafka documentation for more information.
acks=0is not yet supported.api_version
vaild vaule:
0,1, or2Kafka Produce API version used to send messages to the broker. Only version
2carries the message timestamp, allowing the broker to store it; with the default1, messages may be recorded without a usable timestamp. Available in API7 Enterprise from version 3.9.14.key
Key used for allocating partitions for messages.
timeout
vaild vaule:
greater than 0
Timeout for the upstream to send data.
meta_format
vaild vaule:
defaultororiginFormat to collect the request information. Setting to
defaultcollects the information in JSON format andorigincollects the information with the original HTTP request. See the example for more details.log_format
Custom log format using key-value pairs in JSON format. Values can reference built-in variables.
In APISIX from 3.15.0, log format nested structures are supported up to five levels deep. In API7 Enterprise, only flat key-value structures are supported; nested structures are not yet supported.
You can also configure log format on a global scale using the plugin metadata, which configures the log format for all
kafka-loggerplugin instances. If the log format configured on the individual plugin instance differs from the log format configured on plugin metadata, the log format configured on the individual plugin instance takes precedence. See the example for more details.log_format_extra
Additional log fields to overlay on the default log entry, using key-value pairs in JSON format. Values can reference built-in variables. Unlike
log_format, which replaces the entire default log entry,log_format_extrakeeps the default entry and adds the configured fields on top of it. Whenlog_formatis set,log_format_extrais ignored. Available in API7 Enterprise from version 3.9.15.include_req_body
If true, include the request body in the log. Note that if the request body is too big to be kept in the memory, it can not be logged due to NGINX's limitations.
include_req_body_expr
An array of one or more conditions in the form of APISIX expressions. Used when the
include_req_bodyis true. Request body would only be logged when the expressions configured here evaluate to true.max_req_body_bytes
vaild vaule:
greater than or equal to 1
Maximum request body allowed in bytes. Request bodies falling within this limit will be pushed to Kafka. If the size exceeds the configured value, the body will be truncated before being pushed to Kafka.
include_resp_body
If true, include the response body in the log.
include_resp_body_expr
An array of one or more conditions in the form of APISIX expressions. Used when the
include_resp_bodyis true. Response body would only be logged when the expressions configured here evaluate to true.max_resp_body_bytes
vaild vaule:
greater than or equal to 1
Maximum response body allowed in bytes. Response bodies falling within this limit will be pushed to Kafka. If the size exceeds the configured value, the body will be truncated before being pushed to Kafka.
cluster_name
vaild vaule:
greater than or equal to 1
Name of the cluster. Used when there are two or more Kafka clusters. Only works if
producer_typeis set toasync.producer_batch_num
vaild vaule:
greater than or equal to 1
The number of messages to send in one batch. Same as the
batch_numparameter in lua-resty-kafka.producer_batch_size
vaild vaule:
greater than or equal to 0
The size of the TCP send buffer to use when sending data. Same as the
batch_sizeparameter in lua-resty-kafka, but in bytes.producer_max_buffering
vaild vaule:
greater than or equal to 1
Maximum number of Kafka producer messages that the async producer can buffer locally. Same as the
max_bufferingparameter in lua-resty-kafka. This buffer is not capped bymax_pending_entries. Memory usage also depends on worker count,batch_max_size, log format, and whether request or response bodies are logged.producer_time_linger
vaild vaule:
greater than or equal to 1
Flush time. Same as the
flush_timeparameter in lua-resty-kafka, but in seconds.meta_refresh_interval
vaild vaule:
greater than or equal to 1
Time interval to auto refresh the metadata. Same as the
refresh_intervalparameter in lua-resty-kafka, but in seconds.name
Unique identifier of the plugin for the batch processor. If you use Prometheus to monitor APISIX metrics, the name is exported in
apisix_batch_process_entries.batch_max_size
vaild vaule:
greater than 0
The number of log entries allowed in one batch. Once reached, the batch will be sent to the logging service. Setting this parameter to 1 means immediate processing.
inactive_timeout
vaild vaule:
greater than 0
The maximum time in seconds to wait for new logs before sending the batch to the logging service. The value should be smaller than
buffer_duration.buffer_duration
vaild vaule:
greater than 0
The maximum time in seconds from the earliest entry allowed before sending the batch to the logging service.
retry_delay
vaild vaule:
greater than or equal to 0
The time interval in seconds to retry sending the batch to the logging service if the batch was not successfully sent.
max_retry_count
vaild vaule:
greater than or equal to 0
The maximum number of unsuccessful retries allowed before dropping the log entries.
Plugin Metadata
log_format
Custom log format using key-value pairs in JSON format. Values can reference built-in variables.
In APISIX from 3.15.0, log format nested structures are supported up to five levels deep. In API7 Enterprise, only flat key-value structures are supported; nested structures are not yet supported.
log_format_extra
Additional log fields to overlay on the default log entry, using key-value pairs in JSON format. Values can reference built-in variables. Unlike
log_format, which replaces the entire default log entry,log_format_extrakeeps the default entry and adds the configured fields on top of it. Whenlog_formatis set,log_format_extrais ignored. Available in API7 Enterprise from version 3.9.15.max_pending_entries
vaild vaule:
greater than or equal to 1
Maximum number of unprocessed entries allowed in the batch processor. When this limit is reached, new entries will be dropped until the backlog is reduced. This setting does not limit the async Kafka producer's local buffer.