# Kafka Logger Configuration

The `kafka-logger` plugin pushes request and response logs as JSON objects to Apache Kafka clusters in batches, allowing for customizable log formats to enhance data management.<!-- --> <!-- -->Configuration covers plugin parameters and plugin metadata.

## Parameters[​](#parameters "Direct link to Parameters")

See plugin [common configurations](https://docs.api7.ai/apisix/reference/plugin-common-configurations.md) for configuration options available to all plugins.

* broker\_list

  object

  ***

  Deprecated. Use `brokers` instead. A map of Kafka broker hosts to their ports. Configure either `broker_list` or `brokers`.

* brokers

  array

  valid value:

  greater than 0

  ***

  List of Kafka broker nodes. Configure either `brokers` or the deprecated `broker_list`.

  * host

    string

    required

    ***

    The host of Kafka broker.

  * port

    integer

    required

    ***

    The port of Kafka broker.

  * sasl\_config

    object

    ***

    The SASL configuration of Kafka broker

    * mechanism

      string

      default: `PLAIN`

      valid value:

      `PLAIN`, `SCRAM-SHA-256`, or `SCRAM-SHA-512`

      ***

      The mechanism of SASL configuration.

      The `SCRAM-SHA-256` and `SCRAM-SHA-512` options are available in API7 Enterprise from version 3.8.16 and APISIX from version 3.15.0.

    * user

      string

      required

      ***

      The user of SASL configuration.

    * password

      string

      required

      ***

      The password of SASL configuration. The value is encrypted with AES before being stored in etcd.

* tls

  object

  ***

  TLS configuration for connecting to Kafka brokers. Setting this object makes the plugin connect over TLS; omit it to connect in plaintext. Introduced in API7 Enterprise 3.9.17 and 3.10.4, and APISIX 3.18.0.

  * verify

    boolean

    default: `false`

    ***

    If true, verify the Kafka broker's TLS certificate against the configured trusted CA store.

    The default is `false`, so enabling `tls` encrypts the connection but does not authenticate the broker, which leaves it open to an active man-in-the-middle. Set it to `true` in production.

* kafka\_topic

  string

  required

  ***

  Target topic to push the logs for organization.

* producer\_type

  string

  default: `async`

  valid value:

  `async` or `sync`

  ***

  Kafka producer mode. In `async` mode, messages are buffered locally before being sent to Kafka. In `sync` mode, messages are sent without using the async producer buffer.

* required\_acks

  integer

  default: `1`

  valid value:

  -1 or 1

  ***

  Number of acknowledgements the leader needs to receive for the producer to consider the request complete. This controls the durability of the sent records. See [Kafka documentation](https://kafka.apache.org/documentation/#producerconfigs_acks) for more information. `acks=0` is not yet supported.

* api\_version

  integer

  default: `1`

  valid value:

  `0`, `1`, or `2`

  ***

  Kafka Produce API version used to send messages to the broker. Only version `2` carries the message timestamp, allowing the broker to store it; with the default `1`, messages may be recorded without a usable timestamp. Kafka 0.10 or later is required for version `2`.

  Kafka 4 removes Produce API versions 0–2 and is not compatible with this plugin. Use Kafka 3.x until the plugin supports Produce API version 3 or later.

  Introduced in API7 Enterprise 3.9.14 and 3.10.1, and APISIX 3.18.0.

* key

  string

  ***

  Key used for allocating partitions for messages.

* timeout

  integer

  default: `3`

  valid value:

  greater than 0

  ***

  Timeout for the upstream to send data.

* meta\_format

  string

  default: `default`

  valid value:

  `default` or `origin`

  ***

  Format to collect the request information. Setting to `default` collects the information in JSON format and `origin` collects the information with the original HTTP request. See the [example](https://docs.api7.ai/hub/kafka-logger.md#log-in-different-meta-log-formats) for more details.

* log\_format

  object

  ***

  Custom log format using key-value pairs in JSON format. Values can reference [built-in variables](https://docs.api7.ai/apisix/reference/built-in-variables.md).

  When `meta_format` is `default`, if `include_req_body` or `include_resp_body` is true, reference `$request_body` or `$resp_body` to add the collected body to the custom log entry. The options do not add body fields to a custom format automatically.

  When `meta_format` is `origin`, this parameter is ignored.

  In APISIX from 3.15.0, log format nested structures are supported up to five levels deep. In API7 Enterprise, only flat key-value structures are supported; nested structures are not yet supported.

  You can also configure log format on a global scale using the [plugin metadata](https://docs.api7.ai/apisix/key-concepts/plugin-metadata.md), which configures the log format for all `kafka-logger` plugin instances. If the log format configured on the individual plugin instance differs from the log format configured on plugin metadata, the log format configured on the individual plugin instance takes precedence.

* log\_format\_extra

  object

  ***

  Additional fields to add to the default log entry, using key-value pairs in JSON format. Values can reference [built-in variables](https://docs.api7.ai/apisix/reference/built-in-variables.md). A configured field does not overwrite an existing default field. A plugin instance takes precedence over plugin metadata; setting an empty object on the instance disables the metadata value. When `log_format` is configured or `meta_format` is `origin`, `log_format_extra` is ignored.

  Configure additional fields for all `kafka-logger` instances with [plugin metadata](https://docs.api7.ai/apisix/key-concepts/plugin-metadata.md). See the [example](https://docs.api7.ai/hub/kafka-logger.md#add-request-and-response-headers-with-plugin-metadata) for more details.

  Introduced in API7 Enterprise 3.9.15 and 3.10.2, and APISIX 3.18.0.

* include\_req\_body

  boolean

  default: `false`

  ***

  If true, collect the request body. When `meta_format` is `default`, add it to the default log fields. When `meta_format` is `origin`, append it to the original HTTP request. Request bodies buffered in memory or a temporary file can be logged, and `max_req_body_bytes` limits the logged body size.

* include\_req\_body\_expr

  array\[array]

  ***

  An array of one or more conditions in the form of [APISIX expressions](https://docs.api7.ai/apisix/reference/apisix-expressions.md). When `include_req_body` is true, the expressions control proactive body reading and the `request.body` field in the default log format. A custom `log_format` or `log_format_extra` field that references `$request_body` can still include a body already buffered by NGINX when the expressions evaluate to false. The `origin` meta format also appends any available body without reevaluating these expressions. Do not use this option as a data-redaction boundary.

* max\_req\_body\_bytes

  integer

  default: `524288`

  valid value:

  greater than or equal to 1

  ***

  Maximum request body allowed in bytes. Request bodies falling within this limit will be pushed to Kafka. If the size exceeds the configured value, the body will be truncated before being pushed to Kafka.

* include\_resp\_body

  boolean

  default: `false`

  ***

  If true, collect the response body. With the default log fields, it is added to `response.body`. With a custom `log_format`, reference `$resp_body` to add it to the log entry. When `meta_format` is `origin`, the response body is not added. The `max_resp_body_bytes` parameter limits the logged body size.

* include\_resp\_body\_expr

  array\[array]

  ***

  An array of one or more conditions in the form of [APISIX expressions](https://docs.api7.ai/apisix/reference/apisix-expressions.md). Used when the `include_resp_body` is true. Response body would only be logged when the expressions configured here evaluate to true.

* max\_resp\_body\_bytes

  integer

  default: `524288`

  valid value:

  greater than or equal to 1

  ***

  Maximum response body allowed in bytes. Response bodies falling within this limit will be pushed to Kafka. If the size exceeds the configured value, the body will be truncated before being pushed to Kafka.

* cluster\_name

  integer

  default: `1`

  valid value:

  greater than or equal to 1

  ***

  Name of the cluster. Used when there are two or more Kafka clusters. Only works if `producer_type` is set to `async`.

* producer\_batch\_num

  integer

  default: `200`

  valid value:

  greater than or equal to 1

  ***

  The number of messages to send in one batch. Same as the `batch_num` parameter in [lua-resty-kafka](https://github.com/doujiang24/lua-resty-kafka).

* producer\_batch\_size

  integer

  default: `1048576`

  valid value:

  greater than or equal to 0

  ***

  The size of the TCP send buffer to use when sending data. Same as the `batch_size` parameter in [lua-resty-kafka](https://github.com/doujiang24/lua-resty-kafka), but in bytes.

* producer\_max\_buffering

  integer

  default: `50000`

  valid value:

  greater than or equal to 1

  ***

  Maximum number of Kafka producer messages that the async producer can buffer locally. Same as the `max_buffering` parameter in [lua-resty-kafka](https://github.com/doujiang24/lua-resty-kafka). This buffer is not capped by `max_pending_entries`. Memory usage also depends on worker count, `batch_max_size`, log format, and whether request or response bodies are logged.

* producer\_time\_linger

  integer

  default: `1`

  valid value:

  greater than or equal to 1

  ***

  Flush time. Same as the `flush_time` parameter in [lua-resty-kafka](https://github.com/doujiang24/lua-resty-kafka), but in seconds.

* meta\_refresh\_interval

  integer

  default: `30`

  valid value:

  greater than or equal to 1

  ***

  Time interval to auto refresh the metadata. Same as the `refresh_interval` parameter in [lua-resty-kafka](https://github.com/doujiang24/lua-resty-kafka), but in seconds.

* name

  string

  default: `kafka logger`

  ***

  Unique identifier of the plugin for the batch processor. If you use [Prometheus](https://docs.api7.ai/hub/prometheus.md) to monitor APISIX metrics, the name is exported in `apisix_batch_process_entries`.

* batch\_max\_size

  integer

  default: `1000`

  valid value:

  greater than 0

  ***

  The number of log entries allowed in one batch. Once reached, the batch will be sent to the logging service. Setting this parameter to 1 means immediate processing.

* inactive\_timeout

  integer

  default: `5`

  valid value:

  greater than 0

  ***

  The maximum time in seconds to wait for new logs before sending the batch to the logging service. The value should be smaller than `buffer_duration`.

* buffer\_duration

  integer

  default: `60`

  valid value:

  greater than 0

  ***

  The maximum time in seconds from the earliest entry allowed before sending the batch to the logging service.

* retry\_delay

  integer

  default: `1`

  valid value:

  greater than or equal to 0

  ***

  The time interval in seconds to retry sending the batch to the logging service if the batch was not successfully sent.

* max\_retry\_count

  integer

  default: `0`

  valid value:

  greater than or equal to 0

  ***

  The maximum number of unsuccessful retries allowed before dropping the log entries.

## Plugin Metadata[​](#plugin-metadata "Direct link to Plugin Metadata")

* log\_format

  object

  ***

  Custom log format using key-value pairs in JSON format. Values can reference [built-in variables](https://docs.api7.ai/apisix/reference/built-in-variables.md).

  Ignored by plugin instances where `meta_format` is `origin`.

  In APISIX from 3.15.0, log format nested structures are supported up to five levels deep. In API7 Enterprise, only flat key-value structures are supported; nested structures are not yet supported.

* log\_format\_extra

  object

  ***

  Additional fields to add to the default log entry, using key-value pairs in JSON format. Values can reference [built-in variables](https://docs.api7.ai/apisix/reference/built-in-variables.md). A configured field does not overwrite an existing default field. A plugin instance takes precedence over plugin metadata; setting an empty object on the instance disables the metadata value. When `log_format` is configured or a plugin instance uses `meta_format: origin`, `log_format_extra` is ignored.

  Introduced in API7 Enterprise 3.9.15 and 3.10.2, and APISIX 3.18.0.

* max\_pending\_entries

  integer

  default: `8192`

  valid value:

  greater than or equal to 1

  ***

  Maximum number of entries waiting in the batch processor. New entries are discarded when the backlog reaches the limit. This setting does not limit the async Kafka producer's local buffer.

  The default changed to `8192` in APISIX 3.18.0 and in API7 Enterprise 3.9.19 on the 3.9 line and 3.10.6 on the 3.10 line. In API7 Enterprise 3.9.18 and 3.10.5, and in earlier APISIX versions, omitting the parameter leaves the backlog unlimited.

  See [Batch Processor](https://docs.api7.ai/apisix/reference/batch-processor.md#configure-the-pending-entry-limit) for sizing and verification guidance.
