rocketmq-logger

The rocketmq-logger plugin pushes request and response logs as JSON objects to your RocketMQ clusters in batches and supports the customization of log formats.

Examples

The examples below demonstrate how you can configure rocketmq-logger plugin for different scenarios.

To follow along the examples, start a sample RocketMQ cluster:

Docker
Kubernetes

docker-compose.yml
version: "3"

services:
  rocketmq_namesrv:
    image: apacherocketmq/rocketmq:4.6.0
    container_name: rmqnamesrv
    restart: unless-stopped
    ports:
      - "9876:9876"
    command: sh mqnamesrv
    networks:
      rocketmq_net:

  rocketmq_broker:
    image: apacherocketmq/rocketmq:4.6.0
    container_name: rmqbroker
    restart: unless-stopped
    ports:
      - "10909:10909"
      - "10911:10911"
      - "10912:10912"
    depends_on:
      - rocketmq_namesrv
    command: sh mqbroker -n rmqnamesrv:9876 -c ../conf/broker.conf
    networks:
      rocketmq_net:

networks:
  rocketmq_net:

Start containers:

docker compose up -d

In a few seconds, the name server and broker should start.

Create the TopicTest topic:

docker exec -i rmqnamesrv rm /home/rocketmq/rocketmq-4.6.0/conf/tools.yml
docker exec -i rmqnamesrv /home/rocketmq/rocketmq-4.6.0/bin/mqadmin updateTopic -n rmqnamesrv:9876 -t TopicTest -c DefaultCluster

Create a Kubernetes manifest file for the RocketMQ name server and broker deployments:

rocketmq-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: aic
  name: rocketmq-namesrv
spec:
  replicas: 1
  selector:
    matchLabels:
      app: rocketmq-namesrv
  template:
    metadata:
      labels:
        app: rocketmq-namesrv
    spec:
      containers:
      - name: rocketmq-namesrv
        image: apacherocketmq/rocketmq:4.6.0
        command: ["sh", "mqnamesrv"]
        ports:
        - containerPort: 9876
---
apiVersion: v1
kind: Service
metadata:
  namespace: aic
  name: rocketmq-namesrv
spec:
  selector:
    app: rocketmq-namesrv
  ports:
  - port: 9876
    targetPort: 9876
  type: ClusterIP
---
apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: aic
  name: rocketmq-broker
spec:
  replicas: 1
  selector:
    matchLabels:
      app: rocketmq-broker
  template:
    metadata:
      labels:
        app: rocketmq-broker
    spec:
      containers:
      - name: rocketmq-broker
        image: apacherocketmq/rocketmq:4.6.0
        command: ["sh", "mqbroker", "-n", "rocketmq-namesrv:9876", "-c", "../conf/broker.conf"]
        ports:
        - containerPort: 10909
        - containerPort: 10911
        - containerPort: 10912
---
apiVersion: v1
kind: Service
metadata:
  namespace: aic
  name: rocketmq-broker
spec:
  selector:
    app: rocketmq-broker
  ports:
  - name: fastlisten
    port: 10909
    targetPort: 10909
  - name: listen
    port: 10911
    targetPort: 10911
  - name: haservice
    port: 10912
    targetPort: 10912
  type: ClusterIP

Apply the manifests:

kubectl apply -f rocketmq-deployment.yaml

Once the pods are running, create the TopicTest topic:

kubectl exec -n aic deploy/rocketmq-namesrv -- sh -c \
  "rm -f /home/rocketmq/rocketmq-4.6.0/conf/tools.yml && \
   /home/rocketmq/rocketmq-4.6.0/bin/mqadmin updateTopic \
   -n rocketmq-namesrv:9876 -t TopicTest -c DefaultCluster"

Wait for messages in the configured RocketMQ topic:

Docker
Kubernetes

docker run -it --name rockemq_consumer -e NAMESRV_ADDR=localhost:9876 --net host apacherocketmq/rocketmq:4.6.0 sh tools.sh org.apache.rocketmq.example.quickstart.Consumer

In a few seconds, the consumer should start and listen for messages from APISIX:

01:32:17.823 [main] DEBUG i.n.u.i.l.InternalLoggerFactory - Using SLF4J as the default logging framework
Consumer Started.

Open a new terminal session for the following steps working with APISIX.

After sending requests to APISIX in the following examples, run the following command to print messages from the TopicTest topic:

kubectl exec -n aic deploy/rocketmq-namesrv -- sh -c \
  "/home/rocketmq/rocketmq-4.6.0/bin/mqadmin printMsg \
   -n rocketmq-namesrv:9876 -t TopicTest"

Log in Different Meta Log Formats

The following example demonstrates how you can enable the rocketmq-logger plugin on a route, which logs client requests to the route and pushes logs to RocketMQ. You will also understand the differences between the default and origin meta log formats.

Create a route with rocketmq-logger as follows:

Admin API
ADC
Ingress Controller

curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d '{
    "id": "rocketmq-logger-route",
    "uri": "/anything",
    "plugins": {
      "rocketmq-logger": {
        "nameserver_list": [ "127.0.0.1:9876" ],
        "topic": "TopicTest",
        "key": "key1",
        "timeout": 30,
        "meta_format": "default",
        "batch_max_size": 1
      }
    },
    "upstream": {
      "nodes": {
        "httpbin.org:80": 1
      },
      "type": "roundrobin"
    }
  }'

adc.yaml
services:
  - name: httpbin
    routes:
      - uris:
          - /anything
        name: rocketmq-logger-route
        plugins:
          rocketmq-logger:
            nameserver_list:
              - "127.0.0.1:9876"
            topic: "TopicTest"
            key: "key1"
            timeout: 30
            meta_format: "default"
            batch_max_size: 1
    upstream:
      type: roundrobin
      nodes:
        - host: httpbin.org
          port: 80
          weight: 1

Synchronize the configuration to the gateway:

adc sync -f adc.yaml

Gateway API
APISIX CRD

rocketmq-logger-ic.yaml
apiVersion: v1
kind: Service
metadata:
  namespace: aic
  name: httpbin-external-domain
spec:
  type: ExternalName
  externalName: httpbin.org
---
apiVersion: apisix.apache.org/v1alpha1
kind: PluginConfig
metadata:
  namespace: aic
  name: rocketmq-logger-plugin-config
spec:
  plugins:
    - name: rocketmq-logger
      config:
        nameserver_list:
          - "rocketmq-namesrv.aic.svc:9876"
        topic: "TopicTest"
        key: "key1"
        timeout: 30
        meta_format: "default"
        batch_max_size: 1
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  namespace: aic
  name: rocketmq-logger-route
spec:
  parentRefs:
    - name: apisix
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /anything
      filters:
        - type: ExtensionRef
          extensionRef:
            group: apisix.apache.org
            kind: PluginConfig
            name: rocketmq-logger-plugin-config
      backendRefs:
        - name: httpbin-external-domain
          port: 80

rocketmq-logger-ic.yaml
apiVersion: apisix.apache.org/v2
kind: ApisixUpstream
metadata:
  namespace: aic
  name: httpbin-external-domain
spec:
  ingressClassName: apisix
  externalNodes:
  - type: Domain
    name: httpbin.org
---
apiVersion: apisix.apache.org/v2
kind: ApisixRoute
metadata:
  namespace: aic
  name: rocketmq-logger-route
spec:
  ingressClassName: apisix
  http:
    - name: rocketmq-logger-route
      match:
        paths:
          - /anything*
      upstreams:
        - name: httpbin-external-domain
      plugins:
        - name: rocketmq-logger
          enable: true
          config:
            nameserver_list:
              - "rocketmq-namesrv.aic.svc:9876"
            topic: "TopicTest"
            key: "key1"
            timeout: 30
            meta_format: "default"
            batch_max_size: 1

Apply the configuration:

kubectl apply -f rocketmq-logger-ic.yaml

❶ meta_format: set to the default log format.

❷ batch_max_size: set to 1 to send the log entry immediately.

Send a request to the route to generate a log entry:

curl -i "http://127.0.0.1:9080/anything"

You should see a log entry similar to the following:

{
  "client_ip": "127.0.0.1",
  "upstream": "34.197.122.172:80",
  "start_time": 1744727400000,
  "request": {
    "headers": {
      "host": "127.0.0.1:9080",
      "accept": "*/*",
      "user-agent": "curl/8.6.0"
    },
    "querystring": {},
    "size": 86,
    "uri": "/anything",
    "url": "http://127.0.0.1:9080/anything",
    "method": "GET"
  },
  "route_id": "rocketmq-logger-route",
  "apisix_latency": 8.9998455047607,
  "upstream_latency": 503,
  "latency": 511.99984550476,
  "response": {
    "size": 617,
    "headers": {
      "content-length": "391",
      "connection": "close",
      "date": "Tue, 15 Apr 2025 14:30:00 GMT",
      "server": "APISIX/3.15.0",
      "content-type": "application/json"
    },
    "status": 200
  },
  "server": {
    "hostname": "apisix",
    "version": "3.15.0"
  },
  "service_id": ""
}

Update the rocketmq-logger meta log format to origin:

Admin API
ADC
Ingress Controller

curl "http://127.0.0.1:9180/apisix/admin/routes/rocketmq-logger-route" -X PATCH \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d '{
    "plugins": {
      "rocketmq-logger": {
        "meta_format": "origin"
      }
    }
  }'

Update adc.yaml to change meta_format to origin:

adc.yaml
services:
  - name: httpbin
    routes:
      - uris:
          - /anything
        name: rocketmq-logger-route
        plugins:
          rocketmq-logger:
            nameserver_list:
              - "127.0.0.1:9876"
            topic: "TopicTest"
            key: "key1"
            timeout: 30
            meta_format: "origin"
            batch_max_size: 1
    upstream:
      type: roundrobin
      nodes:
        - host: httpbin.org
          port: 80
          weight: 1

Synchronize the configuration to the gateway:

adc sync -f adc.yaml

Gateway API
APISIX CRD

Update rocketmq-logger-ic.yaml to change meta_format to origin in the PluginConfig:

rocketmq-logger-ic.yaml
apiVersion: apisix.apache.org/v1alpha1
kind: PluginConfig
metadata:
  namespace: aic
  name: rocketmq-logger-plugin-config
spec:
  plugins:
    - name: rocketmq-logger
      config:
        nameserver_list:
          - "rocketmq-namesrv.aic.svc:9876"
        topic: "TopicTest"
        key: "key1"
        timeout: 30
        meta_format: "origin"
        batch_max_size: 1

Apply the updated configuration:

kubectl apply -f rocketmq-logger-ic.yaml

Update rocketmq-logger-ic.yaml to change meta_format to origin in the ApisixRoute:

rocketmq-logger-ic.yaml
apiVersion: apisix.apache.org/v2
kind: ApisixRoute
metadata:
  namespace: aic
  name: rocketmq-logger-route
spec:
  ingressClassName: apisix
  http:
    - name: rocketmq-logger-route
      match:
        paths:
          - /anything*
      upstreams:
        - name: httpbin-external-domain
      plugins:
        - name: rocketmq-logger
          enable: true
          config:
            nameserver_list:
              - "rocketmq-namesrv.aic.svc:9876"
            topic: "TopicTest"
            key: "key1"
            timeout: 30
            meta_format: "origin"
            batch_max_size: 1

Apply the updated configuration:

kubectl apply -f rocketmq-logger-ic.yaml

Send a request to the route again to generate a new log entry:

curl -i "http://127.0.0.1:9080/anything"

You should see a log entry in the raw HTTP request format:

GET /anything HTTP/1.1
host: 127.0.0.1:9080
user-agent: curl/8.6.0
accept: */*

Log Request and Response Headers With Plugin Metadata

The following example demonstrates how you can customize log format using plugin metadata and built-in variables to log specific headers from request and response.

In APISIX, plugin metadata is used to configure the common metadata fields of all plugin instances of the same plugin. It is useful when a plugin is enabled across multiple resources and requires a universal update to their metadata fields.

First, create a route with rocketmq-logger as follows:

Admin API
ADC
Ingress Controller

curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d '{
    "id": "rocketmq-logger-route",
    "uri": "/anything",
    "plugins": {
      "rocketmq-logger": {
        "nameserver_list": [ "127.0.0.1:9876" ],
        "topic": "TopicTest",
        "key": "key1",
        "timeout": 30,
        "meta_format": "default",
        "batch_max_size": 1
      }
    },
    "upstream": {
      "nodes": {
        "httpbin.org:80": 1
      },
      "type": "roundrobin"
    }
  }'

adc.yaml
services:
  - name: httpbin
    routes:
      - uris:
          - /anything
        name: rocketmq-logger-route
        plugins:
          rocketmq-logger:
            nameserver_list:
              - "127.0.0.1:9876"
            topic: "TopicTest"
            key: "key1"
            timeout: 30
            meta_format: "default"
            batch_max_size: 1
    upstream:
      type: roundrobin
      nodes:
        - host: httpbin.org
          port: 80
          weight: 1

Synchronize the configuration to the gateway:

adc sync -f adc.yaml

Gateway API
APISIX CRD

rocketmq-logger-ic.yaml
apiVersion: v1
kind: Service
metadata:
  namespace: aic
  name: httpbin-external-domain
spec:
  type: ExternalName
  externalName: httpbin.org
---
apiVersion: apisix.apache.org/v1alpha1
kind: PluginConfig
metadata:
  namespace: aic
  name: rocketmq-logger-plugin-config
spec:
  plugins:
    - name: rocketmq-logger
      config:
        nameserver_list:
          - "rocketmq-namesrv.aic.svc:9876"
        topic: "TopicTest"
        key: "key1"
        timeout: 30
        meta_format: "default"
        batch_max_size: 1
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  namespace: aic
  name: rocketmq-logger-route
spec:
  parentRefs:
    - name: apisix
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /anything
      filters:
        - type: ExtensionRef
          extensionRef:
            group: apisix.apache.org
            kind: PluginConfig
            name: rocketmq-logger-plugin-config
      backendRefs:
        - name: httpbin-external-domain
          port: 80

rocketmq-logger-ic.yaml
apiVersion: apisix.apache.org/v2
kind: ApisixUpstream
metadata:
  namespace: aic
  name: httpbin-external-domain
spec:
  ingressClassName: apisix
  externalNodes:
  - type: Domain
    name: httpbin.org
---
apiVersion: apisix.apache.org/v2
kind: ApisixRoute
metadata:
  namespace: aic
  name: rocketmq-logger-route
spec:
  ingressClassName: apisix
  http:
    - name: rocketmq-logger-route
      match:
        paths:
          - /anything*
      upstreams:
        - name: httpbin-external-domain
      plugins:
        - name: rocketmq-logger
          enable: true
          config:
            nameserver_list:
              - "rocketmq-namesrv.aic.svc:9876"
            topic: "TopicTest"
            key: "key1"
            timeout: 30
            meta_format: "default"
            batch_max_size: 1

Apply the configuration:

kubectl apply -f rocketmq-logger-ic.yaml

❶ meta_format: set to the default log format. It is important to note that this is mandatory if you would like to customize log format with plugin metadata. If meta_format is set to origin, the log entries will remain in origin format.

❷ batch_max_size: set to 1 to send the log entry immediately.

Next, configure the plugin metadata for rocketmq-logger:

Admin API
ADC
Ingress Controller

curl "http://127.0.0.1:9180/apisix/admin/plugin_metadata/rocketmq-logger" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d '{
    "log_format": {
      "host": "$host",
      "@timestamp": "$time_iso8601",
      "client_ip": "$remote_addr",
      "env": "$http_env",
      "resp_content_type": "$sent_http_Content_Type"
    }
  }'

adc.yaml
plugin_metadata:
  - name: rocketmq-logger
    log_format:
      host: "$host"
      "@timestamp": "$time_iso8601"
      client_ip: "$remote_addr"
      env: "$http_env"
      resp_content_type: "$sent_http_Content_Type"

Synchronize the configuration to the gateway:

adc sync -f adc.yaml

rocketmq-logger-metadata.yaml
apiVersion: apisix.apache.org/v1alpha1
kind: GatewayProxy
metadata:
  namespace: aic
  name: apisix-config
spec:
  provider:
    type: ControlPlane
    controlPlane:
      service:
        name: apisix-admin
        port: 9180
      auth:
        type: AdminKey
        adminKey:
          value: edd1c9f034335f136f87ad84b625c8f1
  pluginMetadata:
    rocketmq-logger:
      log_format:
        host: "$host"
        "@timestamp": "$time_iso8601"
        client_ip: "$remote_addr"
        env: "$http_env"
        resp_content_type: "$sent_http_Content_Type"

Apply the configuration:

kubectl apply -f rocketmq-logger-metadata.yaml

❶ log the custom request header env.

❷ log the response header Content-Type.

Send a request to the route with the env header:

curl -i "http://127.0.0.1:9080/anything" -H "env: dev"

You should see a log entry similar to the following:

{
  "host": "127.0.0.1",
  "client_ip": "127.0.0.1",
  "resp_content_type": "application/json",
  "route_id": "rocketmq-logger-route",
  "env": "dev",
  "@timestamp": "2025-04-15T14:30:00+00:00"
}

Log Request Bodies Conditionally

The following example demonstrates how you can conditionally log request body.

Create a route with rocketmq-logger as follows:

Admin API
ADC
Ingress Controller

curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d '{
    "plugins": {
      "rocketmq-logger": {
        "nameserver_list": [ "127.0.0.1:9876" ],
        "topic": "TopicTest",
        "key": "key1",
        "timeout": 30,
        "meta_format": "default",
        "batch_max_size": 1,
        "include_req_body": true,
        "include_req_body_expr": [["arg_log_body", "==", "yes"]]
      }
    },
    "upstream": {
      "nodes": {
        "httpbin.org:80": 1
      },
      "type": "roundrobin"
    },
  "uri": "/anything",
  "id": "rocketmq-logger-route"
}'

adc.yaml
services:
  - name: httpbin
    routes:
      - uris:
          - /anything
        name: rocketmq-logger-route
        plugins:
          rocketmq-logger:
            nameserver_list:
              - "127.0.0.1:9876"
            topic: "TopicTest"
            key: "key1"
            timeout: 30
            meta_format: "default"
            batch_max_size: 1
            include_req_body: true
            include_req_body_expr:
              - - "arg_log_body"
                - "=="
                - "yes"
    upstream:
      type: roundrobin
      nodes:
        - host: httpbin.org
          port: 80
          weight: 1

Synchronize the configuration to the gateway:

adc sync -f adc.yaml

Gateway API
APISIX CRD

rocketmq-logger-ic.yaml
apiVersion: v1
kind: Service
metadata:
  namespace: aic
  name: httpbin-external-domain
spec:
  type: ExternalName
  externalName: httpbin.org
---
apiVersion: apisix.apache.org/v1alpha1
kind: PluginConfig
metadata:
  namespace: aic
  name: rocketmq-logger-plugin-config
spec:
  plugins:
    - name: rocketmq-logger
      config:
        nameserver_list:
          - "rocketmq-namesrv.aic.svc:9876"
        topic: "TopicTest"
        key: "key1"
        timeout: 30
        meta_format: "default"
        batch_max_size: 1
        include_req_body: true
        include_req_body_expr:
          - - "arg_log_body"
            - "=="
            - "yes"
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  namespace: aic
  name: rocketmq-logger-route
spec:
  parentRefs:
    - name: apisix
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /anything
      filters:
        - type: ExtensionRef
          extensionRef:
            group: apisix.apache.org
            kind: PluginConfig
            name: rocketmq-logger-plugin-config
      backendRefs:
        - name: httpbin-external-domain
          port: 80

rocketmq-logger-ic.yaml
apiVersion: apisix.apache.org/v2
kind: ApisixUpstream
metadata:
  namespace: aic
  name: httpbin-external-domain
spec:
  ingressClassName: apisix
  externalNodes:
  - type: Domain
    name: httpbin.org
---
apiVersion: apisix.apache.org/v2
kind: ApisixRoute
metadata:
  namespace: aic
  name: rocketmq-logger-route
spec:
  ingressClassName: apisix
  http:
    - name: rocketmq-logger-route
      match:
        paths:
          - /anything*
      upstreams:
        - name: httpbin-external-domain
      plugins:
        - name: rocketmq-logger
          enable: true
          config:
            nameserver_list:
              - "rocketmq-namesrv.aic.svc:9876"
            topic: "TopicTest"
            key: "key1"
            timeout: 30
            meta_format: "default"
            batch_max_size: 1
            include_req_body: true
            include_req_body_expr:
              - - "arg_log_body"
                - "=="
                - "yes"

Apply the configuration:

kubectl apply -f rocketmq-logger-ic.yaml

❶ include_req_body: set to true to include request body.

❷ include_req_body_expr: only include request body if the URL query string log_body is yes.

Send a request to the route with an URL query string satisfying the condition:

curl -i "http://127.0.0.1:9080/anything?log_body=yes" -X POST -d '{"env": "dev"}'

You should see the request body logged:

{
  ...,
    "method": "POST",
    "body": "{\"env\": \"dev\"}",
    "size": 183
  }
}

Send a request to the route without any URL query string:

curl -i "http://127.0.0.1:9080/anything" -X POST -d '{"env": "dev"}'

You should not observe the request body in the log.

info

If you have customized the log_format in addition to setting include_req_body or include_resp_body to true, the plugin would not include the bodies in the logs.

As a workaround, you may be able to use the NGINX variable $request_body in the log format, such as:

{
  "rocketmq-logger": {
    ...,
    "log_format": {"body": "$request_body"}
  }
}

Examples​

Log in Different Meta Log Formats​

Log Request and Response Headers With Plugin Metadata​

Log Request Bodies Conditionally​

Examples

Log in Different Meta Log Formats

Log Request and Response Headers With Plugin Metadata

Log Request Bodies Conditionally