mqtt-proxy
The mqtt-proxy plugin validates the protocol name and level in an MQTT CONNECT packet before proxying the TCP connection to an MQTT broker. It supports MQTT 3.1.x and MQTT 5.0 and is configured on a stream route.
The plugin also exposes the MQTT client ID as mqtt_client_id. A consistent-hash upstream can use this variable to keep connections from the same client ID on the same broker.
Examples
The examples configure a stream listener and two Eclipse Mosquitto brokers before proxying MQTT 3.1.1 and MQTT 5 traffic. The final example adds consistent hashing to keep connections with the same client ID on the same broker.
Configure the Stream Listener
The gateway must listen for stream traffic on the port used by the stream route.
- Host or Docker
- Kubernetes (Helm)
Add the stream listener to config.yaml:
apisix:
proxy_mode: http&stream
stream_proxy:
tcp:
- 9100
Reload the gateway after saving the configuration.
For the APISIX Helm chart, enable the stream service and expose the listener:
service:
stream:
enabled: true
tcp:
- 9100
For the API7 Gateway Helm chart, configure the equivalent listener:
gateway:
stream:
enabled: true
tcp:
- addr: 9100
Apply the complete values file with the chart and version already used by the deployment.
Start the Example MQTT Brokers
The runnable examples use Docker or Kubernetes. Both workflows start two Eclipse Mosquitto brokers in the same environment as the gateway.
- Docker
- Kubernetes
Create the broker configuration:
listener 1883 0.0.0.0
allow_anonymous true
persistence false
log_dest stdout
Set GATEWAY_CONTAINER to the running APISIX or API7 Gateway container:
export GATEWAY_CONTAINER=replace-with-gateway-container-name
Create a dedicated network and connect the gateway:
docker network create gateway-mqtt-proxy-net
docker network connect gateway-mqtt-proxy-net "$GATEWAY_CONTAINER"
Start two brokers on the same network. The network aliases match the Kubernetes Service addresses used below, so the same route configuration works in both environments:
docker run -d \
--name mqtt-broker-1 \
--network gateway-mqtt-proxy-net \
--network-alias mqtt-broker-1.aic.svc.cluster.local \
-v "$PWD/mosquitto.conf:/mosquitto/config/mosquitto.conf:ro" \
eclipse-mosquitto:2.1.2-alpine
docker run -d \
--name mqtt-broker-2 \
--network gateway-mqtt-proxy-net \
--network-alias mqtt-broker-2.aic.svc.cluster.local \
-v "$PWD/mosquitto.conf:/mosquitto/config/mosquitto.conf:ro" \
eclipse-mosquitto:2.1.2-alpine
Create the broker configuration, a headless Service, and a two-replica StatefulSet:
apiVersion: v1
kind: ConfigMap
metadata:
namespace: aic
name: mqtt-broker-config
data:
mosquitto.conf: |
listener 1883 0.0.0.0
allow_anonymous true
persistence false
log_dest stdout
---
apiVersion: v1
kind: Service
metadata:
namespace: aic
name: mqtt-brokers
spec:
clusterIP: None
selector:
app: mqtt-broker
ports:
- name: mqtt
port: 1883
targetPort: mqtt
---
apiVersion: v1
kind: Service
metadata:
namespace: aic
name: mqtt-broker-1
spec:
selector:
statefulset.kubernetes.io/pod-name: mqtt-broker-0
ports:
- name: mqtt
port: 1883
targetPort: mqtt
---
apiVersion: v1
kind: Service
metadata:
namespace: aic
name: mqtt-broker-2
spec:
selector:
statefulset.kubernetes.io/pod-name: mqtt-broker-1
ports:
- name: mqtt
port: 1883
targetPort: mqtt
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
namespace: aic
name: mqtt-broker
spec:
serviceName: mqtt-brokers
replicas: 2
selector:
matchLabels:
app: mqtt-broker
template:
metadata:
labels:
app: mqtt-broker
spec:
containers:
- name: mosquitto
image: eclipse-mosquitto:2.1.2-alpine
ports:
- name: mqtt
containerPort: 1883
readinessProbe:
tcpSocket:
port: mqtt
initialDelaySeconds: 2
periodSeconds: 3
volumeMounts:
- name: config
mountPath: /mosquitto/config/mosquitto.conf
subPath: mosquitto.conf
readOnly: true
volumes:
- name: config
configMap:
name: mqtt-broker-config
Apply the resources and wait for the brokers:
kubectl apply -f mqtt-brokers.yaml
kubectl rollout status statefulset/mqtt-broker -n aic
Set GATEWAY_STREAM_HOST to the in-cluster DNS name of the gateway Service that exposes port 9100:
export GATEWAY_STREAM_HOST=replace-with-gateway-stream-service-dns
The Kubernetes examples use the default cluster domain cluster.local. If the cluster uses another domain, replace cluster.local in the Admin API and ADC broker addresses.
The fixture permits anonymous connections without encryption for evaluation. Production deployments should add authentication, encryption, and network controls appropriate to their topology. Because mqtt-proxy inspects the MQTT CONNECT packet, it cannot process connections that use TLS passthrough.
Proxy MQTT 3.1.1 Traffic
The following example configures a stream route on port 9100 to accept MQTT 3.1.1 connections and proxy them to the configured broker backend.
- Admin API
- ADC
- Ingress Controller
For APISIX, create the stream route with the Admin API:
curl "http://127.0.0.1:9180/apisix/admin/stream_routes/mqtt-route" -X PUT \
-H "X-API-KEY: ${ADMIN_API_KEY}" \
-H "Content-Type: application/json" \
-d @- <<'EOF'
{
"server_port": 9100,
"plugins": {
"mqtt-proxy": {
"protocol_name": "MQTT",
"protocol_level": 4
}
},
"upstream": {
"type": "roundrobin",
"nodes": {
"mqtt-broker-1.aic.svc.cluster.local:1883": 1
}
}
}
EOF
services:
- name: mqtt-service
labels:
docs-example: mqtt-proxy
upstream:
name: default
scheme: tcp
nodes:
- host: mqtt-broker-1.aic.svc.cluster.local
port: 1883
weight: 1
stream_routes:
- name: mqtt-route
server_port: 9100
plugins:
mqtt-proxy:
protocol_name: MQTT
protocol_level: 4
Preview changes owned by this example:
adc diff -f adc.yaml \
--include-resource-type service \
--label-selector docs-example=mqtt-proxy
Synchronize the reviewed service configuration:
adc sync -f adc.yaml \
--include-resource-type service \
--label-selector docs-example=mqtt-proxy
- Gateway API
- APISIX CRD
Ensure that the Gateway has a TCP listener named tcp on port 9100, as shown in Proxy TCP Traffic by Port. Then create a TCPRoute, configure the TCP upstream scheme, and attach the plugin with an L4RoutePolicy:
apiVersion: gateway.networking.k8s.io/v1
kind: TCPRoute
metadata:
namespace: aic
name: mqtt-route
spec:
parentRefs:
- name: apisix
sectionName: tcp
rules:
- backendRefs:
- name: mqtt-brokers
port: 1883
---
apiVersion: apisix.apache.org/v1alpha1
kind: BackendTrafficPolicy
metadata:
namespace: aic
name: mqtt-brokers
spec:
targetRefs:
- group: ""
kind: Service
name: mqtt-brokers
sectionName: mqtt
scheme: tcp
---
apiVersion: apisix.apache.org/v1alpha1
kind: L4RoutePolicy
metadata:
namespace: aic
name: mqtt-route
spec:
targetRefs:
- group: gateway.networking.k8s.io
kind: TCPRoute
name: mqtt-route
plugins:
- name: mqtt-proxy
config:
protocol_name: MQTT
protocol_level: 4
apiVersion: apisix.apache.org/v2
kind: ApisixUpstream
metadata:
namespace: aic
name: mqtt-brokers
spec:
ingressClassName: apisix
scheme: tcp
---
apiVersion: apisix.apache.org/v2
kind: ApisixRoute
metadata:
namespace: aic
name: mqtt-route
spec:
ingressClassName: apisix
stream:
- name: mqtt-route
protocol: TCP
match:
ingressPort: 9100
backend:
serviceName: mqtt-brokers
servicePort: 1883
plugins:
- name: mqtt-proxy
enable: true
config:
protocol_name: MQTT
protocol_level: 4
Apply the selected configuration:
kubectl apply -f mqtt-proxy-ic.yaml
- Docker
- Kubernetes
Open a terminal and subscribe directly to the first broker:
docker exec mqtt-broker-1 \
mosquitto_sub -h 127.0.0.1 -p 1883 \
-V mqttv311 -t docs/mqtt
In another terminal, publish through the gateway listener:
docker exec mqtt-broker-1 \
mosquitto_pub -h "$GATEWAY_CONTAINER" -p 9100 \
-V mqttv311 -t docs/mqtt -m "Hello through the gateway"
The subscriber should print Hello through the gateway.
In separate terminals, subscribe directly to each broker:
kubectl exec -n aic mqtt-broker-0 -- \
mosquitto_sub -h 127.0.0.1 -p 1883 \
-V mqttv311 -t docs/mqtt
In the second terminal, subscribe to the other broker:
kubectl exec -n aic mqtt-broker-1 -- \
mosquitto_sub -h 127.0.0.1 -p 1883 \
-V mqttv311 -t docs/mqtt
In another terminal, publish through the gateway listener:
kubectl exec -n aic mqtt-broker-0 -- \
mosquitto_pub -h "$GATEWAY_STREAM_HOST" -p 9100 \
-V mqttv311 -t docs/mqtt -m "Hello through the gateway"
One subscriber should print Hello through the gateway.
Proxy MQTT 5 Traffic
The following example configures a stream route with the protocol name MQTT and protocol level 5 to accept MQTT 5 CONNECT packets and proxy their messages through the gateway. Connections using other protocol levels are rejected.
- Admin API
- ADC
- Ingress Controller
For APISIX, create the stream route with the Admin API:
curl "http://127.0.0.1:9180/apisix/admin/stream_routes/mqtt-route" -X PUT \
-H "X-API-KEY: ${ADMIN_API_KEY}" \
-H "Content-Type: application/json" \
-d @- <<'EOF'
{
"server_port": 9100,
"plugins": {
"mqtt-proxy": {
"protocol_name": "MQTT",
"protocol_level": 5
}
},
"upstream": {
"type": "roundrobin",
"nodes": {
"mqtt-broker-1.aic.svc.cluster.local:1883": 1
}
}
}
EOF
services:
- name: mqtt-service
labels:
docs-example: mqtt-proxy
upstream:
name: default
scheme: tcp
nodes:
- host: mqtt-broker-1.aic.svc.cluster.local
port: 1883
weight: 1
stream_routes:
- name: mqtt-route
server_port: 9100
plugins:
mqtt-proxy:
protocol_name: MQTT
protocol_level: 5
Preview changes owned by this example:
adc diff -f adc.yaml \
--include-resource-type service \
--label-selector docs-example=mqtt-proxy
Synchronize the reviewed service configuration:
adc sync -f adc.yaml \
--include-resource-type service \
--label-selector docs-example=mqtt-proxy
- Gateway API
- APISIX CRD
Ensure that the Gateway has a TCP listener named tcp on port 9100, as shown in Proxy TCP Traffic by Port. Then create a TCPRoute, configure the TCP upstream scheme, and attach the plugin with an L4RoutePolicy:
apiVersion: gateway.networking.k8s.io/v1
kind: TCPRoute
metadata:
namespace: aic
name: mqtt-route
spec:
parentRefs:
- name: apisix
sectionName: tcp
rules:
- backendRefs:
- name: mqtt-brokers
port: 1883
---
apiVersion: apisix.apache.org/v1alpha1
kind: BackendTrafficPolicy
metadata:
namespace: aic
name: mqtt-brokers
spec:
targetRefs:
- group: ""
kind: Service
name: mqtt-brokers
sectionName: mqtt
scheme: tcp
---
apiVersion: apisix.apache.org/v1alpha1
kind: L4RoutePolicy
metadata:
namespace: aic
name: mqtt-route
spec:
targetRefs:
- group: gateway.networking.k8s.io
kind: TCPRoute
name: mqtt-route
plugins:
- name: mqtt-proxy
config:
protocol_name: MQTT
protocol_level: 5
apiVersion: apisix.apache.org/v2
kind: ApisixUpstream
metadata:
namespace: aic
name: mqtt-brokers
spec:
ingressClassName: apisix
scheme: tcp
---
apiVersion: apisix.apache.org/v2
kind: ApisixRoute
metadata:
namespace: aic
name: mqtt-route
spec:
ingressClassName: apisix
stream:
- name: mqtt-route
protocol: TCP
match:
ingressPort: 9100
backend:
serviceName: mqtt-brokers
servicePort: 1883
plugins:
- name: mqtt-proxy
enable: true
config:
protocol_name: MQTT
protocol_level: 5
Apply the selected configuration:
kubectl apply -f mqtt-proxy-ic.yaml
- Docker
- Kubernetes
Subscribe with an MQTT 5 client:
docker exec mqtt-broker-1 \
mosquitto_sub -h 127.0.0.1 -p 1883 \
-V mqttv5 -t docs/mqtt-v5
Publish an MQTT 5 message through the gateway:
docker exec mqtt-broker-1 \
mosquitto_pub -h "$GATEWAY_CONTAINER" -p 9100 \
-V mqttv5 -t docs/mqtt-v5 -m "Hello MQTT 5"
The subscriber should print Hello MQTT 5.
Subscribe directly to each broker in separate terminals:
kubectl exec -n aic mqtt-broker-0 -- \
mosquitto_sub -h 127.0.0.1 -p 1883 \
-V mqttv5 -t docs/mqtt-v5
In the second terminal, subscribe to the other broker:
kubectl exec -n aic mqtt-broker-1 -- \
mosquitto_sub -h 127.0.0.1 -p 1883 \
-V mqttv5 -t docs/mqtt-v5
Publish an MQTT 5 message through the gateway:
kubectl exec -n aic mqtt-broker-0 -- \
mosquitto_pub -h "$GATEWAY_STREAM_HOST" -p 9100 \
-V mqttv5 -t docs/mqtt-v5 -m "Hello MQTT 5"
One subscriber should print Hello MQTT 5.
Load Balance by MQTT Client ID
The following example configures a consistent-hash upstream using mqtt_client_id so connections from the same client continue to reach the same broker. It uses MQTT 5 traffic to verify the affinity.
When a client sends an empty ID, the gateway falls back to the client IP for consistent hashing.
- Admin API
- ADC
- Ingress Controller
For APISIX, create the stream route with the Admin API:
curl "http://127.0.0.1:9180/apisix/admin/stream_routes/mqtt-route" -X PUT \
-H "X-API-KEY: ${ADMIN_API_KEY}" \
-H "Content-Type: application/json" \
-d @- <<'EOF'
{
"server_port": 9100,
"plugins": {
"mqtt-proxy": {
"protocol_name": "MQTT",
"protocol_level": 5
}
},
"upstream": {
"type": "chash",
"key": "mqtt_client_id",
"nodes": {
"mqtt-broker-1.aic.svc.cluster.local:1883": 1,
"mqtt-broker-2.aic.svc.cluster.local:1883": 1
}
}
}
EOF
services:
- name: mqtt-service
labels:
docs-example: mqtt-proxy
upstream:
name: default
scheme: tcp
type: chash
key: mqtt_client_id
nodes:
- host: mqtt-broker-1.aic.svc.cluster.local
port: 1883
weight: 1
- host: mqtt-broker-2.aic.svc.cluster.local
port: 1883
weight: 1
stream_routes:
- name: mqtt-route
server_port: 9100
plugins:
mqtt-proxy:
protocol_name: MQTT
protocol_level: 5
Preview changes owned by this example:
adc diff -f adc.yaml \
--include-resource-type service \
--label-selector docs-example=mqtt-proxy
Synchronize the reviewed service configuration:
adc sync -f adc.yaml \
--include-resource-type service \
--label-selector docs-example=mqtt-proxy
- Gateway API
- APISIX CRD
Create the backend and route policies with client-ID consistent hashing and MQTT 5:
apiVersion: gateway.networking.k8s.io/v1
kind: TCPRoute
metadata:
namespace: aic
name: mqtt-route
spec:
parentRefs:
- name: apisix
sectionName: tcp
rules:
- backendRefs:
- name: mqtt-brokers
port: 1883
---
apiVersion: apisix.apache.org/v1alpha1
kind: BackendTrafficPolicy
metadata:
namespace: aic
name: mqtt-brokers
spec:
targetRefs:
- group: ""
kind: Service
name: mqtt-brokers
sectionName: mqtt
scheme: tcp
loadbalancer:
type: chash
hashOn: vars
key: mqtt_client_id
---
apiVersion: apisix.apache.org/v1alpha1
kind: L4RoutePolicy
metadata:
namespace: aic
name: mqtt-route
spec:
targetRefs:
- group: gateway.networking.k8s.io
kind: TCPRoute
name: mqtt-route
plugins:
- name: mqtt-proxy
config:
protocol_name: MQTT
protocol_level: 5
Create an upstream policy that hashes broker endpoints by mqtt_client_id and a stream route that accepts MQTT 5:
apiVersion: apisix.apache.org/v2
kind: ApisixUpstream
metadata:
namespace: aic
name: mqtt-brokers
spec:
ingressClassName: apisix
scheme: tcp
loadbalancer:
type: chash
hashOn: vars
key: mqtt_client_id
---
apiVersion: apisix.apache.org/v2
kind: ApisixRoute
metadata:
namespace: aic
name: mqtt-route
spec:
ingressClassName: apisix
stream:
- name: mqtt-route
protocol: TCP
match:
ingressPort: 9100
backend:
serviceName: mqtt-brokers
servicePort: 1883
plugins:
- name: mqtt-proxy
enable: true
config:
protocol_name: MQTT
protocol_level: 5
Apply the selected configuration:
kubectl apply -f mqtt-proxy-ic.yaml
- Docker
- Kubernetes
Subscribe directly to each broker in separate terminals:
docker exec mqtt-broker-1 \
mosquitto_sub -h 127.0.0.1 -p 1883 \
-V mqttv5 -t docs/balance
In the second terminal, subscribe to the other broker:
docker exec mqtt-broker-2 \
mosquitto_sub -h 127.0.0.1 -p 1883 \
-V mqttv5 -t docs/balance
Publish through the gateway with two client IDs:
docker exec mqtt-broker-1 \
mosquitto_pub -h "$GATEWAY_CONTAINER" -p 9100 \
-V mqttv5 -i publisher-1 \
-t docs/balance -m "from publisher-1"
docker exec mqtt-broker-1 \
mosquitto_pub -h "$GATEWAY_CONTAINER" -p 9100 \
-V mqttv5 -i publisher-2 \
-t docs/balance -m "from publisher-2"
The messages can reach the same or different brokers. Repeat either publish command to verify that the same client ID continues to reach the same broker while the upstream set remains unchanged.
Subscribe directly to the first broker:
kubectl exec -n aic mqtt-broker-0 -- \
mosquitto_sub -h 127.0.0.1 -p 1883 \
-V mqttv5 -t docs/balance
In a second terminal, subscribe to the other broker:
kubectl exec -n aic mqtt-broker-1 -- \
mosquitto_sub -h 127.0.0.1 -p 1883 \
-V mqttv5 -t docs/balance
Publish through the gateway with two client IDs:
kubectl exec -n aic mqtt-broker-0 -- \
mosquitto_pub -h "$GATEWAY_STREAM_HOST" -p 9100 \
-V mqttv5 -i publisher-1 \
-t docs/balance -m "from publisher-1"
kubectl exec -n aic mqtt-broker-0 -- \
mosquitto_pub -h "$GATEWAY_STREAM_HOST" -p 9100 \
-V mqttv5 -i publisher-2 \
-t docs/balance -m "from publisher-2"
The messages can reach the same or different brokers. Repeat either publish command to verify that the same client ID continues to reach the same broker while the upstream set remains unchanged.