MQTT to Kafka

Use CaseMQTTKafka Gateway

The Problem

IoT devices and pub/sub clients speak MQTT, but Kafka is the durable event log the rest of the platform is built on. Bridging the two usually means running and operating a separate MQTT broker, then writing a connector to move messages, session state, and retained messages into Kafka, while reconciling MQTT semantics like QoS and retained messages that Kafka doesn't have natively.

How Zilla Solves It

Zilla's mqtt binding terminates the MQTT protocol directly, decoding publish, subscribe, and session traffic into application streams. The mqtt-kafka binding then proxies those streams onto Kafka topics: session state, retained messages, and published messages each land on their own compacted topic. There is no MQTT broker to run separately; MQTT connections terminate straight onto Kafka.

Architecture

MQTT over WebSocket is supported the same way: a ws binding upgrades the HTTP connection and routes it to the same mqtt · server, so browser and native MQTT clients share the same Kafka-backed broker.

Topic Mapping

The mqtt-kafka proxy requires three Kafka topics, declared under options.topics:

# MQTT messages to Kafka topics
north_mqtt_kafka_mapping:
  type: mqtt-kafka
  kind: proxy
  options:
    topics:
      sessions: mqtt-sessions
      messages: mqtt-messages
      retained: mqtt-retained
    clients:
      - place/{identity}/#
  exit: north_kafka_cache_client

sessions stores MQTT session state and requires a compact cleanup.policy.
messages is the default topic for published and subscribed messages.
retained stores retained messages and should also use a compact cleanup.policy.

options.clients extracts a client identity from the topic filter (for example place/{identity}/#), so messages from the same client land on the same Kafka partition.

Routing by Topic Pattern

Additional routes match specific MQTT topic filters and redirect them to a different Kafka topic. Here, any publish or subscribe under device/# or place/+/device/# is routed to the mqtt-devices topic instead of the default mqtt-messages topic:

routes:
  - when:
      - publish:
          - topic: place/+/device/#
          - topic: device/#
      - subscribe:
          - topic: place/+/device/#
          - topic: device/#
    with:
      messages: mqtt-devices
    exit: north_kafka_cache_client

This lets a single MQTT server fan traffic out to multiple Kafka topics based on topic pattern, without any application-level routing logic.

Get Started

zilla.yaml

name: zilla-mqtt-kafka-broker
bindings:
  #region entrypoint
  # Proxy service entrypoint
  north_tcp_server:
    type: tcp
    kind: server
    options:
      host: 0.0.0.0
      port:
        - 7114
        - 7183
    routes:
      - when:
          - port: 7114
        exit: north_http_server
      - when:
          - port: 7183
        exit: north_mqtt_server
  #endregion entrypoint

  #region server
  # WebSocket server
  north_http_server:
    type: http
    kind: server
    routes:
      - when:
          - headers:
              :scheme: http
              :authority: localhost:7114
              upgrade: websocket
        exit: north_ws_server
  north_ws_server:
    type: ws
    kind: server
    routes:
      - when:
          - protocol: mqtt
        exit: north_mqtt_server

  # Shared MQTT server
  north_mqtt_server:
    type: mqtt
    kind: server
    exit: north_mqtt_kafka_mapping
  #endregion server

  #region kafka_mapping
  # MQTT messages to Kafka topics
  north_mqtt_kafka_mapping:
    type: mqtt-kafka
    kind: proxy
    options:
      topics:
        sessions: mqtt-sessions
        messages: mqtt-messages
        retained: mqtt-retained
      clients:
        - place/{identity}/#
    exit: north_kafka_cache_client
    #endregion kafka_mapping
    #region device_mapping
    routes:
      - when:
          - publish:
              - topic: place/+/device/#
              - topic: device/#
          - subscribe:
              - topic: place/+/device/#
              - topic: device/#
        with:
          messages: mqtt-devices
        exit: north_kafka_cache_client
    #endregion device_mapping

  #region kafka_sync
  # Kafka sync layer
  north_kafka_cache_client:
    type: kafka
    kind: cache_client
    exit: south_kafka_cache_server
  south_kafka_cache_server:
    type: kafka
    kind: cache_server
    options:
      bootstrap:
        - mqtt-messages
        - mqtt-retained
        - mqtt-devices
    exit: south_kafka_client
    #endregion kafka_sync

  #region kafka_client
  # Connect to Kafka
  south_kafka_client:
    type: kafka
    kind: client
    options:
      servers:
        - ${{env.KAFKA_BOOTSTRAP_SERVER}}
    exit: south_kafka_tcp_client
  south_kafka_tcp_client:
    type: tcp
    kind: client
    #endregion kafka_client

telemetry:
  exporters:
    stdout_logs_exporter:
      type: stdout

Try the example

The mqtt.kafka.proxy example in the Zilla repository runs a working MQTT-to-Kafka broker with a local Kafka broker, all from a single docker compose up. For a guided, step-by-step build of this same configuration, see Running an MQTT Kafka broker.