SSE from Kafka

Use CaseSSEKafka Gateway

The Problem

Browsers need a simple way to receive a live feed of Kafka messages, but Kafka consumers aren't designed to be exposed directly to untrusted clients over the internet, and every browser connection needs its own tracked offset, reconnect handling, and fanout from a shared topic.

How Zilla Solves It

Zilla's sse-kafka binding adapts a Kafka topic into a Server-Sent Events stream. Each browser connection becomes an independent SSE stream reading from the same Kafka topic, with progress tracked through the SSE id field so a reconnecting client resumes exactly where it left off, no separate fanout service or WebSocket server required.

Architecture

Reliable Delivery and Reconnects

Each SSE event's id field encodes the Kafka partition offset (as an etag) or, when configured, the message key and etag together. When a client reconnects, its last-event-id header carries the last id it saw, and Zilla resumes the stream from that position automatically, no client-side bookkeeping required.

north_sse_kafka_mapping:
  type: sse-kafka
  kind: proxy
  routes:
    - when:
        - path: /events
      with:
        topic: events
      exit: north_kafka_cache_client

A Kafka tombstone (a null-value message) is delivered to the client as a delete event, identifying the deleted key from the event id. This makes sse-kafka a natural pairing with the http-kafka binding, which uses the same etag value for conditional if-match requests.

Get Started

zilla.yaml

name: SSE-example
bindings:
  # Proxy service entrypoint
  north_tcp_server:
    type: tcp
    kind: server
    options:
      host: 0.0.0.0
      port: 7114
    exit: north_http_server
  north_http_server:
    type: http
    kind: server
    options:
      access-control:
        policy: cross-origin
    routes:
      - when:
          - headers:
              :scheme: http
              :authority: localhost:7114
              :path: /events
        exit: north_sse_server
      - when:
          - headers:
              :scheme: http
              :authority: localhost:7114
        exit: east_http_filesystem_mapping

  # UI html file server
  east_http_filesystem_mapping:
    type: http-filesystem
    kind: proxy
    routes:
      - when:
          - path: /{path}
        with:
          path: ${params.path}
        exit: east_filesystem_server
  east_filesystem_server:
    type: filesystem
    kind: server
    options:
      location: /var/www/

  # SSE Server With an exit to Kafka
  north_sse_server:
    type: sse
    kind: server
    exit: north_sse_kafka_mapping
  #region sse_kafka_mapping
  north_sse_kafka_mapping:
    type: sse-kafka
    kind: proxy
    routes:
      - when:
          - path: /events
        with:
          topic: events
        exit: north_kafka_cache_client
  #endregion sse_kafka_mapping

  # Kafka sync layer
  north_kafka_cache_client:
    type: kafka
    kind: cache_client
    exit: south_kafka_cache_server
  south_kafka_cache_server:
    type: kafka
    kind: cache_server
    options:
      bootstrap:
        - events
    exit: south_kafka_client

  # Connect to Kafka
  south_kafka_client:
    type: kafka
    kind: client
    options:
      servers:
        - ${{env.KAFKA_BOOTSTRAP_SERVER}}
    exit: south_kafka_tcp_client
  south_kafka_tcp_client:
    type: tcp
    kind: client

telemetry:
  exporters:
    stdout_logs_exporter:
      type: stdout

Try the example

The sse.kafka.fanout example in the Zilla repository runs a working browser fanout demo, complete with an index.html test page, from a single docker compose up. See also sse.proxy.jwt for adding JWT auth to an SSE stream.