HTTP to Kafka

Use CaseHTTPKafka Gateway

The Problem

Client apps and services need a REST API in front of Kafka: create a resource with a POST, read it back with a GET, update it with a PUT, delete it with a DELETE. Building that normally means writing and operating a service that owns a Kafka producer and consumer, translates HTTP semantics into Kafka reads and writes, and handles correlation, idempotency, and response merging by hand.

How Zilla Solves It

Zilla's http-kafka binding maps HTTP request-response semantics directly onto Kafka produce and fetch operations. Each route matches an HTTP method and path, and declares whether it produces a message to a topic or fetches messages from one. No proxy service, Kafka client library, or custom code required.

Architecture

Binding	Role
`http · server`	Terminates the inbound HTTP connection
`http-kafka · proxy`	Maps HTTP routes to Kafka produce and fetch operations
`kafka · cache_client` / `kafka · cache_server`	Keep a local, continuously updated cache of the mapped topics
`kafka · client`	Connects to the Kafka broker

Produce: Create and Update

A route with capability: produce maps an HTTP request to a Kafka message. Sending an Idempotency-Key header on a POST sets the Kafka message key, so retried requests are safe to replay. A PUT to /items/{id} produces a new message under the same key to update it, and a DELETE produces a tombstone (a blank message) for the key to remove it.

- when:
    - method: POST
      path: /items
  exit: north_kafka_cache_client
  with:
    capability: produce
    topic: items-snapshots
    key: ${idempotencyKey}

Fetch: Read One and Read Many

A route with capability: fetch maps an HTTP GET to a Kafka log-compacted topic. Fetching /items/{id} filters by message key, and fetching /items merges every distinct key on the topic into a single JSON response.

- when:
    - method: GET
      path: /items
  exit: north_kafka_cache_client
  with:
    capability: fetch
    topic: items-snapshots
    merge:
      content-type: application/json

Status 200 responses include an etag header for conditional if-none-match requests. Clients can also send prefer: wait=N to long-poll for up to N seconds instead of repeatedly polling; Zilla responds immediately once a matching message arrives.

Correlated Request-Response

For request-response style APIs where a downstream consumer processes the request and produces a reply, a produce route can set reply-to and async.location so a client can either wait synchronously or come back later with prefer: respond-async to poll for the result. See with.capability: produce for the full option set.

Get Started

zilla.yaml

name: example
bindings:
  north_tcp_server:
    type: tcp
    kind: server
    options:
      host: 0.0.0.0
      port:
        - 7114
    routes:
      - when:
          - port: 7114
        exit: north_http_server
  north_http_server:
    type: http
    kind: server
    routes:
      - when:
          - headers:
              :scheme: http
        exit: north_http_kafka_mapping
  north_http_kafka_mapping:
    type: http-kafka
    kind: proxy
    routes:
      #region rest_create
      - when:
          - method: POST
            path: /items
        exit: north_kafka_cache_client
        with:
          capability: produce
          topic: items-snapshots
          key: ${idempotencyKey}
      #endregion rest_create
      #region rest_update
      - when:
          - method: PUT
            path: /items/{id}
        exit: north_kafka_cache_client
        with:
          capability: produce
          topic: items-snapshots
          key: ${params.id}
      #endregion rest_update
      #region rest_delete
      - when:
          - method: DELETE
            path: /items/{id}
        exit: north_kafka_cache_client
        with:
          capability: produce
          topic: items-snapshots
          key: ${params.id}
      #endregion rest_delete
      #region rest_retrieve_all
      - when:
          - method: GET
            path: /items
        exit: north_kafka_cache_client
        with:
          capability: fetch
          topic: items-snapshots
          merge:
            content-type: application/json
      #endregion rest_retrieve_all
      #region rest_retrieve_id
      - when:
          - method: GET
            path: /items/{id}
        exit: north_kafka_cache_client
        with:
          capability: fetch
          topic: items-snapshots
          filters:
            - key: ${params.id}
      #endregion rest_retrieve_id
  north_kafka_cache_client:
    type: kafka
    kind: cache_client
    exit: south_kafka_cache_server
  south_kafka_cache_server:
    type: kafka
    kind: cache_server
    options:
      bootstrap:
        - items-snapshots
    exit: south_kafka_client
  south_kafka_client:
    type: kafka
    kind: client
    options:
      servers:
        - ${{env.KAFKA_BOOTSTRAP_SERVER}}
    exit: south_tcp_client
  south_tcp_client:
    type: tcp
    kind: client
telemetry:
  exporters:
    stdout_logs_exporter:
      type: stdout

Try the example

The http.kafka.crud example in the Zilla repository runs this full CRUD API against a local Kafka broker with a single docker compose up. For a from-scratch walkthrough of the same API, see the REST over Kafka tutorial.

Other ready-to-run HTTP-to-Kafka examples in the Zilla repository: http.kafka.async, http.kafka.cache, http.kafka.oneway, and http.kafka.sync.