gRPC to Kafka

Use CasegRPCKafka Gateway

The Problem

gRPC services often need to hand work off to Kafka, either to decouple the caller from a slower downstream process, to replay and audit requests, or to fan a stream out to multiple consumers. Wiring a gRPC service directly to a Kafka client means writing custom code to serialize the request, correlate it with a response, and handle reconnects for streaming calls.

How Zilla Solves It

Zilla's grpc binding terminates gRPC unary and streaming calls, matching routes by fully-qualified service method (with wildcard support). The grpc-kafka binding then maps each matched route onto Kafka: request messages are produced to a topic with a zilla:correlation-id header, and the correlated response is read back from a reply-to topic. The proto service definition itself can be loaded from a catalog, so Zilla knows the service shape without any generated server code.

Architecture

Declaring the Service

The grpc · server binding loads the proto file from a catalog and routes matched methods to the grpc-kafka proxy:

north_grpc_kafka_mapping:
  type: grpc-kafka
  kind: proxy
  routes:
    - when:
        - method: example.EchoService/*
      exit: north_kafka_cache_client
      with:
        capability: produce
        topic: echo-messages
        acks: leader_only
        reply-to: echo-messages

Mapping RPC Shapes to Kafka

The same grpc-kafka produce capability handles every RPC shape gRPC supports, because Kafka topics are just ordered streams of correlated messages:

Simple RPC: a single request message and a single response message, correlated by zilla:correlation-id.
Server-side streaming: the server produces a stream of response messages to the reply-to topic; Kafka holds them all without regard to whether the client is still connected.
Client-side streaming: the client produces a stream of request messages to the request topic before a single correlated response comes back.
Bidirectional streaming: request and response messages can share a single topic when both sides need the same message history.

Requests sent with an idempotency-key metadata header can be safely replayed by the client; the downstream consumer detects and ignores duplicates carrying the same key and correlation ID.

Fanout with the Fetch Capability

For streaming a Kafka topic out to gRPC clients (the reverse direction from produce), a route can use capability: fetch instead, with optional key and header filters:

routes:
  - when:
      - method: example.FanoutService/*
    exit: kafka_cache_client
    with:
      capability: fetch
      topic: messages
      filters:
        key: custom-key
        headers:
          custom-text: custom-value

Calling Back into gRPC

To go the other direction, replaying Kafka request messages back into a live gRPC service, the kafka-grpc remote_server binding reads from the request topic, invokes the corresponding gRPC method, and writes the response back to the reply-to topic.

Get Started

zilla.yaml

name: gRPC-example
catalogs:
  host_filesystem:
    type: filesystem
    options:
      subjects:
        echo:
          path: proto/echo.proto
bindings:
  # Proxy service entrypoint
  north_tcp_server:
    type: tcp
    kind: server
    options:
      host: 0.0.0.0
      port: 7151
    exit: north_http_server
  north_http_server:
    type: http
    kind: server
    options:
      versions:
        - h2
      access-control:
        policy: cross-origin
    exit: north_grpc_server

  # gRPC service definition
  north_grpc_server:
    type: grpc
    kind: server
    catalog:
      host_filesystem:
        - subject: echo
    routes:
      - when:
          - method: example.EchoService/*
        exit: north_grpc_kafka_mapping

  # Proxy a gRPC service to a Kafka topic
  #region grpc_kafka_mapping
  north_grpc_kafka_mapping:
    type: grpc-kafka
    kind: proxy
    routes:
      - when:
          - method: example.EchoService/*
        exit: north_kafka_cache_client
        with:
          capability: produce
          topic: echo-messages
          acks: leader_only
          reply-to: echo-messages
  #endregion grpc_kafka_mapping
  north_kafka_cache_client:
    type: kafka
    kind: cache_client
    exit: south_kafka_cache_server
  south_kafka_cache_server:
    type: kafka
    kind: cache_server
    options:
      bootstrap:
        - echo-messages
    exit: south_kafka_client

  # Connect to Kafka
  south_kafka_client:
    type: kafka
    kind: client
    options:
      servers:
        - ${{env.KAFKA_BOOTSTRAP_SERVER}}
    exit: south_kafka_tcp_client
  south_kafka_tcp_client:
    type: tcp
    kind: client

telemetry:
  exporters:
    stdout_logs_exporter:
      type: stdout

echo.proto

syntax = "proto3";

package example;

service EchoService
{
  rpc EchoSimple(EchoMessage) returns (EchoMessage);
}

message EchoMessage
{
  string message = 1;
}

Try the example

The grpc.kafka.echo example in the Zilla repository runs this Echo service against a local Kafka broker with a single docker compose up. For a full walkthrough with a multi-method service (simple, server-streaming, client-streaming, and bidirectional RPCs), see the RouteGuide microservice on Kafka guide, or grpc.kafka.proxy and grpc.kafka.fanout for more examples.