gRPC over Kafka

The grpc-kafka binding lets a gRPC service be backed entirely by a Kafka topic. Each unary call is produced onto a Kafka topic, and the response is read back from a reply topic, so the gRPC client never talks to Kafka directly. This tutorial builds a gRPC echo service on top of a single Kafka topic.

Prerequisites

• Docker Compose

Set up the gRPC Kafka proxy

Create zilla.yaml, docker-compose.yaml, and echo.proto in the same directory.

zilla.yaml

name: gRPC-example
catalogs:
  host_filesystem:
    type: filesystem
    options:
      subjects:
        echo:
          path: proto/echo.proto
bindings:
  # Proxy service entrypoint
  north_tcp_server:
    type: tcp
    kind: server
    options:
      host: 0.0.0.0
      port: 7151
    exit: north_http_server
  north_http_server:
    type: http
    kind: server
    options:
      versions:
        - h2
      access-control:
        policy: cross-origin
    exit: north_grpc_server

  # gRPC service definition
  north_grpc_server:
    type: grpc
    kind: server
    catalog:
      host_filesystem:
        - subject: echo
    routes:
      - when:
          - method: example.EchoService/*
        exit: north_grpc_kafka_mapping

  # Proxy a gRPC service to a Kafka topic
  north_grpc_kafka_mapping:
    type: grpc-kafka
    kind: proxy
    routes:
      - when:
          - method: example.EchoService/*
        exit: north_kafka_cache_client
        with:
          capability: produce
          topic: echo-messages
          acks: leader_only
          reply-to: echo-messages
  north_kafka_cache_client:
    type: kafka
    kind: cache_client
    exit: south_kafka_cache_server
  south_kafka_cache_server:
    type: kafka
    kind: cache_server
    options:
      bootstrap:
        - echo-messages
    exit: south_kafka_client

  # Connect to Kafka
  south_kafka_client:
    type: kafka
    kind: client
    options:
      servers:
        - ${{env.KAFKA_BOOTSTRAP_SERVER}}
    exit: south_kafka_tcp_client
  south_kafka_tcp_client:
    type: tcp
    kind: client

telemetry:
  exporters:
    stdout_logs_exporter:
      type: stdout

docker-compose.yaml

version: '3'
services:

  zilla:
    image: ghcr.io/aklivity/zilla:latest
    pull_policy: always
    depends_on:
      - kafka
    ports:
      - 7151:7151
    environment:
      KAFKA_BOOTSTRAP_SERVER: "kafka:29092"
    volumes:
      - ./zilla.yaml:/etc/zilla/zilla.yaml
      - ./echo.proto:/etc/zilla/proto/echo.proto
    command: start -v -e

  kafka:
    image: bitnami/kafka:3.5
    hostname: kafka
    ports:
      - 9092:9092
      - 29092:9092
    environment:
      ALLOW_PLAINTEXT_LISTENER: "yes"
      KAFKA_CFG_NODE_ID: "1"
      KAFKA_CFG_BROKER_ID: "1"
      KAFKA_CFG_CONTROLLER_QUORUM_VOTERS: "1@127.0.0.1:9093"
      KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP: "CLIENT:PLAINTEXT,INTERNAL:PLAINTEXT,CONTROLLER:PLAINTEXT"
      KAFKA_CFG_CONTROLLER_LISTENER_NAMES: "CONTROLLER"
      KAFKA_CFG_LOG_DIRS: "/tmp/logs"
      KAFKA_CFG_PROCESS_ROLES: "broker,controller"
      KAFKA_CFG_LISTENERS: "CLIENT://:9092,INTERNAL://:29092,CONTROLLER://:9093"
      KAFKA_CFG_INTER_BROKER_LISTENER_NAME: "INTERNAL"
      KAFKA_CFG_ADVERTISED_LISTENERS: "CLIENT://localhost:9092,INTERNAL://kafka:29092"
      KAFKA_CFG_AUTO_CREATE_TOPICS_ENABLE: "true"

  kafka-init:
    image: bitnami/kafka:3.5
    command:
      - "/bin/bash"
      - "-c"
      -  |
        /opt/bitnami/kafka/bin/kafka-topics.sh --bootstrap-server kafka:29092 --create --if-not-exists --topic echo-messages
    depends_on:
      - kafka
    init: true

echo.proto

syntax = "proto3";

package example;

service EchoService
{
  rpc EchoSimple(EchoMessage) returns (EchoMessage);
}

message EchoMessage
{
  string message = 1;
}

The north_http_server binding negotiates HTTP/2 (h2), which gRPC requires. The north_grpc_server binding loads echo.proto from a filesystem catalog and routes any example.EchoService method to north_grpc_kafka_mapping. That binding produces each call onto the echo-messages topic and reads the reply back from the same topic using reply-to.

Run Zilla and Kafka

docker-compose up --detach

Call the echo service

Use grpcurl to call EchoSimple:

docker run -v ./echo.proto:/proto/echo.proto -it --rm fullstorydev/grpcurl \
-plaintext -proto proto/echo.proto -d '{"message":"Hello World"}' host.docker.internal:7151 example.EchoService.EchoSimple

The call is produced to the echo-messages topic and the same message is read back as the gRPC response.

Remove the running containers

docker-compose down

Next Steps

• Walk through the gRPC to Kafka use case for more on RPC-over-Kafka patterns.
• See the grpc-kafka binding reference for every option.
• Try out more gRPC examples in the Zilla examples directory: grpc.echo, grpc.kafka.echo, grpc.kafka.fanout, grpc.kafka.proxy, and grpc.proxy.