Skip to main content

Virtual Kafka Clusters

Use CaseMulti-tenantZilla Plus
★Zilla Plus This use case is implemented by Zilla Plus, Aklivity's commercial edition of Kafka Gateway.

The Problem

Multiple teams or tenants sharing one Kafka cluster need isolation: their own topic namespace, consumer groups, access control lists, and quotas, without provisioning and operating a separate physical cluster per tenant.

How Zilla Solves It

Zilla Plus's kafka-proxy binding maps a virtual cluster onto a shared physical Kafka cluster entirely through configuration, with no changes to the underlying cluster. Clients connect to a cluster-specific external bootstrap hostname; Zilla Plus proxies the Kafka protocol through to the shared internal brokers while presenting per-cluster topic names, consumer group names, ACLs, and quotas. Because the proxy carries no per-connection state, adding another virtual cluster is a configuration change, not a new deployment.

Virtual Cluster Overview
Virtual Cluster Overview

Architecture

Key Capabilities

  • Multiple virtual clusters over one physical cluster: each virtual cluster is defined by its own kafka-proxy binding instance and cluster-id.
  • Stateless architecture: no custom metadata store to manage between proxy instances.
  • Virtual topics, consumer groups, and ACLs: mapped one-to-one to system topics, consumer groups, and ACLs on the shared cluster.
  • Client quotas: applied per virtual cluster, scoped by the connecting client ID.
  • Backwards compatible: without a cluster-id, the proxy behaves as a plain pass-through, so virtual clusters are opt-in.

Configure a Virtual Cluster

A kafka-proxy binding maps an external, per-cluster hostname pattern to the shared internal broker endpoints:

tenant_a_kafka_proxy:
  type: kafka-proxy
  kind: proxy
  options:
    cluster-id: tenant-a
    external:
      host: kafka-#.tenant-a.example.net
      port: 9093
    internal:
      host: b-#.kafka.internal.net
      port: 9094
  exit: tls_client

The # placeholder in external.host and internal.host encodes the broker number, so each virtual cluster can expose its own externally addressable broker list while all of them route to the same internal brokers. cluster-id scopes the virtual namespace used for that binding's topics, consumer groups, ACLs, and quotas. A second tenant is configured as a second kafka-proxy binding with its own cluster-id and external hostname pattern, reached through ordinary Zilla routing (for example, by matching the incoming TLS server name or port at the entrypoint).

Deploy

Zilla Plus deploys the kafka-proxy binding as an auto-scaling, stateless proxy fleet behind a network load balancer. See Deploy Zilla Plus on AWS ECS Fargate or Deploying Zilla Plus on AWS EKS for deployment steps, and the Secure Public Access and Secure Private Access use cases for combining virtual clusters with edge authentication.