Troubleshooting Guides

These guides cover common issues encountered while operating a Kafka Gateway deployment. Each entry lists the Symptom you'll observe, the likely Cause, and the Fix to apply.

Quick reference

Match what you're seeing to jump straight to the fix. See Log Events for the full event reference.

Symptom	Guide
`BINDING_KAFKA_SASL_AUTHENTICATION_FAILED`, `BINDING_KAFKA_UNSUPPORTED_SASL_MECHANISM`	Why does the Kafka client fail to authenticate to the broker?
`BINDING_KAFKA_API_VERSION_REJECTED`, `BINDING_KAFKA_*_AUTHORIZATION_FAILED`	Why does a request get an API version, topic, group, or cluster authorization error?
`BINDING_KAFKA_BROKER_CONNECTION_FAILED`	Why does the connection to a Kafka broker keep failing or dropping?
Kafka client can't connect via Zilla Plus	Why does my Kafka client fail to connect via Zilla Plus?
`CATALOG__UNRETRIEVABLE_`	Why is a schema lookup from the catalog failing?
`MODEL_*_VALIDATION_FAILED`	Why does a message get rejected as invalid?
`BINDING_MQTT_KAFKA_NON_COMPACT_SESSIONS_TOPIC`	Why does the mqtt-to-kafka sessions topic get rejected?

Authentication & authorization

Why does the Kafka client fail to authenticate to the broker?

Symptom: Zilla raises BINDING_KAFKA_SASL_AUTHENTICATION_FAILED or BINDING_KAFKA_UNSUPPORTED_SASL_MECHANISM, logged with the client identity and the broker's error.

Cause: the kafka client binding authenticates to the broker with the mechanism and credentials configured under options.authorization (or the deprecated options.sasl). ..._AUTHENTICATION_FAILED fires when the broker's SASL handshake response comes back with an error, or, for SCRAM, the server's nonce doesn't validate. ..._UNSUPPORTED_SASL_MECHANISM fires instead when the broker doesn't recognize the configured mechanism at all, rather than rejecting the credentials.

Fix: confirm the username/password (PLAIN, SCRAM) or the token sourced from a guard (OAuthBearer) match what the broker expects, and that the configured mechanism matches one the broker's listener actually has enabled and advertises. See SASL for the mechanisms Kafka Gateway supports: PLAIN, SCRAM, and OAuthBearer.

Why does a request get an API version, topic, group, or cluster authorization error?

Symptom: one of the following events, based on the broker's response error code:

Event	Meaning
BINDING_KAFKA_API_VERSION_REJECTED	The broker doesn't support the API version being requested.
BINDING_KAFKA_TOPIC_AUTHORIZATION_FAILED	The authenticated principal lacks ACL permission on the topic.
BINDING_KAFKA_GROUP_AUTHORIZATION_FAILED	The authenticated principal lacks ACL permission on the consumer group.
BINDING_KAFKA_CLUSTER_AUTHORIZATION_FAILED	The authenticated principal lacks cluster-level permission for the requested operation.
BINDING_KAFKA_TRANSACTIONAL_ID_AUTHORIZATION_FAILED	The authenticated principal lacks ACL permission on the transactional id.

Cause: once a client authenticates, the broker still checks the resulting principal's ACLs against every request, and separately checks whether it supports the API version being negotiated. Zilla surfaces each outcome as its own event.

Fix: check the ACLs granted to the authenticated principal, whether that's the SASL identity or the mTLS certificate's principal, against the specific topic, group, or transactional id named in the failing request. See SASL and mTLS for how Kafka Gateway establishes that principal.

Connectivity

Why does the connection to a Kafka broker keep failing or dropping?

Symptom: Zilla raises BINDING_KAFKA_BROKER_CONNECTION_FAILED, logged with the broker host and port.

Cause: the kafka client binding maintains a pool of connections to the brokers in the cluster, each reached over tcp, and usually TLS or mTLS. This event fires when a broker connection is aborted or reset after it was already established and assigned to serve requests.

Fix: this usually points at the network path or the broker itself rather than at Zilla's own configuration. Confirm the broker at the logged host and port is up and reachable, that nothing between Zilla and the broker (a firewall or load balancer) is closing idle connections, and, if TLS is in the path, that the broker's certificate is still valid and trusted. Resets that happen right after the TLS handshake usually point at a trust or certificate mismatch in the tls client binding rather than a broker outage.

Why does my Kafka client fail to connect via Zilla Plus?

Symptom: the Kafka client can't connect via Zilla Plus.

Cause: the DNS names aren't set up correctly to point to the NLB Load Balancer deployed via the provided CloudFormation template, or the client.properties file isn't configured with the correct keystore or truststore.

Fix: confirm your DNS and client.properties are configured correctly, then work through the following checks in order.

1. Verify TCP connectivity

nc -v <b-1-broker-dns-name> 9094

The nc output should be as shown below, and nc should stay connected:

Connection to <b-1-broker-dns-name> port 9094 [tcp/*] succeeded!

2. Verify TLS connectivity

Verify TLS connectivity with your client key and signed client certificate:

openssl s_client \
-connect <b-1-broker-dns-name>:9094 \
-servername <b-1-broker-dns-name> \
-cert client.cert \
-key client.key.pem

Note: if you followed Create Server Certificate to create the server certificate instead of Create Server Certificate (LetsEncrypt), then you will need to export the CA certificate and have openssl trust the exported CA certificate:

openssl s_client \
-connect <b-1-broker-dns-name>:9094 \
-servername <b-1-broker-dns-name> \
-cert client.cert \
-key client.key.pem
-CAfile Certificate.pem

The openssl output should be as shown below:

...
    Verify return code: 0 (ok)
---

Info

A successful handshake means openssl shows no error and stays connected — the client-to-proxy leg is fine. If instead the backend TLS handshake from the Zilla Plus proxy to MSK fails, openssl disconnects right after the handshake with no openssl error code — that points at the proxy-to-MSK leg, not your client.

3. Verify Kafka connectivity

kcat \
-L \
-b <b-1-broker-dns-name>:9094 \
-X security.protocol=ssl \
-X ssl.certificate.location=client.cert \
-X ssl.key.location=client.key.pem

kcat \
-L \
-b <b-1-broker-dns-name>:9094 \
-X security.protocol=ssl \
-X ssl.certificate.location=client.cert \
-X ssl.key.location=client.key.pem \
-X ssl.ca.location=Certificate.pem

The kcat output should show the list of brokers and topics accessible to the client.

Schema & data validation

Why is a schema lookup from the catalog failing?

Symptom: an UNRETRIEVABLE event named for the catalog type in use, for example CATALOG_APICURIO_REGISTRY_UNRETRIEVABLE_ARTIFACT_ID for Apicurio Registry, or CATALOG_SCHEMA_REGISTRY_UNRETRIEVABLE_SCHEMA_ID for a generic Schema Registry-compatible catalog. The full event reference lists the equivalent events for Karapace and for Filesystem catalogs.

Cause: a schema-backed model (avro, json, protobuf) resolves its schema at runtime from a configured catalog. This event fires when Zilla can't fetch the schema, by ID, subject, or subject and version.

Info

A _SUBJECT_VERSION_STALE_* variant is less urgent: the fetch failed, but a previously cached schema is still being served, so validation continues against stale data instead of failing outright.

Fix: check that the catalog's configured url (remote catalogs) or path (filesystem) is reachable and correct, that the subject, topic, or ID referenced by the model actually exists in the registry, and, for remote catalogs, that any required credentials are still valid. See Catalog for every catalog type Kafka Gateway supports.

Why does a message get rejected as invalid?

Symptom: one of the following events:

Event	What it means
MODEL_AVRO_VALIDATION_FAILED	The payload doesn't decode against the Avro schema resolved from the model's catalog.
MODEL_JSON_VALIDATION_FAILED	The payload doesn't validate against the JSON schema resolved from the model's catalog.
MODEL_PROTOBUF_VALIDATION_FAILED	The payload doesn't decode against the Protobuf schema resolved from the model's catalog.
MODEL_CORE_VALIDATION_FAILED	The payload doesn't satisfy a primitive model, such as a string that isn't valid UTF-8 or a value that doesn't parse as the declared numeric type.

Cause: a kafka cache_client or cache_server with a model configured on a topic's key or value validates every message against that model. A message that doesn't conform raises one of the events above.

Fix: check that the producing client is actually serializing with the schema or type declared on the topic's model, and, for schema-backed models, that the catalog is resolving the schema version the producer really used rather than an older or newer one. If a view translates between formats, confirm the payload matches the format on the side closest to the client, not the one stored on the topic. See Model for the full set of primitive and schema-backed model types.

MQTT

Why does the mqtt-to-kafka sessions topic get rejected?

Symptom: the MQTT connection is reset and Zilla raises BINDING_MQTT_KAFKA_NON_COMPACT_SESSIONS_TOPIC, logged with the offending topic name.

Cause: the mqtt-kafka proxy stores MQTT session state on a dedicated Kafka topic, declared under options.topics.sessions, and that topic is required to have a compact cleanup.policy so only the latest state for each session key is kept. This event fires when the configured sessions topic isn't log compacted.

Fix: check the cleanup.policy on the sessions topic named in the event and set it to compact. See MQTT to Kafka for the required topic mapping, and MQTT over Kafka for a working example that provisions the sessions and retained topics with compaction enabled.

Log Events

Named events raised by the kafka and mqtt-kafka bindings, along with the catalog and model extensions, show up in the log format as <event name> - <details>. For events raised by the surrounding tcp, tls, http, and guard bindings in a Kafka Gateway pipeline, see the full event reference.

The kafka binding raises the following events:

Event	What to check
BINDING_KAFKA_SASL_AUTHENTICATION_FAILED	The credentials or token sourced from `options.authorization` don't match what the broker expects for the configured SASL mechanism.
BINDING_KAFKA_UNSUPPORTED_SASL_MECHANISM	The configured `mechanism` isn't one the broker's listener advertises.
BINDING_KAFKA_API_VERSION_REJECTED	The broker doesn't support the Kafka protocol API version being negotiated.
BINDING_KAFKA_TOPIC_AUTHORIZATION_FAILED	The authenticated principal's ACLs don't grant access to the topic in the request.
BINDING_KAFKA_GROUP_AUTHORIZATION_FAILED	The authenticated principal's ACLs don't grant access to the consumer group in the request.
BINDING_KAFKA_CLUSTER_AUTHORIZATION_FAILED	The authenticated principal lacks cluster-level permission for the requested operation.
BINDING_KAFKA_TRANSACTIONAL_ID_AUTHORIZATION_FAILED	The authenticated principal's ACLs don't grant access to the transactional id in the request.
BINDING_KAFKA_PRODUCE_ERROR	The broker rejected a produce request, for example an oversized or malformed record batch; check the logged error code and topic.
BINDING_KAFKA_OFFSET_COMMIT_FAILED	An offset commit was rejected by the broker, for example with oversized commit metadata.
BINDING_KAFKA_BROKER_CONNECTION_FAILED	The connection to the broker at the logged host and port was aborted or reset after being established; check broker health and any intermediate network path.
BINDING_KAFKA_AUTHORIZATION_FAILED	A generic authorization failure identified by client identity.

The mqtt-kafka binding raises one event of its own:

Event	What to check
BINDING_MQTT_KAFKA_NON_COMPACT_SESSIONS_TOPIC	The topic configured under `options.topics.sessions` doesn't have `cleanup.policy=compact` set.

Each configured catalog raises a matching pair of RETRIEVED/UNRETRIEVABLE events per lookup kind (by ID, or by subject and version), named for the catalog type: CATALOG_APICURIO_REGISTRY_*, CATALOG_CONFLUENT_SCHEMA_REGISTRY_* (Zilla Plus), CATALOG_KARAPACE_SCHEMA_REGISTRY_*, and the generic CATALOG_SCHEMA_REGISTRY_*. An UNRETRIEVABLE event means the schema fetch failed; check the catalog's url or path, and confirm the referenced subject, topic, or ID exists. A _STALE_* variant means Zilla is still serving a previously cached schema despite the failed refetch. The filesystem catalog instead raises a single CATALOG_FILESYSTEM_FILE_NOT_FOUND when the configured schema file can't be read. See the full event reference for every catalog event name.

Each schema-backed or primitive model raises one validation event when a message doesn't conform: MODEL_AVRO_VALIDATION_FAILED, MODEL_JSON_VALIDATION_FAILED, MODEL_PROTOBUF_VALIDATION_FAILED, and MODEL_CORE_VALIDATION_FAILED for the primitive types. Check the payload against the schema or type declared on the topic's model, and against the exact schema version the catalog resolved.