Managed Kafka in KRaft mode, and the small-cluster trap

Kafka on Simplifyd runs in KRaft mode. There is no ZooKeeper ensemble to size, patch or wake up for at night — the controllers hold the metadata quorum themselves. That much is just following where upstream Kafka went.

The interesting part is what happens when someone provisions the smallest possible cluster.

Why a one-broker cluster hangs

Kafka creates internal topics for its own bookkeeping, the best known being __consumer_offsets. By default those topics ask for a replication factor of 3. On a healthy production cluster that is exactly what you want.

On a single-broker development cluster it is unsatisfiable. Kafka cannot place three replicas on one broker, so the topic is never created, so the cluster never reports ready. Nothing in the logs says "your replication factor is impossible" — it simply waits.

Anyone who has stood up a scratch Kafka to test a consumer has hit this and assumed the operator was broken.

What we do instead

We clamp internal topic replication to the actual broker count at provision time. A one-broker cluster gets a replication factor of one and comes up in under a minute. A three-broker cluster gets three. You do not configure it and you cannot get it wrong.

Two topologies

• Standalone puts the broker and controller roles on a single node. It is for development, and we are direct about that — one node is one failure domain.
• Cluster splits brokers and controllers into separate node pools that scale independently, which is what you want when the thing on the other end is a payment stream.

Storage scales with brokers. Controllers hold metadata, not your messages, so they take a fixed small volume rather than whatever you sized your brokers for — you are not billed for broker-sized disks on nodes that will never fill them.

What we did not ship

There is no public endpoint. Kafka clients reconnect to whatever address the cluster advertises, which means genuine external access needs a stable, unique public address for every broker. The shared-IP approach that works for a normal HTTP service cannot express that, and shipping something that half-works over a protocol this stateful would be worse than shipping nothing.

Clusters are reachable from services in the same environment. If you need to reach one from outside, talk to us about what you are trying to do.

Managed does not mean "we ran the same install script you would have." It means the defaults that quietly fail at small scale were found and fixed before you met them.