The problem
Kafka's failure behaviour is easy to read about and hard to trust until you've watched it. What does a producer actually see when a broker dies, and when does Kafka refuse writes instead of risking them?
A 3-broker Kafka cluster on Kubernetes, broken on purpose in three ways, with what actually happened and a runbook for each.
Kafka's failure behaviour is easy to read about and hard to trust until you've watched it. What does a producer actually see when a broker dies, and when does Kafka refuse writes instead of risking them?
The cluster has one KRaft controller and three brokers. The orders topic has 3 partitions, replication factor 3 and min.insync.replicas=2. A producer writes 50 messages a second and one consumer reads them.
| Drill | What I broke | What happened |
|---|---|---|
| Broker down | Killed 1 of 3 brokers | All partitions under-replicated, one leader failed over, one 503 ms latency spike, no errors. Back in sync in about 25 s. |
| Consumer lag | Stopped the consumer for about 70 s | Lag grew to 2,403. The producer was unaffected. Caught up in under 18 s. |
| Not enough replicas | Killed 2 of 3 brokers | NOT_ENOUGH_REPLICAS 1,079 times. Writes rejected for about 3.5 min, then full recovery. |
One broker down is a non-event at this replication setting. The alert to watch is UnderReplicatedPartitions, not producer errors.
Lag is a consumer problem, not a broker problem. Kafka holds the data, so the question is whether consumers catch up before retention deletes it.
acks=all with min.insync.replicas trades availability for safety. With two brokers gone, Kafka refused writes instead of keeping a single copy, so the producer has to handle that failure.
After recovery, leadership is uneven. Returning brokers come back as followers, and Kafka moves leaders back on its own every five minutes, or you can force it with a preferred leader election.
It needs Docker with about 6 GB of RAM, kind and kubectl. One script brings up the cluster, Strimzi, Kafka and the traffic. Each drill folder has break.sh, the output.txt from a real run and runbook.md.