Kafka Failure Drills

reading settings
font:
size:
spacing:

A 3-broker Kafka cluster on Kubernetes, broken on purpose in three ways, with what actually happened and a runbook for each.

to get back in sync after losing 1 of 3 brokers, with no errors
~25s
messages of consumer lag, cleared in under 18s
2,403
NOT_ENOUGH_REPLICAS errors returned, rather than storing writes on one broker
1,079

The problem

Kafka's failure behaviour is easy to read about and hard to trust until you've watched it. What does a producer actually see when a broker dies, and when does Kafka refuse writes instead of risking them?

What changed

  • Showed that losing one broker of three is a non-event with replication factor 3 and min.insync.replicas=2. Every partition was under-replicated, one leader failed over, and there was one 503 ms latency spike and no errors.
  • Measured consumer lag growing at about 50 messages a second to 2,403 while the consumer was stopped for about 70 seconds. The producer was unaffected, and the consumer caught up in under 18 seconds.
  • Showed Kafka choosing safety over availability. With two brokers gone it returned NOT_ENOUGH_REPLICAS 1,079 times and rejected writes for about 3.5 minutes, then recovered fully when the brokers returned.

What I worked on

  • I set up the cluster on kind with the Strimzi operator, running Kafka in KRaft mode with the controller and the brokers in separate pools, so killing brokers never breaks the metadata quorum.
  • I wrote each drill as a script that breaks the cluster, and kept the output of a real run and a runbook beside it.
  • I ran a steady 50 messages a second with acks=all and a 10-second delivery timeout, so failures show up quickly.

The cluster has one KRaft controller and three brokers. The orders topic has 3 partitions, replication factor 3 and min.insync.replicas=2. A producer writes 50 messages a second and one consumer reads them.

The three drills

DrillWhat I brokeWhat happened
Broker downKilled 1 of 3 brokersAll partitions under-replicated, one leader failed over, one 503 ms latency spike, no errors. Back in sync in about 25 s.
Consumer lagStopped the consumer for about 70 sLag grew to 2,403. The producer was unaffected. Caught up in under 18 s.
Not enough replicasKilled 2 of 3 brokersNOT_ENOUGH_REPLICAS 1,079 times. Writes rejected for about 3.5 min, then full recovery.

What I learned

One broker down is a non-event at this replication setting. The alert to watch is UnderReplicatedPartitions, not producer errors.

Lag is a consumer problem, not a broker problem. Kafka holds the data, so the question is whether consumers catch up before retention deletes it.

acks=all with min.insync.replicas trades availability for safety. With two brokers gone, Kafka refused writes instead of keeping a single copy, so the producer has to handle that failure.

After recovery, leadership is uneven. Returning brokers come back as followers, and Kafka moves leaders back on its own every five minutes, or you can force it with a preferred leader election.

Running it

It needs Docker with about 6 GB of RAM, kind and kubectl. One script brings up the cluster, Strimzi, Kafka and the traffic. Each drill folder has break.sh, the output.txt from a real run and runbook.md.