Distributed Consensus Reloaded: Apache ZooKeeper and Replication in Apache Kafka

This post discusses consensus in distributed systems, focusing on Apache ZooKeeper as an example. Consensus is essential for coordinating processes in distributed systems and is often solved by dependencies like Apache ZooKeeper, Consul, etcd, or homebrewed versions based on Raft. The authors explain that while these systems vary on the features they expose, their core solves a fundamental problem: agreement. They delve into how consensus manifests in distributed systems and some caveats, using the replication scheme of Apache Kafka as an example. The post also covers atomic broadcast, which is implemented by ZooKeeper, and its relation to consensus. Furthermore, it discusses the role of consensus in Kafka's replication protocol and how it uses ZooKeeper for coordination and metadata management of topics.