Introducing Kafka-Kit: Tools for Scaling Kafka

The text discusses how the company uses Apache Kafka as their messaging persistence layer for handling large amounts of data. They have developed a collection of tools called Kafka-Kit, which includes topicmappr and autothrottle, to handle partition to broker mappings, failed broker replacements, storage based partition rebalancing, and replication auto-throttling. The two primary tools are designed for data placement and replication auto-throttling. They also explain their capacity planning methods and how they use Kafka-Kit in this process. The long term goal is to continue refining the coordination between scaling resource pools and mapping capacity to the right place at the right time, making data movement and recovery hands-off.