Using Apache Kafka to process 1 trillion inter-service messages

Cloudflare has been using Kafka since 2014 and currently runs 14 distinct Kafka clusters across multiple data centers with roughly 330 nodes. The company uses Kafka to decouple microservices, enabling concurrent work on features and products. They have built internal tools to ease adoption of Kafka, including the Messagebus-Client library and a schema registry called Messagebus Schema. Additionally, they provide a connector framework for easy spinning up services that can read from a system of record and push it somewhere else. To ensure observability, automated metrics and alert creation are provided wherever possible. As usage of Kafka continues to grow at Cloudflare, the company remains committed to improving tooling around Messagebus and making it easier to use, customizable, and observe.