Migrating Kafka to the Cloud: How Skai Went From 90K to 1.8K Topics

We managed an online advertising campaign using an interactive tool that let us pivot, analyze, filter, and display performance across all channels. However, as we added features and accrued more data, our UI updates became slower due to the relational databases' inability to scale. We investigated change-data-capture replication with fast in-memory databases using Apache Kafka and Debezium connector. Our initial architecture improved performance but had limitations, including high maintenance costs and inefficient utilization. To address these challenges, we consolidated tables into shared topics and pipelines, reducing the number of topics from 90K to 1.8K. We also implemented a new topic type for heavy-load tables with six partitions and a caching mechanism to reduce API calls. Our re-architecture reduced topics by over 90% without sacrificing performance, freeing up brokers and reducing maintenance time and cloud budget.