Company
Date Published
Author
Stanislav Kozlovski
Word count
4327
Language
English
Hacker News points
None

Summary

Tiered storage for Apache Kafka is a simple yet groundbreaking feature that alleviates common operational problems by reducing local broker data. It significantly eases difficult operational work, unlocks faster recovery from incident scenarios, lowers latency, and reduces storage costs by 3-9x. This feature has been recently released with Kafka 3.9 in November 2024 and has been tested at GB/s scale in Aiven, surpassing 150 GB/s in certain deployments. Tiered storage allows you to scale your storage and CPU/throughput separately, making it simpler operationally by reducing local data per broker and minimizing operational pain points such as broker failure recovery, disk failure recovery, single-partition reassignment, and cluster scale-ups/downs. It also alleviates IOPS problems by using SSDs cost-effectively and reduces the number of brokers to manage when storage is the bottleneck. Tiered storage offers faster latency for local data, unlocks more flexibility in instance type choice, and lowers storage costs by 3-9x, making Kafka simpler, better, and cheaper.