Stateful Stream Processing with Kafka and Go

Mux Data has transitioned from using Amazon Web Services' (AWS) Kinesis service to the open-source Apache Kafka distributed log for processing video analytics beacons. The move was made in order to unlock significant cost savings, cloud portability, and greater control over performance. This post discusses some of the engineering work performed during this transition, including a primer on Apache Kafka and its use cases, as well as details about how Mux Data uses Kafka for stateful processing of video views beacons. The sticky partitioning scheme was introduced to minimize the impact of rebalances during consumer group rebalances. Additionally, the importance of monitoring consumer group session timeouts is highlighted, as it can affect the shutdown sequence and trigger a rebalance prematurely. Overall, Kafka has proven to be a reliable and flexible solution for Mux Data's needs.