Scaling data processing with an embedded key-value store: insights from our migration to BadgerDB

The Mux Data team migrated from a shared, distributed Riak key-value (KV) store to the BadgerDB embedded KV store, resulting in lower operational costs, boosted scalability, and reduced latency. This change was made due to challenges with scaling up and down in response to changing load on their databases. The team found that managing the number of Riak nodes was a manual process and that it led to poor utilization of resources. They also experienced issues with stream processing latency as Riak often became a write-throughput bottleneck during large live events. By scoping the cache to a Kafka topic partition, they were able to follow a "Shared Nothing" architecture, which allowed them to optimize for the performance of frequently occurring activities and reduce operational costs. The team chose BadgerDB due to its high performance writes and reads, control over garbage collection frequency, granular data layout on disk, and compression capabilities. Since migrating from Riak to BadgerDB, they have seen significantly lower latency and been able to handle larger-scale events without any scaling concerns.