Building a Consistent Kafka ConsumerHow we built a performant and robust Kafka to ClickHouse sink worker

The text discusses the development of a robust solution for transferring events from Kafka to ClickHouse without duplicates, which is crucial for accurate billing in OpenMeter. Initially, ksqlDB and Kafka Connect were used, but scalability issues led to moving deduplication logic out of the API layer and building a custom Go Kafka Consumer. The new consumer ensures consistent deduplication, exactly-once inserts into ClickHouse, event validation, and horizontal scaling through partitions. It also improves processing efficiency by handling invalid events and managing data volumes effectively.