How OpenMeter Uses ClickHouse for Usage MeteringLeveraging ClickHouse and Kafka Connect for Robust Metering

OpenMeter uses ClickHouse, a column-oriented database, to store and query historical usage data. It ingests events into Kafka topics first and then transfers them to ClickHouse using Kafka Connect. The ClickHouse Kafka Connect Sink plugin ensures exactly-once delivery between Kafka topics and ClickHouse tables. OpenMeter conducts pre-aggregation of usage events into one-minute tumbling windows, which are stored in a long-term database for real-time metering at scale. It leverages the Dead Letter Queue pattern to handle failure scenarios. In future efforts, OpenMeter plans to explore leveraging scalable stream processing technologies like Arroyo to reduce batch sizes before sending data to ClickHouse and optimize queries using MaterializedViews.