HTTP Analytics for 6M requests per second using ClickHouse

Cloudflare has evolved its analytics pipeline over the last year to provide HTTP traffic analytics to its customers. The old data pipeline was built in 2014 and faced challenges as the company grew, including a single point of failure (SPOF) for Postgres and Citus main, complex codebase, high maintenance cost, and many dependencies. To address these issues, Cloudflare replaced the old pipeline with a new one based on ClickHouse, an open-source column-oriented database management system. The new data pipeline is more scalable, fault-tolerant, and easier to operate, providing analytics for over 7 million customers' domains. Additionally, historical data was transferred from the old Citus cluster into ClickHouse during the switch. Looking forward, Cloudflare plans to develop products like Log Push and Logs SQL API to enhance its analytics offerings.