/plushcap/analysis/clickhouse/clickhouse-building-a-data-warehouse-with-clickhouse-part-2

How we built our Internal Data Warehouse at ClickHouse: A year later

What's this blog post about?

The blog post discusses how ClickHouse's internal data warehouse (DWH) has evolved over the past year to support a more diverse set of users, data sources, and access points. It highlights the use of ClickHouse and dbt as primary components in the stack that have enabled DWH to support real-time data processing into regular batch reporting. The post also covers how the architecture of DWH has been configured with nineteen raw data sources, handling 6 billion rows and 50 TBs of data daily. It mentions how dbt centralized transformation logic related to batch reporting in one place, making it easier to manage growing complexity as SQL became the way business logic was encoded. The post also discusses incorporating more real-time data into DWH and configuring additional access points for users. Finally, it outlines future plans for scaling DWH by decentralizing compute resources and exploring AI features.

Company
ClickHouse

Date published
Sept. 10, 2024

Author(s)
Mihir Gokhale

Word count
1958

Hacker News points
None found.

Language
English


By Matt Makai. 2021-2024.