The evolution of SQL-based observability

SQL-based observability with ClickHouse has matured significantly over the past year, making it a viable option for managing large volumes of logs, traces, and metrics data. The recent addition of JSON support in core ClickHouse, combined with improvements to the OpenTelemetry collector and Grafana plugin, have made SQL-based observability more accessible and efficient. Key takeaways from this blog post include: 1. SQL-based observability with ClickHouse is now a viable option for managing large volumes of logs, traces, and metrics data. 2. The recent addition of JSON support in core ClickHouse has made SQL-based observability more accessible and efficient. 3. Improvements to the OpenTelemetry collector and Grafana plugin have also contributed to the maturation of SQL-based observability with ClickHouse. 4. Users should identify their query access patterns early and choose a sensible primary/ordering key for their tables to ensure optimal performance. 5. Materialized views are essential in meeting performance expectations and transforming OTel data to fit users' access patterns. 6. Time series engines like ClickHouse can be used as the storage backend for Prometheus metrics, offering scalable, high-performance storage with native multi-node support in a memory-efficient architecture with support for high cardinality data. 7. Ongoing improvements are planned for the time series table engine to unlock more of ClickHouse's performance potential within Prometheus, providing a more efficient, scalable solution for robust, long-term metrics storage and analytics.