Company
Date Published
Author
Anais Dotis-Georgiou
Word count
3509
Language
English
Hacker News points
None

Summary

The new InfluxDB engine is built on top of Rust, Apache Arrow, DataFusion, and Parquet. These technologies help achieve the requirements for the new InfluxDB engine, including no limits on cardinality, best-in-class performance on analytics queries, separate compute from storage, operator control over memory usage, bulk data export and import, and broader ecosystem compatibility. Rust is chosen due to its exceptional performance and reliability, while Apache Arrow provides efficient columnar data exchange and interoperability with emerging standards in the data and analytics ecosystem. DataFusion is an extensible query execution framework that uses Apache Arrow as its in-memory format, providing best-in-class performance on analytics queries. Parquet is a column-oriented durable file format that enables bulk data export and import, and facilitates data storage at the edge. InfluxData's commitment to open source is reflected in their contributions to upstream projects like DataFusion and Arrow, demonstrating a symbiotic relationship between code and community engagement.