Company
Date Published
Author
Paul Dix
Word count
1606
Language
English
Hacker News points
2

Summary

InfluxDB IOx is a new Rust and Apache Arrow based core for InfluxDB that aims to manage the data lifecycle of time series data by optimizing real-time ingest, buffering, in-memory data, and writing out large blocks of immutable data in a compressed format. The system defines APIs and configuration to manage data movement as it arrives and periodically in the background, allowing data to be sent to other InfluxDB IOx servers for processing or to object storage in the form of write ahead log segments or Parquet files and their summaries. Data is logically organized into three main components: the MutableBuffer, the ReadBuffer, and ObjectStore, which work together to provide a consistent logical organization describing data, allowing queries to work across all sources and be combined at query time. The system also provides features such as single-writer, many-reader semantics for interactions with object storage, and atomic switching between MutableBuffer Chunk and ReadBuffer Chunk.