Shift Left: Headless Data Architecture, Part 1

The headless data architecture separates data storage, management, optimization, and access from the services that write, process, and query it. This approach allows for managing data from a single logical location, simplifying regulatory compliance as data resides in one place instead of being copied around to every processing engine. Headless data architecture can encompass multiple data formats, including streams and tables. Streams provide low-latency access to incremental data, while tables offer efficient bulk-query capabilities. Key components for enabling tables in a headless data architecture include table storage and optimization, an Iceberg catalog containing metadata, schemas, and table information, transactions support, time travel capabilities, and a central pluggable data layer. The main benefits of this architecture are not having to copy data around, eliminating multiple copies of data, choosing the most suitable processing head for each task, and having a single point of access control.