Shift Left: Headless Data Architecture, Part 2

A headless data architecture is a formalization of a data access layer at the center of an organization. It encompasses streams and tables, providing consistent data access for both operational and analytical use cases. Streams enable low-latency capabilities to react timely to events, while tables provide higher-latency but extremely batch-efficient querying capabilities. Building this architecture requires shifting work left from downstream to upstream in the source system. A shift-left approach simplifies data access, reduces costs, and provides a more efficient way to create, access, and use data compared to traditional multi-hop architectures. The medallion architecture is a popular form of the multi-hop architecture, but it has several drawbacks such as latency, cost, brittleness, and redundancy. A headless data architecture addresses these issues by shifting left and using stream-first data products composed of a stream (powered by Apache Kafka®) and its related table (powered by Apache Iceberg™). This approach provides data freshness in sub-seconds, making data access cheaper, easier, and faster across the organization.