Apache Iceberg 101: The Table Format Reshaping Data Lakes

Apache Iceberg is an open-source table format designed for high scale technical problems, such as supporting multiple concurrent readers and writers over updates and metadata changes. It separates storage of data from compute, allowing companies to choose the storage optimized for their use case and connect it to any querying or computing engine they need. Iceberg's benefits include handling high-scale data, cost optimization, mixed-compute support, and open format preference. However, its Achilles heel is the Iceberg Catalog, which requires a service running somewhere to act as the authority for updates. Currently, no catalog supports all storage providers, but this is expected to change in the future.