Understanding Iceberg Support in the Databricks Ecosystem

Spark's support for Iceberg has led to its popularity, with Databricks having strong support for both technologies. Unity Catalog, a core part of any Databricks deployment, can now act as an Iceberg Catalog. UniForm allows automatic generation of Iceberg metadata tables alongside Delta Lake without the need for conversion or creation of specific Iceberg-formatted tables within Databricks. However, compatibility is not perfect yet and some additional steps may be required. Accessing these tables directly can be done using the Iceberg Catalog REST API for Unity Catalog. Writing to these catalogs is currently not supported. The acquisition of Tabular by Databricks in June 2024 has brought together two major players in the data management field, and it will be interesting to see how their combined expertise will shape the future of Iceberg and Data Lakehouse architectures.