How we built our AI Lakehouse

AssemblyAI has developed an AI Lakehouse solution to manage and store large volumes of audio data and metadata effectively. The primary goals of this project are to democratize data access while ensuring security and compliance, consolidate datasets across the organization in a high-quality manner, and shift dataset quality responsibility to the requester. The design of their AI Lakehouse is intended to efficiently manage, store, and serve large volumes of data, offering fast access and robust analytics capabilities. They chose Google Cloud Storage (GCS) for blob storage and Bigtable for metadata storage due to its favorable cost-to-performance ratio and compatibility with their needs. The solution they chose for integrating metadata into BigQuery is leveraging BigQuery Scheduled Queries to create a BigQuery native table from the Storage Layer every 24 hours, focusing on essential data and updating it daily. This approach provides a balance between simplicity, performance, and cost-effectiveness while maintaining heavy-duty, detailed tables for higher resolution queries if needed.