Company
Date Published
Jan. 3, 2025
Author
Matthew Keep
Word count
666
Language
English
Hacker News points
None

Summary

Apache Airflow was successfully used by Circle, a digital financial technology company, to manage their blockchain data orchestration and scale their data platform. Circle initially implemented CI/CD practices, automated testing frameworks, and enhanced data quality checks to overcome challenges in their managed Airflow environment. Despite these improvements, they realized the limits of performance with Managed Workflows for Apache Airflow (MWAA) and began exploring alternative managed Airflow solutions like Astro from Astronomer.io. Circle is now evaluating Astro due to its support for the Kubernetes Executor, which offers flexibility in using containerized Airflow environments, as well as features such as seamless local development, streamlined environments, and data observability with Astro Observe.