Company
Date Published
Nov. 7, 2024
Author
Matthew Keep
Word count
943
Language
English
Hacker News points
None

Summary

Bloomberg, a leading financial data provider, transitioned from manual processes to an automated data orchestration framework using Apache Airflow to handle its mortgage ETL pipeline. The previous workflow was labor-intensive, complex, and inefficient, with high risks of failure and limited observability. By adopting Apache Airflow, Bloomberg achieved a 51% reduction in run time, eliminated key person risk, and improved workflow monitoring. The event-driven ETL pipeline now comprises over 100 tasks, using various Airflow operators and custom implementations.