Airflow in Action: Scaling Insights from Bosch and 1.2 Million Pipeline Runs Per Day
The Bosch Group, a leading global technology supplier, uses Apache Airflow® to handle 1.2 million pipeline runs per day in its data-driven applications and analytics. Initially designed to manage 1,000 DAG runs per hour, the company faced challenges due to frequent event storms that impacted data freshness. To address this issue, Bosch scaled Airflow by iterating through various improvements such as tuning hardware, upgrading config parameters, and optimizing DAGs and environments. As a result, they achieved 50,000 DAG runs per hour with an average of 1.7 million events processed daily and single-second latency.
Company
Astronomer
Date published
Nov. 19, 2024
Author(s)
Matthew Keep
Word count
489
Hacker News points
None found.
Language
English