Airflow in Action: Scaling Insights from Bosch and 1.2 Million Pipeline Runs Per Day

The Bosch Group, a leading global technology supplier, uses Apache Airflow® to handle 1.2 million pipeline runs per day in its data-driven applications and analytics. Initially designed to manage 1,000 DAG runs per hour, the company faced challenges due to frequent event storms that impacted data freshness. To address this issue, Bosch scaled Airflow by iterating through various improvements such as tuning hardware, upgrading config parameters, and optimizing DAGs and environments. As a result, they achieved 50,000 DAG runs per hour with an average of 1.7 million events processed daily and single-second latency.