/plushcap/analysis/astronomer/astronomer-airflow-in-action-bosch

Airflow in Action: Scaling Insights from Bosch and 1.2 Million Pipeline Runs Per Day

What's this blog post about?

The Bosch Group, a leading global technology supplier, uses Apache Airflow® to handle 1.2 million pipeline runs per day in its data-driven applications and analytics. Initially designed to manage 1,000 DAG runs per hour, the company faced challenges due to frequent event storms that impacted data freshness. To address this issue, Bosch scaled Airflow by iterating through various improvements such as tuning hardware, upgrading config parameters, and optimizing DAGs and environments. As a result, they achieved 50,000 DAG runs per hour with an average of 1.7 million events processed daily and single-second latency.

Company
Astronomer

Date published
Nov. 19, 2024

Author(s)
Matthew Keep

Word count
489

Hacker News points
None found.

Language
English


By Matt Makai. 2021-2024.