Airflow in Action: Data Engineering Insights from Uber and It’s 200,000 Data Pipelines

Uber's data platform is essential to its business model, serving millions of customers and drivers across the globe. In 2016, Uber faced challenges managing multiple data workflow tools used by various teams, leading to inefficiencies and security risks. To address these issues, Uber standardized on Apache Airflow as a single workflow system due to its scalability, unified platform capabilities, and ease of use across teams. The company's internal Airflow fork, Piper, is now used by 1,000 teams with 200,000 different pipelines, orchestrating millions of tasks daily. Uber plans to contribute code back to the Apache Airflow project in its upcoming release.