ETL Pipelines with Airflow: the Good, the Bad and the Ugly

Apache Airflow is a popular open-source workflow management platform often used for ETL pipelines. However, it may not be the best choice for all businesses due to its tightly coupled sources and destinations in transfer operators, making it difficult to cover long-tail integrations. An alternative approach is to use Airflow as a scheduler while integrating with other open-source projects like Airbyte for extracting and loading data, and dbt for transforming data within the data warehouse. This combination allows businesses to remove boilerplate code needed with Airflow and handle dependencies between tables in SQL files instead of Airflow DAGs.