/plushcap/analysis/fivetran/orchestrating-elt-in-airflow-catching-every-sync-with-xcom

Orchestrating ELT in Airflow: Catching every sync with XCOM

What's this blog post about?

The Fivetran Airflow provider has released a new version that eliminates issues with hanging sensors. In previous versions, some connectors' syncs would finish before the corresponding sensor could start monitoring them, causing potential problems. This issue was resolved by utilizing XCOM, an Airflow mechanism that allows tasks to pass key-value pairs between each other. The latest version of the Fivetran Airflow provider now stores a timestamp of the connector's previously completed sync in XCOM, which can be used by a sensor through calling xcom_pull. This ensures that the previously completed timestamp will be correct and a sensor will return at the appropriate time, even if it does not start until after the sync that it is supposed to monitor has already completed.

Company
Fivetran

Date published
July 6, 2022

Author(s)
Nick Acosta

Word count
729

Language
English

Hacker News points
None found.


By Matt Makai. 2021-2024.