Orchestrating ELT in Airflow: Catching every sync with XCOM
The Fivetran Airflow provider has released a new version that eliminates issues with hanging sensors. In previous versions, some connectors' syncs would finish before the corresponding sensor could start monitoring them, causing potential problems. This issue was resolved by utilizing XCOM, an Airflow mechanism that allows tasks to pass key-value pairs between each other. The latest version of the Fivetran Airflow provider now stores a timestamp of the connector's previously completed sync in XCOM, which can be used by a sensor through calling xcom_pull. This ensures that the previously completed timestamp will be correct and a sensor will return at the appropriate time, even if it does not start until after the sync that it is supposed to monitor has already completed.
Company
Fivetran
Date published
July 6, 2022
Author(s)
Nick Acosta
Word count
729
Language
English
Hacker News points
None found.