How we scale workflow orchestration with Temporal

Airbyte uses Temporal as its workflow orchestrator to perform data synchronization between a source and a destination. The key characteristics that led to this choice include the ability to assign work to nodes on both Docker Compose and Kubernetes, flexible scheduling using intervals and crontabs, horizontal scalability, minimal overhead for job execution, easy embedding in OSS releases of Airbyte, and compatibility with Postgres as a backing store. Temporal's Java SDK allows expressing the concept of workflows, activities, and child workflows, which are essential building blocks for data synchronization. The platform organizes Temporal workflows to facilitate flexible internal scheduling of syncs and uses Temporal-specific features like signals, queries, and cancellation scopes to handle various orchestrator states.