Using Temporal to Scale Data Synchronization at PeerDB

PeerDB uses the open-source workflow orchestration platform Temporal to manage retries, long-running state, and ensure idempotency in data synchronization across distributed systems. The company's architecture focuses on statefulness, idempotency, retryability, and separation of concerns. They have developed a Connector interface that encapsulates the logic for various operations critical for their data synchronization tasks. Temporal enables PeerDB to divide workflows into multiple activities tailored to the requirements of sources and destinations, providing granular control and visibility over each operation's state and progress. The system scales gracefully with needs, allowing easy expansion to new data sources or destinations without impacting existing ones. Deployment is also made easier through containerization and Kubernetes clusters for horizontal scaling.