Company
Date Published
Author
Robin Guldener
Word count
1496
Language
English
Hacker News points
8

Summary

We recently migrated millions of tasks from Temporal to a Postgres-based task scheduler at Nango, a company that helps engineering teams build product integrations. The usage of their platform has grown significantly over the last year, requiring them to upgrade their stack. They replaced a core part of their infrastructure that schedules and runs interactions with third-party APIs, previously using Temporal. However, they were unable to leverage Temporal's key feature: Seamless resuming of failed workflows due to running untrusted user code. This reduced Temporal to an expensive and complex queuing and scheduling system for Nango. They evaluated different alternatives, including Hatchet and Postgres queuing libraries, before deciding to build their own queue and scheduler on Postgres. The new solution is simple, flexible, and allows full control over how schedules run. Nango was able to migrate millions of tasks with no downtime, using a dry-run strategy, feature flags, and monitoring performance and task completions. The migration was successful, and the new system excelled at handling three different workloads: main relational application database, object store for continuous data syncs, and queue and scheduler for orchestrating interactions with third-party APIs.