Python Async Workers on Fly Machines

This post discusses the use of Fly Machines to implement lightweight background jobs for a Python web application. The author shares their experience with implementing background jobs using Celery, which was overkill and expensive for their specific use case. They then present an alternative solution using Fly Machines, which are lightweight VMs based on Firecracker that start up quickly and have a simple API. The implementation involves writing job parameters to Redis, starting a new machine with the worker code, reading job parameters from Redis, performing the task, writing results back to Redis, and retrieving the results. This approach allows users to only pay for worker resources when they are actually doing work, making it cost-effective for infrequent, expensive background jobs. The author also provides a sample Flask web application that demonstrates this implementation.