Company
Date Published
Author
Ratnadeep Debnath
Word count
282
Language
English
Hacker News points
None

Summary

Zapier relies heavily on RabbitMQ for message queuing in their backend workers, which run on Kubernetes. To manage varying task loads, they previously used CPU-based autoscaling with Horizontal Pod Autoscale (HPA), but encountered issues due to blocking I/O operations in Python. This led to a situation where idle workers could accumulate a growing backlog of messages, causing traffic jams and delays. Zapier sought a solution that would scale their workers based on both CPU usage and message queue length, but found Kubernetes' native HPA lacking this capability. They eventually turned to KEDA as a more suitable solution for dynamic scaling.