How we built a fair multi-tenant queuing system

The article discusses the challenges faced while building a fair multi-tenant queuing system. It explains how they built Inngest as a reliability layer for modern applications, which enables developers to write declarative step functions in code by combining durable execution, event streams, and queues into a single serverless platform. The article highlights the key challenges encountered while developing such a platform: fairness, multi-tenancy, contention, concurrency, read/write load, infrastructure, observability, and customizability. To solve these problems, they developed a brand new queue from scratch for Inngest. This queue is fair, low-latency, and multi-tenant, operating with multiple shared-nothing workers that claim jobs in an almost contention-free way. The article also explains how this works in a multi-tenancy environment, where thousands of users run millions of functions simultaneously. It discusses the concept of unfairness in classic queues and how they built a fair multi-tenant queue to solve it. The solution involves creating separate queues for each function, using a higher level queue to record each function's earliest available job, and ensuring that every function runs independently to guarantee fairness. The conclusion emphasizes the benefits of this approach, which allows queues to become cheap and creates virtual queues to add multiple concurrency levels to each function. This solution provides reliability layers like Inngest, abstracting everything for users declaratively by code so that even fresh engineers can build reliable systems without needing to understand how queueing works.