From 15,000 database connections to under 100: DigitalOcean's tale of tech debt

The text describes the technical rearchitecture undertaken by cloud provider company DigitalOcean to address its growing tech debt. It began as a Rails application in 2011 with three main services - Cloud, Scheduler and DOBE (DigitalOcean BackEnd), all communicating via a MySQL database serving two roles: storing data and brokering communication. As the company grew rapidly, user traffic increased over 10,000% from 2012 to 2016, leading to more events on the database message queue and causing performance drops due to table locks and query backlogs. The engineering staff came up with a three-pronged plan: decrease direct connections on the database, refactor Scheduler's ranking algorithm for improved availability, and absolve the database of its message queue responsibilities. This led to the creation of Event Router, Scheduler V2, and Harpoon, which together significantly reduced the number of direct connections to the database, improved scheduling efficiency, and provided an API for aggregating requests and performing queries on behalf of the database. The story illustrates how DigitalOcean tackled its technical debt by rearchitecting its system design while maintaining simplicity as a core value.