The Unbearable Lightness of Horizontal Scaling

The text discusses the pros and cons of vertical scaling, specifically focusing on a multi-threaded system called Dragonfly. While some technical leaders may initially be skeptical about the benefits of this approach due to its single-node architecture, the author argues that it can address production issues and lower costs. Vertical scaling is compared with horizontal scaling, which is often considered infinite but has limitations such as decreased efficiency when dealing with massive instance sizes and increased vulnerability to failure in stateful systems. The text also highlights some hidden hurdles of horizontal scaling, including the fragility of bigger clusters, the need for larger margins of over-provisioning for smaller nodes, and cloud over-commit issues that can reduce throughput per node. Finally, the author presents Dragonfly as a cost-efficient and well-maintained source-available technology that provides vertical scale out of the box, offering significant potential to reduce infrastructure costs compared to alternatives.