Cassandra counter columns: Nice in theory, hazardous in practice

Apache Cassandra is a distributed database system that scales without limits and is used by Ably Realtime for persistent storage of messages. However, the company experienced disruptions in early 2020 due to issues with counter columns in Cassandra. The problem was eventually traced back to excessive retries on an operation involving counter column updates. To resolve the issue, rate limiting was implemented and the use of counter columns was eliminated from the system. This resulted in a significant reduction in load on the Cassandra instances. The experience highlighted the importance of considering practical considerations alongside theoretical knowledge when building distributed systems at scale.