The Time Our Provider Screwed Us

In October, a special Test in Production session was cohosted by Honeycomb.io and featured speakers sharing their scariest stories of operational outages and other horrors. One such story was told by Paul Biggar, CTO and Co-founder of Dark, who recounted how a massive security breach nearly ended his previous company, CircleCI. The incident began with the discovery of a missing AWS identity and access management key, which led to a full-scale security investigation. CircleCI was affected by the Mongo HQ security breach, as they were a customer of the compromised database provider. After recycling all their keys, CircleCI faced the challenge of communicating with customers and restoring service while ensuring safety. Jesse Robbins, CEO of Orion and former Master of Disaster at Amazon.com, provided valuable guidance on how to handle the situation effectively. Throughout the incident, transparency was prioritized, with regular updates being shared with customers via email and social media. The company managed to retain almost all its customers despite the scare, thanks in part to their transparent response. This experience taught Paul Biggar valuable lessons about the importance of encryption at rest and maintaining transparency during crisis management.