The Penny Glitch That Cost Big

In a special Test in Production session with Honeycomb.io, various speakers shared their horror stories of operational outages and other scary events. Marc Devens, Director of Product at BrandVerge, discussed two such incidents. The first involved Appeagle, a company that provides price management software for third-party sellers on Amazon, eBay, and Walmart. A major pricing error occurred when items were listed as one cent instead of thousands of dollars. The second incident involved the launch of multiple features with an offshore contract team after months of only shipping fixes. Devens emphasized the importance of testing in production and ensuring that disasters never happen again by implementing tests, safety checks, and gating mechanisms to control rollouts and enable quick reversals if necessary.