Two Tales

In October, a special Test in Production session was held with Honeycomb.io where speakers shared their scariest stories of operational outages and other horrors. Eric Pollman, CTO and co-founder of ClearBrain, discussed two such experiences from his time at Google. The first incident involved Google Ads going fully offline for 90+ minutes due to a bad data push that caused servers to crash. The second story was about a "zombie haunted" pipeline that kept developers awake late into the night while processing large amounts of data. These experiences taught valuable lessons, such as checking multiple metrics and automating processes to prevent similar issues in the future.