Testing Infrastructure Changes in Production

Rosemary Wang, a Developer Advocate at HashiCorp, presented her experiences with testing infrastructure changes in production during the November Test in Production Meetup in Berlin, Germany. She discussed how feature flags, canary tests, and A/B tests can be used to control the blast radius when making network updates. Feature toggles or flags tell if something is "on" or "off," preserving the state of infrastructure. Canary testing involves releasing a new feature to a small group of users before rolling it out to everyone else, allowing for smoke testing and early identification of issues. A/B testing can be used in the infrastructure space by measuring how changes affect upstream service level objectives, such as processing speed or cost. These techniques help organize infrastructure blast radius and move from risk aversion to risk mitigation.