Increasing Deployment Safety with Metric Based Canaries

In September, Brandon Leach, Senior Engineering Manager at Lookout, discussed increasing deployment safety with metric-based canaries during a Test in Production Meetup. Lookout is focused on mobile end-point security and has over 100 million devices registered. The company recently rebuilt its service delivery tooling for continuous delivery using Spinnaker, a service delivery platform created by Netflix. This transition resulted in significant improvements in deployment efficiency and service stability. Lookout's next goal was to achieve continuous deployment through testing in production. To do this, they began looking for ways to automate metric-based deployments in Spinnaker. Through collaboration with Armory, Lookout developed Barometer Service, which integrates with Spinnaker and uses time series data from monitoring platforms like DataDog to orchestrate metric-based canaries. The Barometer service compares the metrics of a baseline deployment with those of a canary deployment. If the metrics fall outside a defined deviation, the canary fails. Once the canary stage is completed, both the canary and the baseline server groups are destroyed, and the pipeline continues either as a success or a failure. Lookout has learned that it takes time to identify the appropriate metrics for monitoring deployments, and too many metrics can cause false positives. While metric-based canaries may initially slow down deployment times, they ultimately save time by automating the evaluation process. The company is currently in the process of migrating its canary deployments from Barometer to Kayenta, a platform developed through collaboration between Netflix and Google for automated canary analysis in Spinnaker.