Guest Post: Controlling Elixir Supervisors at Runtime with Feature Flags

Knock, a company that provides simple APIs and a dashboard for introducing thoughtful notifications into products, uses LaunchDarkly to power feature flags. They recently adopted a circuit breaker pattern built around feature flags to make their services more reliable when things fail. The problem they faced was with AWS Kinesis consumers failing in ways that do not gracefully recover when they crash. To address this issue, Knock developed an Elixir supervisor called Circuit Breaker that takes inspiration from the typical circuit breaker pattern and applies it to supervision trees. This approach gives them a lot of flexibility in dealing with failures and extra load in their system. By turning off consumers strategically, they can intentionally degrade their service in order to maintain uptime on more critical parts of their infrastructure. The Circuit Breaker monitors the feature flags for each process and adds or removes child specs based on the flag status. They use Elixir/Erlang Telemetry module extensively at Knock to monitor runtime performance, including crash metrics and log messages. Future improvements include making enhancements to their Kinesis consumer library, abstracting their dependency on LaunchDarkly, and potentially releasing this tool as open source.