Using Control Streams to Manage Apache Flink Applications

Stream processing systems like Apache Flink can easily span a large number of hosts, but managing stateful distributed applications presents challenges in performing administrative tasks without causing downtime. Mux solved this problem by introducing a control stream to their Flink application. This allows for monitoring and inspecting the inner workings of the streaming application, which is crucial for maintaining confidence in its correctness and preventing regression defects. By using RabbitMQ as a new stream-source for control messages, Mux was able to easily add a control stream that has performed well and enabled offline analysis and validation of machine-learning models.