DSE 5.1 Resource Manager, Part 1 – Network Connections Security

The DataStax Enterprise (DSE) Resource Manager is a custom version of the Apache Spark Standalone cluster manager that provides functionality similar to a Spark Master when running Apache Spark™ applications with DSE. It enhances the open source Spark Master implementation by automating the management of the Spark Master and Spark Workers lifecycles, providing high availability through an Apache Cassandra®-based solution, offering distributed and fault tolerant storage for Spark Master recovery data, and simplifying configuration for client applications. In DSE 5.1, the introduction of the DSE Resource Manager adds more ease-of-use, security, and stability by replacing the Spark RPC communication mechanism between Spark Driver and Spark Master with a DSE native CQL protocol. This change bolsters the security of Spark applications and DSE while providing several significant benefits such as enabling plain password authentication, Kerberos or other future authentication mechanisms, and allowing for compression, TLS encryption, multiple contact points, and more precise control over communication between components.