Deploying Kafka Streams and KSQL with Gradle – Part 1: Overview and Motivation

Red Pill Analytics recently collaborated with a Fortune 500 e-commerce company to transform their inventory management system using data streaming. The company had traditionally used large warehouses for shipping, but these were slow and couldn't keep up with modern strategies like same-day dropshipping or industrial vending. They planned to roll out smaller, strategically located distribution centers and selected Oracle Warehouse Management Cloud (Oracle WMS Cloud) as their new warehouse management system. Red Pill Analytics was hired to design and implement data integration processes required to connect Oracle WMS Cloud with the company's on-premises systems, including PeopleSoft, an in-house order management system, and a legacy warehouse management system. The customer had already invested heavily in MuleSoft®, which helped abstract many of these different sources/targets as simple REST APIs. Red Pill Analytics chose to use Apache Kafka® and the Confluent Platform for data streaming due to their ability to operate just-in-time and efficiently manage inventory levels, delivery times, and approaches. They used Oracle GoldenGate for Big Data 12c (OGG for BD) to deliver relational data change events to Kafka and MuleSoft to consume relevant topics and translate them into required API calls. The final solution involved using KSQL primarily as the functional engine for expressing streaming data transformations, and Kafka Streams for the final packaging of payloads before shipping them off to specific APIs.