Airflow in Action: Supercharging Data Science and ML Workflows At Apple

Neha Singla and Sathish Kumar Thangaraj, senior software engineers on Apple’s Data Platform team, presented a session at this year's Airflow Summit demonstrating how they used Jupyter notebooks and Apache Airflow to streamline data science workflows. They tackled common bottlenecks in transitioning experiments from prototype to production, resulting in increased productivity, simplified debugging, and support for large-scale, distributed workflows. The solution involved using the Papermill operator to parameterize and execute Jupyter Notebooks, which was extended by Apple engineers to support multiple languages and runtimes, as well as remote kernels running in Kubernetes clusters. This approach has delivered tangible benefits such as enhanced productivity, scalability, and improved debugging. Looking ahead, the team aims to further enhance their solution by supporting event-driven notebook workflows, improving workflow sharing, and collaborating with the open-source community to expand capabilities.