Company
Date Published
Jan. 22, 2025
Author
Matthew Keep
Word count
548
Language
English
Hacker News points
None

Summary

At the Airflow Summit 2024, Bhavesh Jaisinghani, Data Engineering Manager at Autodesk, shared how his team transformed testing of their data pipelines by creating a secure production-like UAT environment with Astronomer and Apache Airflow. This setup enabled seamless testing of Spark-based workflows, reduced development cycles, and ensured data security for sensitive PII. Autodesk's data platform consists of four primary layers: ingestion, transformation, data warehousing, and BI / reporting, relying heavily on Apache Spark for data processing. The company implemented a UAT environment mirroring production, featuring separate infrastructure, granular access control, data sync utilities, and CI/CD integration. This setup delivered impressive results, including improved data quality by 90%, reduced development cycles by 33%, and technical debt reduction. Autodesk's innovative approach powered by Astronomer and Airflow showcases how secure, scalable UAT environments enable faster, more reliable pipeline development.