Data Orchestration Trends: The Shift From Data Pipelines to Data Products

This article discusses the shift from a data pipeline-centric view to a data product-focused approach, driven by the need for better understanding and management of complex data environments. It explores the concept of data orchestration tools that introduce data assets as first-class objects and how they help reconcile the worlds of data analysts and business users with those of data engineers. The article also covers various abstractions (jobs, tasks, resources, triggers, and data products) and their role in building a Data Mesh. It highlights the importance of focusing on data products instead of data pipelines and why it is better to write declarative code so that data pipelines can update themselves when upstream data assets change.