The Open (aka Modern) Data Stack Distilled into Four Core Tools - Part I

The article explores core open-source tools needed for a company to become data-driven, focusing on integration, transformation, orchestration, analytics, and ML tools as part of the latest open data stack. It introduces the Modern Data Stack (MDS) and explains how it is extendable like lego blocks, usually consisting of data integration, a transformation tool, an Orchestrator, and a Business Intelligence Tool. The Open Data Stack is then defined as a better term for the MDS, focusing on solutions built on open source and open standards covering the data engineering lifecycle. The article presents Airbyte as the number one choice for data integration, highlighting its reliability, extensibility, integrations, and transparency. It also discusses dbt as the king of SQL for data transformation, emphasizing its documentation generation, reusability of SQL statements, testing, source code versioning, added functionality to plain SQL with Jinja Templates, and Python support. For analytics and data visualization (SQL), Metabase is recommended due to its simplicity and ease of setup for non-engineers. It allows users to ask questions about their data and displays answers in formats that make sense, whether a bar chart or a detailed table. The last core data stack tool discussed is Dagster as the data orchestrator, which enforces best practices such as writing declarative, abstracted, idempotent, and type-checked functions to catch errors early. Additional components of the Open Data Stack are also mentioned for inspiration, including semantic layer/metric layer, data quality and data observability, reverse ETL, and data catalogs. The article concludes by encouraging readers to explore these tools in action through a tutorial on configuring Airbyte connections with Python (Dagster).