Harness builds an end to end data platform with dlt + SQLMesh

Harness builds an end-to-end data platform using dlt + SQLMesh. Harness relies on dlt to handle data extraction, normalization, and loading of data sources, with 14 active sources and growing. dlt also handles schema automation and evolution. The company uses BigQuery as its data warehouse. Once the data is in the data warehouse, Harness uses SQLMesh to provide data contracts between the data in the data warehouse and how it's being transformed downstream. dlt and SQLMesh are integrated to achieve interoperability between them. Harness empowers its senior data engineer, Alex Butler, to make technical decisions without needing to go to someone else, allowing him to build everything himself. Alex was previously using Singer, Meltano, and dbt, but found dlt to be a more efficient solution. He migrated his company's core SaaS service pipelines to dlt in just 2 weeks and achieved full confidence that the data pipelines would never break due to changing schemas. Harness has integrated dlt into its own platform, allowing users to control what resources of sources are enabled or disabled through the UI. This integration enables "data democracy" for its product, business, and operation teams, allowing them to independently satisfy a majority of their data needs through no-code self-service. Alex also migrated his company's dbt models to sqlmesh, which provided him with column-level lineage and interoperability between dlt and sqlmesh. The integration allows Harness to switch test, development, and production destinations easily, and provides an operational dashboard that assesses the health of individual columns per model in a fully automated way.