Shift YOURSELF Left

Shift YourSELF Left is a talk by Josh that discusses data quality, data contracts, and shifting left in data engineering. The industry has been emphasizing the need for data contracts and shift left to solve data quality issues, but this approach may not be practical or realistic for all teams. Data contracts can be useful when dealing with external sources of data, but they are insufficient for internal data quality issues. Josh's proposed solution centers on shifting testing responsibilities earlier in the lifecycle in a practical and engineer-friendly way, using techniques such as containerizing data pipelines, integrating them into CI/CD systems, and using lightweight tools like DuckDB. This approach aims to bring integration testing into the development process, reduce dependence on monolithic systems for testing, build more reliable and scalable pipelines, and facilitate collaboration between upstream teams.