Company
Date Published
Author
Ernie Ostic
Word count
1350
Language
English
Hacker News points
None

Summary

Data lineage is the ability to track how data flows through an enterprise, allowing users to understand where data comes from, where it goes to, and what happens to it along the way. This is critical for gaining insights into assets and their impact on pipelines in organizations. However, achieving data lineage is difficult due to various reasons such as business vs technical view, level of detail, scope, and company evolution. Data lineage is often manual and time-consuming, making it challenging to meet regulatory deadlines and prove its accuracy. It plays a crucial role in governance and regulatory compliance by providing trust in data, enabling the identification of data quality issues, and facilitating prioritization of fixes. Companies like MANTA provide end-to-end lineage solutions that visualize data flow across various systems, including mainframe sources, operational systems, and data warehouses.