Why AI shouldn’t reinvent ETL

Recent advancements in AI have led to its use in various fields such as customer support, code writing, music creation, and medical imaging. However, a concerning trend has emerged where the AI community is reinventing data movement (ELT). This involves building extraction and loading pipelines from scratch for APIs, document formats, and databases, which can be time-consuming and wasteful. The main focus should be on improving core products rather than rebuilding existing solutions. Data movement is crucial in AI as it enables models to access specific non-public information. However, building robust extraction pipelines is challenging due to factors like incremental extracts, error handling, authentication, rate limits, and API changes. Instead of reinventing the wheel, AI engineers should consider using existing solutions or portable frameworks for data movement.