Can ML be absorbed by the DBMS?
The relationship between machine learning (ML) and database management systems (DBMS) is complex, with different strategies for integrating ML into DBMS depending on the composition of work. On one end of the spectrum are classic analytics teams using SQL data warehouses, where special SQL syntax for creating and evaluating ML models can be beneficial. BigQuery ML is an example of this approach. On the other end are data science teams primarily using machine learning libraries in Python, who may benefit from embedded DBMS like DuckDB. Many users exist somewhere in between these two extremes, with interoperability being a key challenge. Solutions such as Apache Arrow and Lakehouse aim to address this issue by allowing efficient exchange of relational data between systems. The popularity of each approach will depend on the specific needs and problems people are trying to solve.
Company
Fivetran
Date published
May 26, 2022
Author(s)
George Fraser
Word count
1060
Language
English
Hacker News points
1