How Deep Learning & GBMs Can Change the Tabular Data Landscape

The article discusses how deep learning can revolutionize the handling of tabular data, a crucial yet often overlooked aspect of artificial intelligence. Despite advancements in language processing and image recognition, modern deep learning methods have not been adequately researched or applied to tabular data. Many research papers and online courses assume that "classical" tree-based models and gradient boosting machines (GBMs) outperform deep learning methods on this type of data. However, the article argues that these studies often use standard benchmark datasets, which are limited in scope and do not represent the entire tabular data domain. Furthermore, the baseline models used for benchmarking in tree-based models are highly optimized state-of-the-art techniques, while neural networks pitted against them tend to be simple MLPs or standard architecture variants. The article highlights that times have changed, and tabular datasets now account for a majority of day-to-day data analysis and processing. The domain space encompassed by these datasets has been greatly expanded due to advancements in data collection techniques. Tree-based models are not necessarily "bad choices" for modern tabular data modeling but may fall short when it comes to large and complicated datasets. Deep learning models have limitations, such as interpretability, lack of data, and inability to preprocess data effectively. However, recent studies from Caglar Aytekin demonstrate that neural networks can be understood like any other decision tree, a massive leap in the field of neural network interpretability. Additionally, tabular GANs have been explored for generating synthetic tabular data, with models such as TGAN and CTGAN providing anonymization features to protect user privacy. Incorporating the best of both worlds, many models have applied the tree-based modeling concept to deep learning models, such as GrowNet, Deep Neural Decision Trees, Neural Oblivious Decision Ensembles, and XBNet. Researchers have also been "stealing" the success of attention-based Transformers for tabular data with models like TabTransformer, TabNet, and SAINT. The article concludes by stating that deep learning models have potential in solving predictive tasks involving tabular data, as evidenced by successful applications such as the Kaggle Mechanism of Action (MoA) competition and the Jane Street Market Prediction. The importance of tabular data modeling cannot be ignored, and with modern techniques, its potential is unbounded.