The NeurIPS 2024 Preshow: Zero-Shot Learning: A Misnomer?

Recent research challenges the notion of "zero-shot" capabilities in deep learning models like CLIP and Stable Diffusion. The study by Vishaal Udandarao reveals that multimodal model performance is strongly predicted by concept frequency in pre-training data, suggesting that these models may recognize rather than generalize concepts based on their prevalence in training data. This log-linear relationship implies highly sample-inefficient learning in current multimodal models and highlights a fundamental limitation: they are data-hungry and struggle to learn concepts efficiently, particularly those in the long tail of the distribution. The research emphasizes the need for careful consideration of concept frequency and diversity during data curation to mitigate these challenges.