The NeurIPS 2024 Preshow: What matters when building vision-language models?

Developing Vision Language Models (VLMs) is a complex task with numerous challenges, as highlighted by Hugo Laurençon's research. These models combine language processing capabilities with image processing to generate text based on visual inputs. Key design considerations include architecture choices such as cross-attention and fully autoregressive models, as well as strategies for improving training efficiency like learned pooling techniques and image splitting. Data quality is crucial, with VLMs trained using diverse datasets including synthetic captions. Ongoing evaluation of model performance beyond benchmarks is essential to uncover biases and areas for improvement.