What is Model Soup?

Model Soup is an ensembling technique that improves overall performance by averaging the weights of multiple models instead of combining their individual outputs. This technique has been found to perform better than any individual model on benchmark datasets like ImageNet. Gretel, a company working with synthetic data generation, explored this method to improve model performance on smaller datasets and found promising results. However, they also observed that souping more than five models or certain types of models led to poor performance. Further exploration is needed to determine the effectiveness of Model Soup in different scenarios and identify patterns in its performance.