RankVicuna: Zero-Shot Listwise Document Reranking with Open-Source Large Language Models

In this paper, the authors propose RankVicuna, an efficient and deterministic reranking model for large language models (LLMs). The model is based on Vicuna, which has been fine-tuned using instruction data from Open Assistant. The main advantage of RankVicuna is its smaller size compared to proprietary models like GPT-3.5 and GPT-4, while still achieving comparable performance in terms of ranking metrics such as NDCG@10 and MAP@100. The authors also highlight the importance of data augmentation for ensuring stability in document reordering. They demonstrate that using a teacher model to generate training data examples from a larger model can improve performance, especially when dealing with smaller datasets. Additionally, they showcase the effectiveness of prompt engineering in achieving stable outputs and reducing hallucinations. Overall, RankVicuna offers an open-source alternative for reranking LLM outputs, which could be particularly useful for teams that do not have access to proprietary models like GPT-3.5 or GPT-4. However, further research is needed to evaluate the model's speed and scalability in production settings.