Mixture-of-Agents (MoA): How Collective Intelligence Elevates LLM Performance

The Mixture-of-Agents (MoA) approach combines multiple large language models (LLMs) with different specialties into a single system to improve overall performance and tackle multi-domain use cases. By leveraging the unique strengths of each LLM, MoA generates higher quality outputs compared to direct input prompts. The MoA framework consists of layers containing specialized LLMs that collaborate to solve tasks iteratively. It has been evaluated on benchmark datasets such as AlpacaEval 2.0 and MT-Bench, demonstrating superior performance over state-of-the-art models like GPT-4 family. However, the reliance on multiple LLMs increases latency, impacting user experience due to higher Time to First Token (TTFT). Future work aims to address this by implementing chunk-wise response aggregation while maintaining its performance.