Company
Date Published
Author
Amjad Almahairi
Word count
256
Language
English
Hacker News points
None

Summary

We recently collaborated with the LMSys team to develop an open-source LLM Router that delivers high-quality LLM applications at half the cost by dynamically selecting between high-performance proprietary LLMs and cost-effective open-source models. The LLM classifier directs "simple" queries to Mixtral-8x7B, maintaining high overall response quality while significantly reducing costs. Anyscale and LMSys LLMRouter sets a standard for quality and cost savings, achieving higher quality with lower costs compared to public LLM routing systems. The router is now available as an Anyscale Workspace Template, simplifying training and deployment with built-in observability and debugging tools.