The Practical Guide to Self-Hosting Compound LLM Systems

The article discusses self-hosting large language models (LLMs) and provides actionable advice for those who prefer control and customization while trying to achieve the performance of just calling a managed API. It highlights BentoML's research insights in AI orchestration, demonstrating solutions it developed for optimizing common performance issues when self-hosting models. The article also explores how to integrate BentoML and Milvus to build more powerful GenAI applications. The LLM Doom Stack is introduced as a framework that includes data, operations, orchestration, and AI models. It explains the benefits of using vector databases like Zilliz/Milvus in various LLM-powered systems, particularly retrieval augmented generation (RAG). The article also discusses the challenges and considerations for self-hosting LLMs, such as control, customization, and long-term cost benefits. The article presents key approaches to address these challenges, including inference optimization techniques like batching requests, token streaming, quantization, kernel optimizations, and model parallelism. It also discusses scaling LLM inference with concurrency-based autoscaling, prefix caching for cost savings, and solutions to the cold start problem. Finally, the article explores integrating BentoML and Milvus for more powerful LLM applications, particularly Retrieval Augmented Generation (RAG). It provides resources for building RAG or other types of GenAI APPs using these tools.