An Overview of Milvus Storage System and Techniques to Evaluate and Optimize Its Performance

This guide explores Milvus, an open-source vector database known for its horizontal scalability and fast performance. At the core of Milvus lies its robust storage system, which comprises meta storage, log broker, and object storage. The architecture is organized into four key layers: access layer, coordinator service, worker nodes, and storage. Milvus uses three main storage components to ensure data integrity and availability: meta storage (etcd), object storage (MinIO), and a log broker (Pulsar or Kafka). To evaluate and optimize the performance of Milvus storage, it is crucial to monitor disk write latency, I/O throughput, and disk drive performance. The guide provides recommendations for selecting appropriate block storage options from various cloud providers and offers strategies to enhance MinIO's throughput performance by using SSD or NVMe-type drives.