RAG evaluation metrics are designed to assess the performance of retriever and generator components in Retrieval-Augmented Generation (RAG) systems, which provide context to LLMs for generating tailored outputs. However, these metrics often fall short for use-case-specific applications and may not be sufficient to protect against breaking changes in collaborative development environments. To address this, DeepEval is an open-source evaluation framework that offers a comprehensive set of 14 evaluation metrics, supports parallel test execution, and is deeply integrated with Confident AI, the world's first open-source evaluation infrastructure for LLMs. By incorporating evaluations into CI/CD pipelines, organizations can ensure the quality and reliability of their RAG applications and prevent breaking changes. The framework provides a flexible and customizable solution for evaluating LLMs, including support for parallel test execution, customizable passing thresholds, and integration with popular testing frameworks such as Pytest. With DeepEval, developers can create robust and reliable RAG applications that meet the needs of various use cases and applications.