DataStax has launched several products that support application development in the generative AI and retrieval-augmented generation (RAG) space, including JVector for vector storage and Langflow, a low-code visual tool for rapid GenAI development. However, this led to new testing challenges, particularly assessing semantic performance of modern application stacks. To address these challenges, DataStax integrated the RAGChecker framework with its own products due to its comprehensive capabilities and alignment with strategic goals. RAGChecker evaluates the accuracy, relevance, and completeness of responses generated by large language models (LLMs), which falls outside the typical realm of testing performance features such as latency and throughput. The framework addresses challenges like diverse RAG architectures, dynamic landscape, human involvement, and complex metrics, providing fine-grained diagnostics, supporting dynamic datasets, and aligning with GenAI product stacks. It uses a configurable LLM provider to perform claim-level entailment checks and generates detailed metrics to provide insight into the strengths and weaknesses of various components in the application under test. The integration with Langflow enables varying components of the RAG application prior to evaluation, allowing for iteration over choices made and modification at runtime to come up with repeatable and detailed metrics. This framework establishes a systematic approach to evaluating the effectiveness of RAG applications using clear and understandable scoring metrics, enabling refinement of applications for greater semantic accuracy in the fast-moving world of AI and RAG applications.