[AARR] Evaluation of Retrieval-Augmented Generation: A Survey

The Align AI Research Review introduces an analysis framework called RGAR to systematically assess Retrieval Augmented Generation (RAG) systems. RAG is crucial in NLP for optimal retrieval methods and generating better responses. The RGAR framework considers the Target, Dataset, and Metric comprehensively. It provides relevance, accuracy, and faithfulness by encompassing both potential output and ground truth pairings. The evaluation process includes three key questions: what should be the Evaluation Target, how should the Evaluation Dataset be assessed, and how should the Evaluation Metric be quantified? Retrieval metrics focus on relevance, precision, diversity, and reliability, while generation metrics emphasize coherence, relevance, fluency, and alignment with human perception. The research also discusses additional requirements such as latency, diversity, noise robustness, negative rejection, and counterfactual robustness.