Evaluating the Generation Stage in RAG

In the evaluation of retrieval-augmented generation (RAG), the focus is often on the retrieval stage while the generation phase receives less attention. A series of tests were conducted to assess how different models handle the generation phase, and it was found that Anthropic's Claude outperformed OpenAI's GPT-4 in generating responses. This outcome was unexpected as GPT-4 usually has a strong lead in evaluations. The verbosity of Claude's responses seemed to support accuracy, as the model "thought out loud" to reach conclusions. When prompted to explain itself before answering questions, GPT-4's accuracy improved dramatically, resulting in perfect responses. This raises the question of whether verbosity is a feature or a flaw. Verbose responses may enable models to reinforce correct answers by generating context that enhances understanding. The tests covered various generation challenges beyond straightforward fact retrieval and showed that prompt design plays a significant role in improving response accuracy. For applications that synthesize data, model evaluations should consider generation accuracy alongside retrieval.