The ARC Benchmark: Evaluating LLMs' Reasoning Abilities

The ARC Benchmark is a challenging test for large language models (LLMs) that focuses on their reasoning abilities and knowledge in answering questions. Developed by Clark et al. in 2018, the AI2 Reasoning Challenge benchmark aimed to push LLMs beyond simple fact retrieval tasks and evaluate their ability to answer complex, multi-faceted questions requiring reasoning, commonsense knowledge, and deep comprehension skills. The ARC dataset contains 7787 non-diagram, multiple-choice science questions split into an "Easy Set" and a "Challenge Set." The Challenge Set includes questions that stumped retrieval-based and word co-occurrence algorithms, making them more difficult for LLMs to answer. ARC also offers the ARC Corpus, which contains 14 million sentences relevant to the questions in the dataset, designed to help models solve ARC questions without outright memorizing answers. The benchmark has been used to evaluate various LLMs and continues to be a valuable tool for assessing their question-answering capabilities.