LegalBench: The LLM Benchmark for Legal Reasoning

LegalBench is a collaborative legal reasoning benchmark designed to test the abilities of large language models (LLMs) like GPT-3 and Jurassic. Unlike other benchmarks, LegalBench is an ongoing project that anyone can contribute to. Its goal is not to replace lawyers but to determine the extent to which these systems can execute tasks requiring legal reasoning, thus augmenting, educating, or assisting them. The benchmark includes two types of tasks: IRAC reasoning and non-IRAC reasoning. LegalBench employs the IRAC framework to categorize and evaluate various legal tasks, including issue, rule, application, and conclusion tasks. It also includes non-IRAC tasks, referred to as “classification tasks.” The project is ongoing, with community contributors creating additional tasks according to the benchmark's guidelines.